The Backup Market: It's awful

One of my family (call her "Z") members had a nasty laptop drive failure. I've had three in the past year; two on a work laptop. one on a long-abused home laptop. Laptops are hell on hard drives. Annoying for me, since I'm obsessive about backup. Worse for Z, because she wasn't.

Z asked me about backup, especially for images. I started to give her an update on my longtime backup strategy (Backing up a mixed OS X and Windows 2000 Home Network is dated now, but the general approach is unchanged even though Retrospect Pro is now zombie-ware.). Midway through my email I realized I was ranting.

Ranting. That's what blogs are for. So I'll do the email here instead.

Backup solutions really suck today. I used to do tape backup with multiple redundant offsite rotating tapes. A backup took all night, but I didn't care -- I was asleep. Dantz Retrospect was great back then, and even on the PC there were decent home products. Fast forward 10 years and the options are lousy. I use zombie software - the basically abandoned Retrospect Professional for Windows to backup my XP/OS X LAN. When it dies (probably when 10.5 comes out) there will be no automated alternatives.

Instead of multiple redundant off-site tape backups I use two rotating 300GB USB drives. It would be nice if they both had reliable quite fans and powered down when not in use to keep the drives alive. Nice, but not so. Redundancy is limited to Retrospect's incremental backup -- nice, but it only lasts as long as Retrospect.

I've been waiting for a year for Google to deploy an offsite backup product, but nothing has shown up from them. There are several products built around Amazon's S3, but I hear nothing about how well they work. Apple's offsite backup solution is pathetically small and very expensive. I trust Google to get this right and stick around, but not a small startup.

So what's the average person to do? In Z's case she has only to backup a single XP machine. So, although I've not tested the software, I'd suggest this:

1. Dual rotating USB drives like I use. Every 1-2 weeks carry one offsite. I keep my offsite drive at work. Encrypt the backup, it wouldn't be good to lose an unencrypted backup. Good luck finding one that will spin down when not in use.
   
   
2. Find software that supports the USB drives and encryption. Alas, I don't know a product to support. You want it fully automated; even the push-button solutions aren't automated enough. (Anyone with ideas?). Look for an external drive/enclosure software solution that gets decent reviews and buy two of 'em. Expect to pay at least $500. Real-time backup is nice in theory but it doesn't work with today's machines (RAID excepted, but that's not a real backup). Once a night for everything is enough.
   
   
3. In XP it's almost impossible to backup all your data without backing up the entire hard drive. What can I say, XP sucks too. So plan on backing up EVERYTHING, not just data. You need that to get all the nooks and crannies that XP stuffs data into.
   
   
4. Use a separate, intermittent backup solution for your most precious stuff. Typically burn CDs and DVDs of images and toss them in a drawer at work. Use a photo sharing site that supports full uploads and has a restore service (for a fee send images on DVD). SmugMug does this, Picasa WebAlbums (Google) might.
   
   
5. Test your ability to restore from backup every month or so. Randomly select some files and try the restore. You will be impressed how often this reveals serious problems.
   
   
6. Real geeks also mirror their most important systems every few months. It's easy to restore a mirror then restore data from backup. Even I don't do this religiously however.
   
   
7. Future: If Google ever does offsite backup, I'll switch from the rotating USB drives to a gibabit ethernet network access storage device (NAS) with a RAID array (warm swap). That way one of the backup disks could fail without a problem and I'd have a completely distinct offsite backup solution. Actually, what I really want, is for Google to assemble and sell the NAS device and the software that backs up the NAS to Google's network. They could even lease it as part of a service offering ...
   

That's the best I can come up with. I'll have more specific backup software solutions when I finally have to replace Retrospect Professional.

Which leaves the interesting questions - why is backup so bad? It's even lousy at the corporate level. If I had to rely on our corporate backup solution those two drive failures would have been much worse than annoyances -- I have my own personal backup at work too. My best guesses as to why it's so bad now are:

1. Laptops. I'd guesstimate laptops have quadrupled the risk of large scale data loss. We didn't used to keep all our data on them, now we do. This is not a stable situation. We need to migrate to devices with 20GB of solid state storage (never crashes), intelligent caching and synchronization, and remote network accessible primary data stores. Laptops with 200GB hard drives are a tool of Satan.
   
   
2. Cheap storage: CPUs have disappointed for years, but hard drives have really lived up to Moore's "law". Honking hard drives have allowed for iPhoto and iMovie, and enormous data stores we can't back up.
   
   
3. Historically most people didn't care about their personal data. They really didn't mind that much if it vanished; in fact, it kind of simplified their computing existence. That's only changed recently with digital photography; not enough time to build a market, especially given #7.
   
   
4. Probability estimations were not very important for our evolutionary success. We just aren't much good at probability. Consider the Iraq War as exhibit A and America's 'no-fly list' as exhibit B. Since we can't deal with probability properly, we underfund backup. Since we underfund backup, there's no market for backup solutions.
   
   
5. Since we don't value backup enough, vendors haven't written backup support into file systems and operating sytems. (Apple may do this with OS X 10.5 -- at last.) That makes backup software harder to write and less reliable - especially in XP.
   
   
6. Since current backup solutions are awful, it's easy for almost everyone to basically do nothing.
   
   
7. The expectation that either Google or Microsoft will take over offsite online backup has blocked any serious capital from going to build a competing solution. Why spend all that money when the big guns will take it all away?

Update 12/22/06: More on online backup solutions.