The Backup Market: It's awful

One of my family (call her "Z") members had a nasty laptop drive failure. I've had three in the past year; two on a work laptop. one on a long-abused home laptop. Laptops are hell on hard drives. Annoying for me, since I'm obsessive about backup. Worse for Z, because she wasn't.

Z asked me about backup, especially for images. I started to give her an update on my longtime backup strategy (Backing up a mixed OS X and Windows 2000 Home Network is dated now, but the general approach is unchanged even though Retrospect Pro is now zombie-ware.). Midway through my email I realized I was ranting.

Ranting. That's what blogs are for. So I'll do the email here instead.

Backup solutions really suck today. I used to do tape backup with multiple redundant offsite rotating tapes. A backup took all night, but I didn't care -- I was asleep. Dantz Retrospect was great back then, and even on the PC there were decent home products. Fast forward 10 years and the options are lousy. I use zombie software - the basically abandoned Retrospect Professional for Windows to backup my XP/OS X LAN. When it dies (probably when 10.5 comes out) there will be no automated alternatives.

Instead of multiple redundant off-site tape backups I use two rotating 300GB USB drives. It would be nice if they both had reliable quite fans and powered down when not in use to keep the drives alive. Nice, but not so. Redundancy is limited to Retrospect's incremental backup -- nice, but it only lasts as long as Retrospect.

I've been waiting for a year for Google to deploy an offsite backup product, but nothing has shown up from them. There are several products built around Amazon's S3, but I hear nothing about how well they work. Apple's offsite backup solution is pathetically small and very expensive. I trust Google to get this right and stick around, but not a small startup.

So what's the average person to do? In Z's case she has only to backup a single XP machine. So, although I've not tested the software, I'd suggest this:

1. Dual rotating USB drives like I use. Every 1-2 weeks carry one offsite. I keep my offsite drive at work. Encrypt the backup, it wouldn't be good to lose an unencrypted backup. Good luck finding one that will spin down when not in use.
   
   
2. Find software that supports the USB drives and encryption. Alas, I don't know a product to support. You want it fully automated; even the push-button solutions aren't automated enough. (Anyone with ideas?). Look for an external drive/enclosure software solution that gets decent reviews and buy two of 'em. Expect to pay at least $500. Real-time backup is nice in theory but it doesn't work with today's machines (RAID excepted, but that's not a real backup). Once a night for everything is enough.
   
   
3. In XP it's almost impossible to backup all your data without backing up the entire hard drive. What can I say, XP sucks too. So plan on backing up EVERYTHING, not just data. You need that to get all the nooks and crannies that XP stuffs data into.
   
   
4. Use a separate, intermittent backup solution for your most precious stuff. Typically burn CDs and DVDs of images and toss them in a drawer at work. Use a photo sharing site that supports full uploads and has a restore service (for a fee send images on DVD). SmugMug does this, Picasa WebAlbums (Google) might.
   
   
5. Test your ability to restore from backup every month or so. Randomly select some files and try the restore. You will be impressed how often this reveals serious problems.
   
   
6. Real geeks also mirror their most important systems every few months. It's easy to restore a mirror then restore data from backup. Even I don't do this religiously however.
   
   
7. Future: If Google ever does offsite backup, I'll switch from the rotating USB drives to a gibabit ethernet network access storage device (NAS) with a RAID array (warm swap). That way one of the backup disks could fail without a problem and I'd have a completely distinct offsite backup solution. Actually, what I really want, is for Google to assemble and sell the NAS device and the software that backs up the NAS to Google's network. They could even lease it as part of a service offering ...
   

That's the best I can come up with. I'll have more specific backup software solutions when I finally have to replace Retrospect Professional.

Which leaves the interesting questions - why is backup so bad? It's even lousy at the corporate level. If I had to rely on our corporate backup solution those two drive failures would have been much worse than annoyances -- I have my own personal backup at work too. My best guesses as to why it's so bad now are:

1. Laptops. I'd guesstimate laptops have quadrupled the risk of large scale data loss. We didn't used to keep all our data on them, now we do. This is not a stable situation. We need to migrate to devices with 20GB of solid state storage (never crashes), intelligent caching and synchronization, and remote network accessible primary data stores. Laptops with 200GB hard drives are a tool of Satan.
   
   
2. Cheap storage: CPUs have disappointed for years, but hard drives have really lived up to Moore's "law". Honking hard drives have allowed for iPhoto and iMovie, and enormous data stores we can't back up.
   
   
3. Historically most people didn't care about their personal data. They really didn't mind that much if it vanished; in fact, it kind of simplified their computing existence. That's only changed recently with digital photography; not enough time to build a market, especially given #7.
   
   
4. Probability estimations were not very important for our evolutionary success. We just aren't much good at probability. Consider the Iraq War as exhibit A and America's 'no-fly list' as exhibit B. Since we can't deal with probability properly, we underfund backup. Since we underfund backup, there's no market for backup solutions.
   
   
5. Since we don't value backup enough, vendors haven't written backup support into file systems and operating sytems. (Apple may do this with OS X 10.5 -- at last.) That makes backup software harder to write and less reliable - especially in XP.
   
   
6. Since current backup solutions are awful, it's easy for almost everyone to basically do nothing.
   
   
7. The expectation that either Google or Microsoft will take over offsite online backup has blocked any serious capital from going to build a competing solution. Why spend all that money when the big guns will take it all away?

Update 12/22/06: More on online backup solutions.

The memory could not be read: Anatomy of a complex series of Microsoft defects

This is how it began. A notice on startup from svchost.exe that referenced memory at 0x00000000 could not be read.







Not a very helpful error message. Then I looked into the XP event log, that invaluable and invisible resource that Microsoft hides from users. The only thing worse than an unresponsive fire alarm is a silent fire alarm. Microsoft's technological indictments include hiding fatal errors from users.

There I saw a number of cryptic messages indicating that my automated updates had been failing -- and that this might also have been causing my automated Retrospect Professional backups to fail (that's by far the most serious problem here). That's when the detective work began, a series of probes and explorations powered by Google. The bottom line fixes were, I think:

• Turn off Windows Automatic Update (aka Microsoft Update)
  
• Use Regedit to find "LocalCacheDrive" settings for Office. Notice that the drive letter is "D" when it should be "C". Change it.
• Repair the Office installation.
• Manually run Windows Update.
• Discover Flash update still doesn't work.
• Download Flash uninstaller from Adobe/Macromedia. Uninstall Flash. Reinstall Flash.
• Windows Update now has no errors.
• I also, though I'm not sure it was necessary, did: "net stop WuAuServ", remove "windows\SoftwareDistribution" and "net start WuAuServ"
• Changed Windows Automatic Update to download but do not install.
  

It all seems to have resolved. In retrospect I think this all began when I installed Office with the 'remove install file option'. I use that option because I kept my install files on a hard drive. Alas, a separate hard drive in those days. Drive letter "D". Nowadays drive letter D: does not exist. The bug bites when Office looks for its install files using a drive letter that no longer exists. Office doesn't produce a dialog box or a reasonable error message, it just dies. The automated install process persists daily in the futile update. (Update: I've been told this was an ancient bug with Windows update too.)
I've written recently about Microsoft's mess This is a good example. In the process of debugging I ran into:

1. Listool.exe: the app that was supposed to fix this no longer works the way Microsoft thinks it does. My guess was they found a serious bug in it, stripped out the dangerous functionality, and never updated their many references to it.
2. All those critical alerts in the XP Events log that one has to dig down to find.
3. Incredibly obscure hex error messages.
4. Thousands of web pages related to automatic updater problems.
5. XP error messages with embedded URLs that no longer work or redirect.
6. An incredible bug in office where a change to a drive letter causes an incredible pile-up of cascading bugs.

I can't break the repair process down in much more detail because it was an insanely complex process. All I can do below is provide a set of links to web resources and some of the many different error messages I got along the way. I hope the time I spent putting this together will help someone; it's my way to repay those who's work helped debug this.

Here are of the more and less helpful web pages:

• Office updates are not successfully installed if you deploy updates by using Windows Server Update Services or the Microsoft Update Web site after the path of the local installation source changes
• Office 2003 Tool: Local Installation Source Tool: This looks promising, until you discover that Listool.exe has been broken and Microsoft hasn't noticed. The recommended fix is (remove=all) is not a supported command line option with this version. I suspect they lost the new version and somehow replaced it with an old one.
• Office updates are not successfully installed if you deploy updates by using Windows Server Update Services or the Microsoft Update Web site after the path of the local installation source changes: This was part of the problem, but of course LisTool.exe doesn't work any more. Still, it was a valuable clue.
• Windows Update Broke My Machine (svchost.exe -- application error), and How to Fix It: This was closely related. I followed some of the advice here. In particular "net stop WuAuServ", remove "windows\SoftwareDistribution" and "net start WuAuServ". I also turned off Windows automatic update - permanently.
• You receive a "Error 1327. Invalid Drive... " error message when you install Office 2003: This generally useless article was critical because of this one sentence: "Before you try to install Office 2003, check to see if the MSOCache folder exists. If it exists, you must move it to the location that is specified by the LocalCacheDrive registry subkey. To do this, use the Office 2003 Local Installation Source Tool (LISTool.exe)." I knew LISTool.exe was defective, but now I knew what registry key to manually edit using regedit.
• MS06-020: Vulnerabilities in Macromedia Flash Player from Adobe could allow remote code execution: This Microsoft automatic update failed with "Error Code: 0x643". I had to find the Flash uninstaller, run it, then run the Flash installer. Note the Flash installer has no UI -- it doesn't even say that it completed successfully.
• There are LOT of knowledge base articles related to automatic update failures.
• The Generic Host Process error. I got that one too, but this wasn't all that helpful.
• Solution available for Windows XP "Generic Host Process" error: This was one of the few links from Microsoft's logs and crashes that wasn't broken. I followed the recommendations, but I can't recall if this did anything.
  

Here are a few of the many error message and long entries I worked through:

Faulting application , version 0.0.0.0, faulting module unknown, version 0.0.0.0, fault address
0x00000000.
For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.

Product: Microsoft Office Professional Edition 2003 -- Error 1327. Invalid Drive: D:
For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.

Product: Microsoft Office Professional Edition 2003 - Update 'Security Update for PowerPoint 2003
(KB923091): POWERPNT' could not be installed. Error code 1603. Windows Installer can create logs to help troubleshoot issues with installing software packages. Use the following link for instructions on turning on logging support: http://go.microsoft.com/fwlink/?LinkId=23127

Error Code: 0x52F
Try to install the update again, or request help from one of the following resources.

Product: Microsoft Office Professional Edition 2003 -- Error 1327. Invalid Drive: D:
Product: Microsoft Office Professional Edition 2003 - Update 'Security Update for Excel 2003 (KB923088): EXCEL' could not be installed. Error code 1603. Windows Installer can create logs to help troubleshoot issues with installing software packages.

Faulting application svchost.exe, version 0.0.0.0, faulting module unknown, version 0.0.0.0, fault address 0x00000000.

Update 3/27/07: I think this bug with Windows update scattering files across external drives may be a related flaw. Microsoft might have cleaned up the offending updater, but you need to delete the junk folders. I found I could not delete "updspapi.dll" from the update folder until I assigned full control permissions to myself as administrator.

Update 4/11/07: Another windows update (five fixes) again broke XP with the same error message, but the LocalCacheDrive setting was unchanged. I've disabled Microsoft Update for now. I really need to get rid of my last XP machine.

keywords: drive mapping, drive letter, windows update.

OS X blogging tools:

Five blogging apps I'm thankful for - The Unofficial Apple Weblog (TUAW)

It's a good read. I'm going to try MarsEdit again with Blogger.

CompanionLink: another sync vendor

A correspondent, Joseph, tells me the professional version of CompanionLink 2.0 supports selective synchronization to a Palm device. (Sync only part of the database while at work, all of it at home).

They claim to support a wide range of devices and sync targets, including Act, GoldMine, Notes and Google Calendar. That makes me nervous; it's hard to see how they can manage all of the data model conflicts between so many different devices and platforms. A free evaluation copy is available, but this is not easy software to evaluate. When things go wrong with synchronization they can wreak havoc on irreplaceable data that may be difficult to restore (such as Exchange server data). The vendor does not have any open support forums; that's another sign that this software may not be trouble-free.

I'll ask Joseph to share his experiences.

MacBook Core 2 Duo: My Amazon Review

Below is an edited and updated version of my Amazon review of the Core 2 Duo MacBook. In brief: good product, some odd choices, I'm not all that impressed with Intel, and I had a mediocre experience with Parallels.

Amazon.com: Apple MacBook MA701LL/A 13.3" Notebook PC (2.0 GHz Intel Core 2 Duo, 1 GB RAM, 120 GB Hard Drive, DVD/CD SuperDrive) - Black: Computers & PC Hardware

I bought a build-to-order version of this machine from Apple educational with 2GB of RAM. If one uses the Macintouch link to purchase this from Amazon you get a better deal and help out a web site that provides great Mac tech support.

Thus far I've gone through user migration, several days of light use, and one test of Parallels (which was disappointing). It's worth the money and it's one of the best laptops I've bought (I do miss my 1990 vintage PowerBook 165 though!). I've been nursing my broken-hinges G3 iBook so I could get the 2nd rev of the MacBook, it was worth the wait. I don't see any evidence of the serious heat problems that afflicted the first MacBook and I assume they've mostly fixed the abrupt shutdown problem. The odd looking keyboard is fine and the large trackpad works well -- though I'm not sure the tap-to-click is worth the occasional errant clicks. The accompanying documentation is extremely minimal but very well done and sufficient for most users. Those who need more probably wouldn't read a bigger manual anyway.

You should be happy, especially since the rest of my review will prepare you for the minor annoyances.

Now that I'm done saying nice things, here are the nits. Firstly, it's too sharp. Some whacko at Apple design figured 'sharp' was in this year, and the edges of the case annoy wrists when typing and fingers when carrying by the base. This is not all bad; it ensures good wrist position because it's uncomfortable to type wresting your wrists on the front edge. Overall though, dumb.

The screen is the DVD-friendly widescreen form factor, so the 13.3" is less useable for computing work (photos, etc) than one would wish. Annoying, but hard to escape these days. It comes with a remote for watching DVDs and listening to music, but really I'd rather have had the original iBook form factor. Compared to my older 12" iBook the machine is thinner, longer, wider and about the same weight. The fans run often, are very audible, and can be very noisy, but the base is not excessively hot. PowerPC (legacy applications that invoke Rosetta when used) cause fans to run and the battery to decline quickly.

The battery is twice the size of the G4's and yields comparable or less battery life. I'm unimpressed with the power/heat/performance ratio of Intel's hyped architecture; I had to significantly drop the screen brightness to get a reasonable battery life. The conversion from PowerPC to Intel has thus far underwhelmed me, especially since I was disappointed with Parallels (see below). The machine, for example, takes longer to sleep -- making it much too easy to run off with with a spinning hard drive. You have to shut the case and look for the cycling sleep light.

In terms of connectors I don't miss the modem. The video connector seems proprietary, but I'm used to that from Apple. The VGA and DVI cables are reasonably priced and I bought both (check what comes with this machine). The ability to support two desktops is the one really big improvement over the G4 iBook.

The built in camera is silly. It's low resolution with a tiny lens and awful performance with ambient indoor light. The MagSafe power connector is impressive, even though it means there won't be non-Apple licensed chargers for this machine. Apple has retained the rest of their power adapter design, which has been the best in the industry. (I'm big on power adapters as a leading indicator of product excellence).

The MacBook comes with much less bundled than my G5 iMac. In part that may be because Apple probably wants to avoid non-"universal" (intel-compatible) apps and in part to reduce support costs and product cost. You get Apple's superb iLife apps, some simple games, OmniOutliner, and a "comic" generator. No encyclopedia, etc.

I used Apple's user migration to move 3 user accounts, software, files, etc from an old, old iBook. This requires a firewire cable, so IF YOU WANT TO DO USER MIGRATION BUY OR BORROW A FIREWIRE CABLE. The firewire cable is not included with this MacBook. The migration process is well integrated into initial setup and, all by itself, probably pays for any price difference between the bottom-of-the-barrel Dell laptop and the MacBook. I did run across some minor cleanup tasks, but the iBook had very old stuff. Migration takes longer than you expect and periodically pauses for long minutes; it's not frozen, just thinking. Start it and come back in a few hours.

I bought the InCase neoprene sleeve to protect the MacBook when it's carried in a bag or backpack. It's a handsome, well made case, but it desperately needs some thin, lightweight strap-type nylon carrying handles. Sigh. I guess they omitted them for price and aesthetic reasons, but I recommend looking for something with straps. Not recommended.

Also not-yet-recommended is the famous Parallels Windows emulation environment. I downloaded the trial version and attempted to install Windows 98. Ugh. Awful. Wasted hours. I don't know if it's this bad with every machine or if they've not tuned it to the new MacBook. Flawed documentation, flawed auto-configuration. slow, huge power drain, didn't work, etc. I may try XP with Parallels once I recover. I recommend testing Parallels before you buy. If you do install XP and enable network access you are paying for XP, Parallels, and an antivirus solution.

The bottom line is that this looks like a good OS X laptop without some of the flaws of the first MacBook. Since any OS X laptop is an improvement over an XP laptop it's an excellent choice for everyone. If you intend to run Windows software I think Apple's Boot Camp (separate download) may be more reliable and elegant than Parallels.

Update 11/25/06: The MacBook is almost fast enough to run Aperture. If Apple ever fixes Aperture, it might be a decent platform for smaller projects. It takes longer to go to sleep than my G3 iBook -- it's easy to pick it up and start moving before the lazy blinking light begins.
Update 11/26/06: Adobe/Macromedia Shockwave doesn't run natively on Intel. Since it uses Rosetta, the browser has to use Rosetta too. This rules out most kids game sites.

Parallels Desktop for Mac: Bloody awful

Update 12/2/06: Ahh. The Windows 2000 story is much nicer.

Update 11/23/06: Boot Camp feels kind of crude as well, it uses up more drive space, and it doesn't do as good a job as a VM does controlling nasties. I think I'll wait a month and then try Parallels and XP again with the kids games. If they do better, then I'll go that way.

Update 11/23: I tried again with XP Pro. (I disabled network services for the Mac user because I was reusing my XP serial number for the test -- XP allows 30 days of use this way). This worked far better and took a fraction of the time to install. It didn't, however, manage the kids games. It through range errors, had trouble with resolution switches, etc. Since the kids games are a big part of my interest in running XP on this machine I'm going to try Boot Camp first. I can see how Parallels could be worth it when running a small number of well behaved business applications strictly with XP. They should never, however, have claimed to provide Windows 98 support! (See below.)

Parallels Desktop for Mac has been the toast of the Mac world for months. It's supposed to allow one to host a windows VM on an Intel Mac. I gave the free demo a try on my Mac Book Core-2 Duo with 2GM of RAM.

Wow.

That was really bad. Maybe all those reviews I read came from a parallel universe. I tried configuring it with Windows 98. I don't know where to begin with all the bugs I came across -- before I gave up. Briefly:

1. The documentation is wrong about where the device drivers are stored.

2. The default VM configuration had about 3 errors I had to correct to get things most of the way to working.

3. The necessary device drivers are buried on an image that has to be loaded via the VM configuration panel.

4. It was slow, slow, slow --- and it sucked power.

5. The widespread simple grammatical errors turned out to predict the entire user experience.

Perhaps the problem is it really only supports XP, but I thought Win 98 was a reasonable test. What a waste of time!

Looks like I'll give boot camp a try. I'm only glad I didn't pay money for this hunk of junk.

BlogJet This! for Firefox 2.0

BlogJet is an excellent XP blogging tool, but the author should have done a better job of pointing users to the Firefox 2.0 plugin: BlogJet - Blog - BlogJet This! for Firefox 2.0.

I'm disappointed that he seems to have been planning to charge for the Firefox 2.0 extension update, or even to require an update to a new BlogJet version to get the extension update. It's a great product, but ...