VMware Virtual Machines - the backup problem

It's times like this that I really miss Byte (or BYTE?) magazine. They would have had great coverage of VMWare VMs - how they work, and what the risks are. Now that's specialist knowledge. Knowledge that, when I use Google, is obscured by a haze of marketing material.

The best we non-specialists can do is share our limited experience in blog posts, like this one sharing my experience with VM backup. That's been a problem for me.

First - my experience. I've used VMWare Fusion on my Macs for a few years. I need it less than once a month, typically to launch XPSP4 and run Access or (yech) Quicken. On the other hand, I configured and use a VMWare Workstation on a 64bit Win7 machine at work. That VM is running a Windows 2003 Server environment with terminal server and I use it very frequently.

Both my Fusion and Workstation VMs are configured to store the VM data as many files rather than a single monolithic file. Both are about 80-100 GB in size and store as little of my data as possible; on the Mac the individual .vmdk files vary in size from about 200 to 500 MB. I don't have the Workstation VM at hand but I think its files are all a fixed size.

The host OS X machine is backed up using Time Capsule (sigh) and SuperDuper! (sigh). Neither give me the warm fuzzies of Retrospect at its best. The Windows 7 machine is backed up using (Dantz -> EMC -> Roxio) Retrospect Professional.

I configured both VMs to use multiple files because of the VM backup problem I knew about.

The obvious backup problem for these machines is that if you configure a VM as one monolithic file, then every time you touch it the host system backup software has to backup a 100GB backup event. That will overload Time Machine (Capsule) or Retrospect pretty quickly. (More sophisticated backup software can manage this differently, but I don't think TM or Retrospect can.)

That's why I went with separate files. Backups would only have to manage the files that changed. (Ahh, but how does the backup software know what's changed - esp. if the files are a fixed size?)

I think that approach does work when the VM is shut down. I think it works on my Mac. It doesn't work with Retrospect Professional on the Windows 7 machine where our VM is always running.

I learned that the hard way when we tried to do a restore. The restored VM seemed good at first, but it was soon clear that we'd somehow ended up with different time slices. We had to kill the VM. Fortunately, because I'm justifiably paranoid about backup, we also had a file system backup that was only a few weeks old. Since we don't keep data on the VM we lost very little.

This is a nasty problem. As best I can tell, at least on Windows, Retrospect Professional can't do a reliable backup of a running multi-file VMWare VM. The limited VMWare marketing material I could find suggests this isn't just a Retrospect problem. The solution is, of course, to buy their costly backup software. You can also do backup from within the client OS, but that adds a new level of cost and complexity to overall backup. Retrospect Professional, for example, won't install on Windows 2003 server. For that you need their much more costly server backup.

Now you know what I know. If you know any more, or can point me to anything that's not marketing material, I'd be grateful.

I do miss Byte.

--My Google Reader Shared items (feed)

Transferring the mnspecialhockey.org domain - what remains of Google Apps

The last step of the mnspecialhockey.org migration took place at DreamHost. The provide an optional mixed hosting service. I had some files in a traditional ftp site on a DH server, but also Gmail and Google Apps via Google.

From their control panel I turned off the Gmail and Google Apps connection. (See the posts list below for my prior action).

Then, since DH was also the registrar for mnspecialhockey.org I switched to the new DNS servers. I'll update this post if I learn what really happens to the data that was at the old site.

See also (note, some are old)!

• Closing a Google Apps domain: document liberation and more 2010
• Closing a Google Apps project: downloading sites 2010
• Transferring a Google Apps eNom domain: Awkward!
• How to create a google apps custom domain and move your blogger blogs to your new domain (2007)
• Gordon's Tech: Migrating a domain with related Google Apps and hosting files to a new service (2008). How I configured mnspecialhockey.org Google Apps after moving from Lunarpages.
• Gordon's Tech: Google Apps: Configuration for a non-Google managed domain (2007) - original mnspecialhockey.org Google Apps

Update 8/29/10: After the transfer

The mnspecialhockey.org domain now points to a new site. What happened to the old content?

• sites: they still exist! The URLs are of the form: http://sites.google.com/a/mnspecialhockey.org/sites/system/app/pages/meta/dashboard 
• docs: they still exist! The URL is of the form: http://docs.google.com/a/mnspecialhockey.org/#all
• mail: it still exists for my account (I removed the others) with URL: http://mail.google.com/a/mnspecialhockey.org/?tab=om#inbox

In fact, I can still manage the domain using the URL

https://www.google.com/a/cpanel/mnspecialhockey.org/Dashboard

So I have, at least for the moment, answered the question about what happens to Google Apps content when the original Google App Domain is gone. Turns out, it persists. Even the Domain settings persist (Google Apps thinks the domain is still mnspecialhockey.org, even though it no longer references this site).

If this remains true, it shows there's only a loose coupling between a domain-associated Google Apps suite and the domain itself. 

That's rather nice to know. It's not completely surprising however.

Closing a Google Apps domain: document liberation and more

I'm closing down a Google Apps domain because the domain name is moving (donated) to a new owner. Since I don't know what happens to the data after I transfer the domain name I've so far:

• Exported the "Sites" pages using a Data Liberation Tool
• Deleted all the existing site user accounts

My next step is to get a local copy of all the documents. Again, the DLF has directions. This turns out to be built into the Docs framework. On export you can specify a file format, I chose RTF for documents, Excel for spreadsheets, and native for the rest (PDF).

I also made sure so share all the documents with my personal account, but I wasn't able to transfer ownership. Currently that's not supported in Google Apps.

We had the non-profit version of Google Apps (free from Google) so we had all the yummy pro features, like the ability to associated a secondary domain. I have some I could use and I considered doing that to provide a way to access data post transfer, but I decided to skip that step in the interests of time.

Closing a Google Apps project: downloading sites

When MN Special Hockey signed up with a web hosting project, I needed to transfer the domain.

The domain was managed through my Dreamhost (love 'em - click here to evaluate my discount on their ISP services) accounts. There it has associated Google Apps services. I really didn't know what would happen to the account data when I switched away the host name, so I decided to see what I could backup.

Naturally I turned to Google's Data Liberation Front - my heroes. They provide a Java app for this purpose. It can allegedly be used to move a site from one location to another! 

Sites - the Data Liberation Front

A site can be downloaded using the open-source tool available at the Google Sites Liberation Project Page. Note that this tool requires Java 1.5 or later. Once the tool has been downloaded, double-click on the JAR file to launch the application. 

It opened easily on OS X 10.6.

You will need the documentation. Instead of www.mnspecialhockey.org (didn't fit their model) I tried sites.mnspecialhockey.org and I found two sites to copy:

• http://sites.google.com/a/mnspecialhockey.org/www/
• http://sites.google.com/a/mnspecialhockey.org/mn-special-hockey-site/

From their documentation I think the values are:

• host: sites.google.com (optional)
• domain: mnspecialhockey.org (optional)
• webspace: www and mn-special-hockey-site (two webspaces)

There's an option to download revisions, but I didn't care about that. I was fine with the latest version.

It worked, but the result was not pretty! Without style sheets it renders poorly in Safari, and  some of the character encoding was off. Still, the bulk of the content was there including associated PDFs that were a part of the site.

The Java app is pretty simple. You have to quit and restart to do a second site. Still, did I mention it worked?

The language hole in Blogger's comment spam filtering

This comment evaded Blogger's comment spam filters ....

На нашем Видео каталоге вы можете Посмотреть Видеосюжеты, захотите на Простой видео сайт

It contained an embedded link.

The language is Russian [1]. Google translate gives us ...

On our video directory, you can watch video projects, you want for a simple video site

Blogger needs to treat non-blog-language posts as spam, or at least require review with auto-translation to the native language. This one should have been caught.

If I get another one like this (presumably later today) I'll have to turn on authenticated commenting until Google catches up.

[1] Why would anyone bother to post spam in a language that cannot be read? Well, for one thing, Google's search engine can "read" the link, and so the source gets link kharma. It's also a way to find vulnerable blog targets to further exploit. I'm sure there are other benefits, such as very foolish people curiously clicking on the link; those people likely have vulnerable machines that are easy to pawn.

Cautionary tale about Google Apps for business

Bad outcome with using free Google Apps for business operations.

A few quick comments:

• Dreamhost is the best way to use free Google App. They give you document hosting, domain registration and control, support and service, and associated Google Apps. (see my kickback deal with promo code KATEVA). I use 'em, I like 'em. Unusual company.
• The connection between Google Apps and domain name is tenuous. App content and services will survive domain divorce.
• Gmail has the same problem. So do all Google services, including identity management. There is no support. That scares me.
• Google Apps pricing is a great deal for a school or even a modest business, but it scales oddly and it's too much for a paid Google Apps family account. I'd like to see Google offer more price/service options.

Facebook documentation: How do Pages work anyway?

Apple, like Galactus, is beyond Good and Evil. Google, by contrast, is the cool-kid-wannabe that sold its soul to the Devil for a pittance.

Facebook though, Facebook was born evil. Alas, it cannot be avoided. For example, I help maintain the Minnesota Inline Skating Club facebook page.

Maintaining that Page has been painful because, as one might expect for a demonic creature, Facebook is essentially undocumented. For example, I couldn't find any explanation of the rules for broadcasting "Page" status updates to "followers". (Admin posts go to all Page followers, but I think non-admin posts either go nowhere or are shared only with personal friends.). Nor could I find explanation of best practices for "Page" ownership. (Use an email address specific to the Organization to create a Facebook account that will manage the Page. The best way to proceed is to use the "Create a Page" link to create a Profile and a Page together.)

Thanks to a Pop17 post, however, I have found a few FB official pages that sort of hint at documentation  ....

• Facebook Platform
• Known Issues on Facebook
• Facebook Pages*

Buried in a post on the last link, I found the Box.net hosted Facebook Pages "manual". I guess FB doesn't have a server. The 10 page "manual" doesn't answer any of the questions I listed above.

* Couldn't they have come up with a more searchable and meaningful term than "Pages"? Like I said, pure evil.

Apple's battery charger - hold off for now

We tried NiMH chargers a few years ago, but we gave up because of high battery defect rates and short battery lifespans.

Apple claimed to address the latter, and I was willing to try them out on the former.

Unfortunately the first two batteries we charged were defective:

Amazon.com: John Faughnan "John G...'s review of Apple Battery Charger 

The first two batteries I charged generated a blinking yellow signal. Either the batteries or the charger are defective; I suspect it's the batteries.

It could be just bad luck, but I suggest waiting a while before ordering a charger.

I'd hold off on this device for a while. Wait until there are at least forty Amazon reviews so we find out how common this problem is.

NAVIGON MobileNavigator and Kensington Dash Mount - my reviews

At this time there are only two automotive GPS iPhone Apps worth considering: TomTom and NAVIGON MobileNavigator. Navigon originally cost about $80, but I bought the US/Canada version on sale for $40. TomTom is about $60 for the same app.

You can purchase real-time traffic and personalized routing services for an additional fee. There is no commitment from Navigon to provide any map updates and no information on map updates on their web site. 


There is no documentation for the MobileNavigator. It desperately needs documentation. Some users recommend the manual for the Navigon 8100T.

We've installed the app on two phones, so the per-phone cost was about $20.  Installation requires 3GB of free space, the installed product uses 1.5GB.

Navigon will not work on an iTouch, but it will work on an iPhone without a data plan. It needs a GPS, but since it has to work even when there's no cell service it's not data plan dependent. My son inherited my 3G but has no data plan, Navigon works on his phone (albeit slowly on startup). The lack of compass doesn't hurt once the car is in motion.

I have had only limited prior use of a traditional GPS device.

Very quickly, my impressions of this app are:

• It feels quite similar to a traditional automotive dedicated GPS. This is not a feature. It is only a 2nd class iPhone citizen. Even Contact integration seems to be an afterthought.
• There's no true search. We've grown so accustomed to search on iPhone Maps.app that we've forgotten how amazing that is -- and how complex. You can't simply enter a street name and choose from all the matches. It's very traditional GPS data entry. It's much easier and faster to find locations on Google Maps, Latitude or iPhone Maps.app than in Navigon.
• The turn directions don't say "take the next exit", they say "right turn in 200 feet".  When approaching exits it may not be obvious whether than's the 1st right exit or the 2nd right exit.
• I believe location services work only with the iPhone GPS. They don't have access to the cell tower triangulation or WLAN location services. So they don't work indoors very well.
• Local "Point of Interest" data in Minneapolis St. Paul is very limited.
• The map voice pronunciation is peculiar. Some local streets names were incomprehensible.
• The graphics are acceptable and visible from a dashboard mount. I remember thinking volume control was a bit odd but I can't remember why.
• Some of the routes we're given have been peculiar. It seems to be avoiding highways.
• You really need a mount to safely use the iPhone as a GPS device. I have ordered the older $21 Kensington car and dash mount because newer models don't fit the iPhone 4. We wanted a device that works with both 3GS and 4. I'll review that separately.
• Your phone gets hot using Navigon while charging. This can be a problem on a hot sunny day if the phone is beneath the windshield. Consider routing air flow to cool the phone.

Overall Navigon needs to be a lot more open about their map update plans. It is at best a 2nd rate iPhone app. The installation package of 3GB is going to be a problem for many users. I'm surprised their map data doesn't compress more than that.

It's worth $20 per phone, but only just worth $40 per phone. It's not worth $80.

Update 8/29/10:

My review of the Kensington dash mount ...

The big appeal of this car dash mount is that it works with both iPhone 3GS/3G and with iPhone 4. Very few devices can do that. 

There are two joints. If you adjust both (don't force it) you can mount this on a van windshield. It has stayed in place for days; the iPhone is a lightweight GPS. It will only adhere to glass or smooth plastic, not vinyl. For any other surface you need to use the adhesive mount, the suction cup will bind to that. 

It is lightweight, and so is the phone. So they'll vibrate, you may need to mount so the phone rests on something. 

The side grips spring out; the button and apparatus feel fragile. I try to only close them as much as needed. 

Works well with a standard Apple power connector. 

Note an iPhone will get hot when used as a GPS in this cradle. The cradle doesn't carry heat away, it is an insulator. You will want to let the windshield air conditioner run to keep the phone comfortable.

Also my route profile was misconfigured to "bicycle speed" and "scenic". I don't know if that was the default or the kids. It explains the peculiar routing
--
My Google Reader Shared items (feed)

iPhone's very annoying calendar color limitation

There are three features of the iPhone Calendar that prove Apple is not made of geniuses ...

1. The maximum alert time is 2 days. Sidekick for DOS 2.1 had longer alert times. If I could kick some engineer's butt for every birthday card I've sent late ....
2. iOS can't handle birthdays for anyone over 77. Let's see - my parents, my aunts ...
3. Calendar colors are assigned per server, not per phone. So my Family Calendar (ActiveSync via Google) and my Corporate Calendar (ActiveSync Corporate) are both assigned the color red [1]. I can't tell them apart visually.

Not only are these stupid blunders, they've been stupid blunders for over a year.

No, Apple is not made up of geniuses. Neither is Google. Both companies have lots of limitations.

[1] Quick tip: Calendar color fix for iPhone and iPod touch
... Calendar on the iPhone will assign colors in a certain order for your first five calendars: red, orange, blue, green, and purple. Afterward, any more calendars are assigned a random color out of those five. To manipulate this, use iTunes to sync only selected calendars in the “Info” tab. First sync the one that you want to be red, then sync again including your second calendar that you want to be orange, and so forth. You can create extra calendars to use to “skip” a color if you wish, then delete later at your leisure...

OS X drag folder to file save dialog - sometimes I love this OS

There are a lot of times I'm really irritated by OS X. Significant chunks of it (ex: Parental Controls) are fubared. Some unchangeable configurations (Spotlight) are annoying. Lots of embedded functionality needs to be stripped out into a separately sold product (iChat).

Sometimes though, it's a real pleasure.

I wanted to save a file to a folder I had open in a window. I dragged the target to the "Save As" dialog window and, as expected, the context changed so I could save to it.

There are other ways to duck file navigation, but this is a very useful method. I don't think it's at all new. I think I read of it years ago. I just rediscovered it because it felt like it should work.

If only the rest of OS X worked so well.

Things I hate: Shinywitebox's novel DRM for iShowU

I like paying for good software. It's a quirk, I admit.

I don't even mind well done Digital Rights Management. Without copy protection the novel software I love would die. I'm okay with how FairPlay works for iPhone apps and movies. I'm fine with long software keys -- I keep mine in a Google Doc I can access anywhere.

Where I go bonkers though is when otherwise good companies decide to introduce novel DRM methods, such as key distribution methods. I can tolerate online registration and binary keys for subscription services like Spanning Sync, but I blow a capillary when vendors like Shinywhitebox, makers of the once excellent and inexpensive iShowU screencasting software, do things like this ...

Licensing & Refunds

... At purchase time, we create an account for you on our system so that you can get instant access to your purchase history and retrieve lost unlock keys if need be. You use these same account details to unlock the product (over the internet). After starting the application, click on the “Unlock ..." button that appears in the first dialog, then enter your email address and password (these are the login details you setup when you purchased from us)...

Additionally, at the time of purchase we send out an email with your current set of “unlock keys”. This is simply an alternative method, which you can use on machine that don’t have an internet connection. If you have purchased a product but have not received your unlock keys please first check your spam folders, and failing that, contact support with the transaction number of the sale.

The keys are XML files.

To add aggravation to injury, SWB has enforced this transition for version 1 of their software, now in legacy mode. When I ran my old software I accepted offered upgrades, thinking I was getting some bug fixes. Along the way I got an entirely new registration process and my old key failed. I had to request a new key (support responded within 10 minutes late at night, which is a bit beyond the call of duty).

Come on gang. Please don't do this to your customers. I like paying to upgrade products I like (in their case the new HD product upgrade fee is a pittance), but I really don't have time for this - just time to rant about it.

Please don't make registration process changes part of "free" updates to legacy products. Please don't institute unique DRM procedures that break the way I track the 200 or so software products I've purchased. Just don't.

Here's hoping SWB is reconsidering ...

Project Shredder begins

Gordon's Rules of Acquisition remind me that the purchase price is often the smallest cost of acquisition. So I like to digest each piece of gear before I bring down new prey.

The last meals have gone relatively well. My mother's iPad and my own iPhone 4 were big bites, but they went faster than expected. The Brother 2140 and updated Airport Express were bought for trouble-free implementation and they worked as expected. Migrating my old 3G to my son was more complicated thanks to AT&T's incomprehensible and undocumented contract rules, but it too has passed.

Since Canon refuses to make the dSLR I want [1], I'm delaying replacement of my aging DigitalRebel. So my path is clear for something more challenging.

Hence Project Shredder.

I am bad at filing. I've always been bad, but it's getting worse. Once the easy stuff is tossed, the hard stuff builds up in bins and boxes. I need a better approach [2].

I have some past experience with document and image scanning including developing document management technologies, but I've never felt that the technology was really ready for my home use. Over the past few years, however, OCR and full text indexing have matured into my zone of acceptably pain-free technology. Adobe's 1996-vintage PDF image/text format has become widely accepted [3]; it's the format I've wanted for fifteen years. The Fujitsu ScanSnap line has been stable and respected for at least ten years, and their space-saving S1300 has gotten a great review from a source I trust.

So it's time to make a stab at this. I'll have more to say in future posts about the S1300 when it arrives, and how the entire process plays out at our home [4]. Look for the "document management" tag for related posts.

-- footnotes --
[1] Stop the 3$!#$% megapixels Canon. Give us ISO.
[2] Adobe, like Microsoft, wasn't always a zombie company.
[3] Though Fujitsu's software distribution strategy may lead me to a Torrent client!
[4] I'll keep the way I file receipts and manuals. Receipts get tossed into a bin in chronological order. Manuals get placed in bookcase boxes in alphabetic order with the date of acquisition written on the front page. I weed receipts by tossing the bottom of the bin, the manuals as I add new ones. This process is so efficient I don't see much room for improvement.

Airport Extreme 802.11n range: better at 2.4 than 5

The last time I messed with my home wireless network I jumped through hoops to connect an old 802.11b iBook (yes, the computer, not the app) to an early 2009 Time Machine. A few weeks after I bought that Time Machine Apple revved the line.

Now that I've replaced the Airport Express that now powers my mother's iPad with a new one, I'm sure Apple will rev it along with the upcoming iTV.

In the meantime, I've made one new discovery. In my home, 802.11N 2.4 GHz has significantly greater range than 802.11N 5 GHz. Apparently this is not uncommon.

When I configured the Express to WDS Extend it worked well in our dining room, but it couldn't connect in the living room. I brought it back to the dining room and manually switched the WDS setting from the original 5GHz network I'd selected to the standard 802.11N network [2]. (Time Machine broadcasts both plus 802.11G and 802.11B.) It then connected from the dining room. I repeated this to confirm it wasn't a chance glitch.

So our current (all wireless [1]) home network looks like this:

• Time Machine: 802.11 n/g/b with 802.11n 5GHz and 2.4Ghz
• Airport Express: Extends Time Machine, WDS connection is 802.11n 2.4 GHz
• iMac i5: 802.11n 5 GHz (it's about 10 feet directly below the Time Machine)
• iMac G5: 802.11g
• MacBook Intel: 802.11n 5GHz at the moment, but I bet I have to switch it back to 802.11N 2.4 GHz to get more range
• iPhone 4: 802.11n 2.4 GHz
• iPhone 3G and 3GS: 802.11g
• iPhone 3 (used as iTouch): 802.11g
• iBook: would be 802.11b but I've retired it. It wasn't getting any use in the iPhone era.

Note Apples says the Airport Express will connect up to 10 clients, and the Airport Extreme will connect up to 50 users (I'm not sure where my older Time Machine sits). With our iPhones and guest devices it's not hard to get near to the Express limits.

See also:

• Airport Extreme: Firmware update, Time Capsule bugs, tips and fixes
• Fixing a broken WDS wired to wireless bridge on an Apple Airport Express

[1] I thought the "all wireless" setup would be temporary following some home reconfiguration, but it's worked so well I've stayed with it. I'm surprised.
[2] From Airport Utility select Airport Express. Choose Manual setup, then Wireless. Select the non-5GHz or 5GHz through Wireless Network Name.

--My Google Reader Shared items (feed)

The hidden ever growing iTunes photo cache bug

There's a design flaw/bug in iTunes. When it syncs photos to an iPod or iOS device it creates a photo cache folder. That folder never shrinks, it only grows.

That's merely annoying on the desktop, but it does the same thing in iPhoto. To purge the cache you have to delete it, which requires opening the iPhoto image Package..;

iTunes: Photo sync creates iPod Photo Cache folder

... Control-click the iPhoto Library file and choose Show Package Contents from the shortcut menu.

In the iPhoto Library window, locate the iPod Photo Cache folder and drag it to the Trash...

My hidden iPhoto cache was 1.5 GB.

How exceedingly annoying.