Migrating images from SmugMug to Google's Picasa image store - Lessons in data lock and business models

Life is fractal. That is, sampled at scales small and large, it resembles itself. Sometimes I think this is profound, especially when looking at smoke rings of galaxies, but maybe I'm just having flashbacks.

So you don't need to spend 15 years in healthcare IT to understand why progress is slow. You only need experience migrating an address book between a PalmOS device to a Windows platform to OS X to Google Contacts (see also: synchronization is hell). That will tell you everything you need to know.

Similarly you don't need to be a corporate CEO to understand the strategic role of switching costs. It's very easy to switch between Subaru and Honda, not so easy to change checking accounts or move from Microsoft Office to Google Docs. Business strategies differ [2]; though this is never discussed in the sycophantic business book business. You only need to look at moving photo libraries from SmugMug to Picasa Web Albums [1] to learn this.

I am doing this now, and it's an interesting process. As expected, the switching costs are high. In most cases it's not worth the bother; if you have the original collections in iPhoto or a similar app it's best to abandon the old image libraries and start over. I'll explain first what you can move and what you can't, and what the implications are.

What you can move

• SmugMug will bundle all images into a single archive, hosted on Amazon's S3 servers, and allow download. This is commendable, and better than many vendors offer.

What you can't export from SmugMug (what you lose)

• Original file names.
• All metadata other than what's embedded in EXIF files (so you do get the image acquisition dates).
• Any titles, comments, notes.

This data lock story has interesting implications:

1. If you have the original albums/libraries in iPhoto/Aperture then you usually don't want to bother migrating albums. (In my case I have them, but for a few albums it's marginally more convenient for me to download and upload again even without metadata.
2. SmugMug is minimally useful for offsite image backup. Images without metadata are better than nothing, but you should not consider SmugMug to be any kind of alternative to image backup.
3. If you ever edit annotations online, then you deepen your data lock. Nobody, not even Google, allows export of this kind of metadata (likes, commentary, etc.). The business advantage of  "social," and "Cloud services" comes from the high switching costs created by data lock.

Fortune 500 corporation or photo hobbyist, fractal life means lessons are easy to learn.

[1] Why am I moving? SmugMug decided years ago not to invest in OS X support. I gave them a few years, but then I reluctantly migrated my newer images to Picasa web albums, which had significantly better OS X (and later mobile) support. Now, in the interests of simplifying my online life, I am consolidating and giving up my old collections.

FWIW, here's why I would start with Picasa today, though a professional photographer would certainly prefer SmugMug. My decision has nothing to do with price, SmugMug is $30 a year and if I promoted SmugMug I'd probably pay nothing (referral fees). In fact, since Google charges differently for storage, they might be marginally more costly.

1. Google's data freedom policy (data liberation team) is enough by itself to make my decision. Google supports full data sync, including metadata, to a cross platform app. No other vendor does this.
2. SmugMug's lack of support for metadata export means it's not a supplemental backup store. Google's data freedom means it is.
3. SmugMug has wisely focused on the professional photographer, but that means most of their services aren't useful for me.
4. SmugMug is vastly better for printing images, photo merchandise etc. I print about 3 images a year.
5. Picasa has far better OS X integration, though Apple is no help here.
6. Google's image sharing interface is easier for my mother to use.
7. I need to simplify my technological life. That means Google and Apple for me -- the right balance of coopetition. Between the two of them I get what I need.
8. I am very annoyed that SmugMug auto-renews my account, and offers only immediate cancellation rather than a non-renewal option.

[2] Companies with captive customers face terrible temptations, companies with mobile customers have different temptations. Think of the relationship between divorce laws and the status of women.

Update 3/26/11: I downloaded a few albums, but it was pretty tedious to connect what I downloaded to the album names shows in SmugMug. I have the originals, so I focused on a few albums for a group where I probably didn't keep every original. The rest I vaporized. One less vendor to deal with.

See also:

• Gordon's Notes: smugmug review: full res image sharing and photo printing (4/2004 - when I first became a customer. Back then my tech posts were in Gordon's Notes.)
• Gordon's Tech: Smugmug: just use sRGB and be happy (1/2006). Today we're still struggling with color profiles, but I with Windows 7, IE 9, Chrome, Safari and 10.6 maybe we're slowly getting to what we expected around 1998.)
• Gordon's Tech: SmugMug + PictureSync vs. Google Picasa Web Albums (9/2006). I thought SmugMug's future was limited. Wrong -- they seem healthy still! They found a paying market in professional photographers, and they took full advantage of high switching costs.
• Gordon's Notes: The rise of software rental (aka software as service) (2/2010). Another perspective on switching costs.
• Gordon's Notes: Cloud data: Should I trust (Simplenote) Simperium? (8/2010). Low switching costs make Simperium trustworthy if you assume all the data is public. Low switching costs make for honest and relatively ethical companies. This is, incidentally, why geeks love Dropbox.
• Gordon's Notes: Google has a Data Liberation team (3/2009). More than anything Google says, and more than anything else they have ever done, this has been their most important non-evil act.
• Gordon's Tech: Evernote's import/export test (updated) (7/2008). Evernote is better now, but I have not returned to study them. For them the data lock temptation must be irresistible.
• Gordon's Tech: data lock: All of my Gordon's Tech 'data lock' tagged posts.
• Gordon's Notes: data lock: All of my Gordon's Notes 'data lock' tagged posts. (Yes, I think data lock is interesting!)

Why cropping an 18 megapixel image isn't as good as an adequate telephoto

An 18 megapixel image has twice the data as an 9 megapixel image.

So does that mean a 135 mm telephoto and an 18 megapixel sensor can do as well as a 270mm telephoto and a 9 megapixel image?

No.

Why not?

Because when you crop the 18 megapixel image, and double the apparent size, you also double the image noise.

Blogger's text editor: I make another plea for a rational approach to paragraph formatting

It took considerable restraint for me to write this so politely.

Please stop using div tags to define paragraphs - Blogger Help

My blogs start around 2002, when Blogger used carriage returns to delimit paragraphs. They extend through the time that Blogger used <br> tags with or without hidden carriage returns.

These days, with either the "new" or legacy editor, it's unclear to me what Blogger does. I only know that when I use MarsEdit, Ecto or the native editor I see pairs of div tags. This causes a mess. Depending on which editor I use paragraphs vanish or spacing doubles. If I open older posts with the editors I get no paragraph spacing.

If I go into preferences I see:

"If Yes is selected, single hard-returns entered in the Post Editor will be replaced with single <br /> tags in your blog, and two hard-returns will be replaced with two tags (<br /><br />))."

Of course br tags are no longer used, so this language is simply wrong. If set it to "No" my old posts format incorrectly. To put it mildly, this is a mess.

I would like Blogger to use tags to, you know, delimit paragraphs. I would like Blogger to convert older posts to use <p> tags. Please Blogger, stop using <div> and <br> to define paragraphs, it's not what they're made for.

It's hopeless (yes, I've tried changing templates). Alas, I don't think there's really any way to move my blogs and keep any sort of reasonable formatting across 5,300 or so posts.

See also:

• Blogger's 3 year paragraph debacle - the case of universal line break conversion (10/2010). There have been years of complaints, until most everyone who cares has moved to WordPress. (Spammers don't care.)
• Gordon's Tech: Migrating from Blogger to WordPress - a guide

How I made Gmail and OS X 10.6 Mail.app into a grumpily married couple

There's no way to make Gmail and OS X Mail.app [1] into a happily married couple.

The Google Label (very shallow acyclic graph) vs. IMAP Folders (tree, single inheritance) models cannot be reconciled. If you use Gmail as an OS X Mail.app IMAP server, messages with multiple labels (Inbox, All Mail, Baseball, Work, etc) will be replicated in OS X. A single Gmail message may produce several OS X messages; Label-crazed geeks may find dozens of duplicates. Searches are a mess.

I'm happy to say, however, that I have found a way to make this dysfunctional couple settle down. I used How to make Gmail work well with Mail | Macworld as a guide. It claims too much, but the advice is still useful.

If you follow the articles advice, however, you will lose all your email archives from Mail.app; they will be only on Gmail. I inverted the advice. Using the advanced IMAP controls I turned off IMAP sync for all labels except for these four:

• Inbox
• Drafts
• All Mail
• Trash

This means that my tagging/organization pseudo-folders exist only on Google. None of that is exposed to OS X, the Google "labels" are gone [1]. On the other hand, I still have Spotlight Search and Smart Mailboxes, and my only duplicates are with All Mail and Inbox/Priority Inbox:

These duplicates go away when I move mail from Inbox to All Mail (archive).

Note I don't see the Sent folder (that would create All Mail/Sent folder dupes), but I have no use for the separate Sent folder (I can recreate an equivalent with Smart Search). I prefer the All Mail folder.  I don't bother exposing my Gmail Spam label/folder -- it's immense.

I followed most of the rest of the articles advice. Turns out I wasn't saving OS X Mail.app Sent messages on the server -- I thought I was.

With this configuration Mail.app looks like this:

It ain't a match made in heaven, but it's no longer Hell. Purgatory maybe. Maybe Apple will fix this in Lion?  [1]

[1] The fix is to stop treating Gmail tags as folders, and treat them as, you know, tags. Then support tags in Mail.app and use smart search to create additional derived folders. Who files email any more anyway? I stopped doing that years ago, and it's added years to my life. (By the way, I give a great lecture on email management.)

Update 3/17/2011: Alas, I can’t seem to eliminate the Sent folder. Even though I have the Label turned off on Gmail’s advanced IMAP control, it still produces dupes with the All mail folder. This is a more annoying problem than the small number of dupes in the All Mail folder. Maybe I need to move all my email from Gmail’s “Sent” folder to a new label.

Digital cameras plateau - from my 2005 XT to my 2011 T2i

I bought my Canon G2 in October 2002 [1] and my Digital Rebel XT in Nov 2005. A few days ago I picked up the Canon Digital Rebel T2i (550D) body for $650, about $150 less the minimally improved T3i (same sensor, same processor, some improvements for videographers).

I'm happy with my purchase, though it's not the camera I've been waiting for. I've been waiting for a 12 megapixel camera that could produce quality images at ISO 3200. Instead I got an 18 megapixel camera that can produce quality images at ISO 1600 and occasionally useful images at ISO 3200. I gave up when I saw the paltry improvements from the Canon T2i to the T3i. It feels like digital photography is nearing the limits of current imaging technology; I decided I wouldn't gain much by waiting another year. So my daughter got the XT and I got the T2i instead of the T3i.

Progress is nice, but there are worse things than living with a dSLR that can take good pictures at ISO 1600 and that, after I crop, effectively doubles my image stabilized zoom range. The technology plateau means I'll get more years from my purchase;  it might be 8-10 years before I switch to the MILC future. Although this technology halt isn't all bad for me, for Canon it must be very bad. For camera shops, who've struggling to stay alive in the post-film era [2], this may be a terminal arrest.

I won't bother with a full review of this 1 year old device; there are many good reviews of this camera, as usual Amazon's amateur reviews are among the best. I was surprised by how familiar it is; it feels very much like my six year old XT -- which is quite fine with me. I've never understood complaints that the Rebel body was "too small" -- I have large hands and it feels great to me. The shutter feels softer with less vibration. Of course the video is all new, but the manual focus in video mode makes it a poor fit for many settings [3].

As with the old XT, there are many features of the camera that are aimed at the "JPEG" photographer. Several of these are new, such as ways to manage dust spots, to adjust dynamic range, etc. Similarly there are several features for image management and printing from the camera. None of this interests me. I photograph RAW, review and edit in Aperture 3, then save JPEGs to iPhoto for archiving [4].

The new "Quick" menu and the "auto ISO" are good additions. I live having easy access to the ISO button. On the other hand, I'm seriously bummed that Cannon sacrificed my exposure "meter" control (weighted, average, etc) in favor of the 'picture style'  control. (For "Picture Style" I use Portrait, because it uses a 'level 2' sharpening and no other camera mods. I might drop to 'level 1' sharpening or Neutral/Faithful style -- let Aperture do the rest. Still experimenting.)

The biggest change, other than image quality and light sensitivity (yay) is the SD card. It works with SDHC, SDXC or SD. For video work a "class 6" card is required. I bought a Transcend Class 10 SDHC 16GB for about $24 or so. It is a fine size for me; 300 images use about half the card. Individual RAW images are about 28-30 MB each, the JPEG produced after cropping and processing is about 2.8 to 4.4 MB.

As to the images, they are rather fine. I don't think the T2i's ISO 1600 images are quite as good as the XT's ISO 400 images, but they are much better than the XT's ISO 800 images. The T2i's ISO 3200 images resemble the XT's ISO 800 -- meaning they're only for desperate times. I didn't try ISO 6400, those would have to be grayscale only (so an option, but not for color.) The images do stress my monitors; they don't look nearly as good on my old Dell as on my 27". Curious.

Aperture 3 on my 8GB iMac 5i does well with individual images; I think the GBs of RAM are coming in handy.

Good camera, I'm loving being able to take hockey pictures at 1/125!

[1] I boxed it up recently and left it, with a note at the office. It found a home within minutes!
[2] I visited National Camera with a friend today. The retail display included darkroom gear and a wide variety of picture frames; much of the floor space was film and print focused. 
[3] The T3i has a Canon G2 style flip out LCD, which is handy for tripod videography. If I do any video with this camera though, I'm likely to be handholding. There is one major point in the dSLR's favor though -- unlike the mad zoomers of today's dedicated camcorders the T2i has many affordable wide angle lenses. 
[4] Crazy eh? There's a method to this. If Apple ever provided a true upgrade path from iPhoto to Aperture I'd archive in Aperture, but I'd still archive JPEG and delete the RAW originals. I'm much more interested in image longevity than small quality improvements. After an initial adjustment in Aperture (esp. dynamic range) the JPEG gives me enough room to tweak as needed.

See also:

• Digital cameras 2011-2017 (12/2010 - my predictions. The MILC will rule the high end market by 2017).
• Gordon's Tech: Time to switch to Nikon? (1/2008)
• Gordon's Tech: Nikon has Canon in a headlock ... (1/2008)
• Gordon's Tech: New Canon Digital Rebel: light sensitivity yes! (2/2005 marketers claimed the 350D could shoot at ISO 1600. In reality, results about ISO 400 were very poor)
• Gordon's Tech: Gordon's Digital Rebel XT review (11/2005 - I bought the 350D)

eCamm PhoneView for iPhone and iPad - restore game data, extract contacts

I bought Camm PhoneView for iPhone and iPad (not on App Store) because I wanted to be able to save and restore game data if I needed to remove games from my kid's iOS devices. For example ...

Transferring Angry Birds saved game data to Mac app | Entertainment | Mac 911 | Macworld

...

3. Select Apps in PhoneView’s Data pane.

4. Enable the Show All Apps option.

5. Select Angry Birds and then select its Documents entry.

6. Drag the highscores.lua file to your Mac’s Desktop to make a copy...

You can also use it to get at media, notes and texting records.

Subsequently I received an email inquiry from someone who wanted to copy iPhone Contacts to the desktop. The problem was that these were Exchange Server contacts, so there's no supported way to get them off the phone to another device.

PhoneView allowed him to browse and copy his Contacts to OS X Address Book.

Safari isn't showing my frame contents

Chrome for OS X shows our old family news page as it has appeared for about ten years ...

On the right side of the page we see two "frames", each referencing a page which holds Javascript that renders Google Reader output. [1]

Recently, however, Safari shows this ...

The frames are gone. This HTML no longer works:

<IFRAME SRC="google_share.html" FRAMEBORDER=0 HEIGHT=300 WIDTH=185 TITLE="JGF Shared Posts">

I don't know whether this is a new Safari bug, or if it's a side-effect security measure of some sort. It sure is annoying.

[1] The reasons I did this are lost in the mists of time. I first put this page up around 1994. Sometime after 2000 I came up with the current scheme to display output from Bloglines, later I adopted it to Reader. A plain-jane web page on my site holds the Javascript that references Google's servers. The Frames reference the local pages. They're embedded in simple tables so they resize nicely. I'll be very annoyed if I have to change the design because of a Safari bug, less annoyed if there's a genuine security risk.

Update 3/6/11: I received a helpful response on Apple Discussions. Another 10.6 user can see the frames. On the other hand, I tried from the Apple Store and they didn't show up.

Update 3/7/11: The Frames are back again. I haven't updated OS X. I did, however, change my DNS provider from OpenDNS back to my ISP. OTOH, they only showed up now, and I did that yesterday. They've always worked in Chrome. This is truly odd.

The missing manuals - Spotlight doesn't search Library/Documentation

Apple doesn't always sweat the details.

OS X installs documentation into both Library/Documentation and Users/Library/Documentation. The former location, for example, includes my iMac Users Guide, "Welcome to Snow Leopard" and some help files.

Problem is, Spotlight doesn't index Library/Documentation. It only indexes items within the Users folder. So a Spotlight search won't find those documents.

This isn't new, but it's a detail Apple hasn't bothered with.

Tools for small business (NYT)

How to Make a Small Business Seem Much Bigger - Eileen Zimmerman -  NYTimes references these tools ....

• Batchbook: CRM
• MailChimp: email marketing
• Tungle.me: calendaring and scheduling [1]
• TimeTrade: appointment scheduling, marketing. Compatible with Google Calendar
• Intermedia: hosted Exchange server
• Shoeboxed: scanning service
• virtual employees: deductible expense rather than employee
• Regus: temporary office space, worldwide

[1] It tried Tungle. It looked interesting until I realized it only syncs with one my Google Calendars. My GC view is a synthesis of about 18 calendar sources (no joke). So sync with one doesn't help. Then I realized there's no way kill a tungle.me account. You have to email support -- and that's not well documented. Fail.

Time Capsule - using an external disk for network Time Machine backup

Time Machine and Time Capsule show Apple at its best and its worst.

Best, because only Apple has produced a backup solution that regular people can and will use. Apple pushes Time Machine (or Time Capsule) use -- for the benefit of Appel customers.

Worst because TM and TC are virtually undocumented, they are ill-suited for geek use (no encrypted offsite option), and they are somewhat unreliable [1].

Today I ran into examples of both, and learned two things:

1. If you attach an external drive to a Time Capsule, then Time Machine will treat it much like an internal TC drive. This is not documented, and at least in 2008 TM backup to an AirPort Extreme drive was not reliable and not supported.
2. You can't do a TM backup to USB attached drive, then move the drive to a Time Capsule and continue the backup over WiFi. This sucks.

I learned these things because the 500GB drive on my Time Capsule ran out of free space. I'd already excluded significant bits of my machines from TM backup [2] and reclaimed space so I had to try a different approach. I noticed that the barely used 2TB USB drive attached to my TC displays to TM as an acceptable backup option. Despite past issues, I decided to try using the external drive for my server/iMac and keep the internal drive for my our other Macs. [3]

First I brought the drive to my iMac, erased it with Disk Utility, and connected it as a regular TM managed drive. I then took it upstairs, connected it to the TC, and tried to resume the backup. That didn't work. It turns out TM handles a local disk differently from a TC hosted disk:

• Local disk: TM writes directly to the disk.
• Time Capsule disk (internal or external): TM first creates a sparse disk image, then writes to it.

So when I moved the disk to the TC, my iMac Time Machine.app just started a new (350GB) backup. One that would take days to complete, leaving with only one working backup method instead of my usual redundant backups [2].

The plus side of the sparse disk image approach is that I can use that TC mounted external drive as a file server as well as a TM backup, which is rather handy. That doesn't make up for the relocation hassle.

So now I have two options:

1. Allow TM to start the backup over WiFi. Then bring the disk downstairs. Mount the disk image locally, and see if TM will continue that backup.
2. Bring the TC with external drive to my iMac and backup via GB ethernet. That will compete in a single night, but it will mean we'll be without WiFi until I can put things back.

I'll update the post with which of these two work. I know the 2nd one will work ...

Update: Option 1 doesn't work. I don't have permissions to access the disk image. I could change permissions, but in my OS X experience this is a very risky thing. I could go with the ethernet option, but that's a PITA to move about. I'll try letting my backups chug along for a week or so. I'll start by excluding everything but the Users folder, then let TC gradually fill out the rest over time. If it's too slow, I'll do the ethernet thing.

Update 3/3/11: After about 30 hours TM backed up about 350GB to the WiFi via 802.11n. I never bothered with gB ethernet. The sparse bundle file was set originally at 600GB, even though I only backed up 300GB. I think the initial setting was based on ALL the data on my iMac drive rather; it didn't account for my initially excluding some folders. Later, when I included all data in the backup, it didn't take up any additional storage.

- fn -

[1] Of course I've never used a backup solution that was truly reliable. Even in its glory days Retrospect took some care and feeding.
[2] I also use SuperDuper to clone my primary machine/server nightly to an encrypted disk image and clone my MacBook every month or so. Two SD clones rotate offsite. So I use two completely different backup methods with my data including an offsite option. If Retrospect were to be miraculously resurrected I'd use that instead of SuperDuper. I've evaluated many other popular OS X backup options and failed them all.
[3] I left the TC disk image holding my iMac backup in place. I'll delete it later if all goes well.

See also:

• Freeing up Time Capsule space – and documentation for Time Machine and Time Capsule (12/2009): A pretty complete set of 2009 references, including ways to free up space.
• Gordon's Tech: Time Machine, Time Capsule and offsite backup (2/2010): Using encrypted disk images for offsite backup. I haven't tried this.
• Gordon's Tech: Time Machine - Unable to Complete Backup bug on Time Capsule (2/2010): fixes for an annoying bug
• Gordon's Tech: Time Machine: Fail first, then flaw discovered (12/2009: Permissions bug blocks backup)
• using FSevents for backup - is it reliable enough? (1/2011) - under some conditions a Time Machine backup will omit file versions or files. It's a deep bug.
• TidBITS Networking: Time Machine via AirPort Disk Is Unsupported, Apple Says (4/2008) ... "This option to choose an AirPort Extreme-connected drive first appeared with the release of several related firmware, driver, and operating system updates on 19-Mar-08 (see "AirPort Update Extends Time Capsule, Adds AirDisk Support," 2008-03-19). I speculated at the time that this addition was an error on Apple's part, perhaps due to a debugging feature left turned on that wasn't properly turned off before the updates shipped. This was buttressed in part by the way in which Time Capsule drives - whether an internal drive or ones connected externally via USB - appear via Bonjour in a list of selectable volumes when setting up Time Machine, but AirPort Extreme disks do not."