iCloud transition went as expected - disastrously

I was sure Apple would screw up my iCloud transition.

So I did everyone else on our family account first. I backed up my Contacts. I did a final MobileMe sync on all machines. I logged out of MobileMe on all but my Lion MacBook Air (fully updated).

Then I migrated. All seemed well until the Preference Pane opened. And crashed.

Address Book synchronization failed.

My Address Book now has one entry instead of 1,824 entries.

Apple, your competence is not complete.

Happily, I have backups.

I ended up disconnecting the Air from iCloud and moving my Contacts from my iPhone.

Did I mention that iCloud doesn't render well with Chrome?

And Apple thinks they can do Maps?

Update: 

• My old MacBook, barely able to run Lion, managed to synchronize its Address Book correctly (at least counts and samples match) to iCloud.
• I signed out of iCloud completely on the MacBook Air and logged out. When I logged in again my Address Book entries were back. Looks like they weren't removed, rather the sync failure left Address Book in an unhappy state. I logged out of iCloud completely then logged back in and that restarted iCloud synchronization. It appeared to complete successfully.
• You can't enter an admin un/pw to active Find My Mac, you have to activate it from an admin account (so same iCloud settings have to be active in two accounts.)
• On the plus side, the web version of Contacts is pretty good in Safari. I'll be using that until I convert my iMac to (yech) Lion. I may see if I can wait long enough to go to Mountain Lion directly.

Update 6/27/2012

• There are 1831 contacts on the iCloud site, 1828 on my MacBook Air and MacBook, 1831 in Snow Leopard (no longer synchronizing but at this point should still be in sync) and 1829 on my iPhone. I tried removing all contacts from my iPhone and then restoring from iCloud, that gave me 1828. So there are 3 on iCloud that aren't anywhere else. 
• My MacBook Air (the one that crashed on initial configuration) was hanging on iCloud login, so I tried restarting it in safe mode then tried connecting to iCloud again. I got a curious error message, it seemed as though the machine was in a mixed state part way between MobileMe and iCloud state. From the 'Mail, Contacts and Calendars'  Preference Pane I removed a residual mail account that seemed a 'mix' of MobileMe and iCloud. Then I learned that I was entering an incorrect password, which perhaps didn't fail as it should have because of a partial MobileMe connection. After this I was able to login to MobileMe easily, and I got a System Preferences Directory Services configuration modification notice. Subsequently my Address Book slowly updated without any UI indication of ongoing synchronization. Although my MacBook is now behaving better, it still has 1828 Contacts (Cards) compared to 1831 in iCloud. So those numbers aren't changing.
• After the initial synchronization, I realized updates were not propagating to my OS X Address Books. I unsubscribed from iCloud Contacts, removed all contacts, and resubscribe. At the moment they are updating. (See Apple's very popular iCloud Contacts troubleshooting article for more on this.)
• I'm trying Contact Cleaner to look for problems on one of my Lion machines.
• Looks like I have another update for my Synchronization is Hell blog post. MobileMe, over the past few months at least, was actually working reasonably well.
• Apple Discussion on this post.

Update 6/27/2012: Disaster diagnosed

It appears that each time iCloud synchronizes it duplicates the Notes in many of my Contacts. MobileMe and Snow Leopard Address Book didn't do this.

I've abandoned iCloud. I'm now synchronizing the same way I did my old PalmPilot -- by cable. It works quite well. After some Contact Cleanup I've 1821 contacts on Snow Leopard and 1821 on my phone.

I then copied my Snowie Address Book archive to a thumb drive and found that Lion would import it. Now I have 1821 contacts on my machines. These sync to iCloud still, I'll see how badly that goes and I'll experiment with dropbox distribution.

Update 6/27/2012: I find the extra 3, and a theory on the iCloud bug

I figured out how to find the 3 extra iCloud Contacts. I deleted all the contacts from an Address Book on one of my machines, then I looked at what was left in iCloud.com. There were 3 duplicate addresses left, all dupes of a single good address. So I removed them.

My theory on this bug is that it's one of the oldest bugs in the Mac world -- the CR/LF vs. CR vs. LF bug. Many of the entries in my Address Book started in Outlook on Windows, and moved over via MobileMe Windows. They probably have a line termination issue. This was ok with MobileMe and with Snow Leopard, but iCloud can't handle it.

I'm reasonably sure this is part of the problem, because when I move my Contacts to iCloud I see extra line spaces, even when the notes aren't duplicated.

As I wrote in my Apple Discussion post:

I think I've figured out the bug.

It's the oldest bug in the book -- line termination.

My Address Book contains contacts that started out in Outlook, then went via MobileMe Windows Control Panel to live in MobileMe, and then into my desktop machine.

MobileMe could handle the LF (Mac) vs CRLF (Windows) difference. So can Address Book in Lion and in Snow Leopard.

iCloud can't.

I'm seeing extra line spaces in all of my contacts with notes that started out life on Windows.

Each time iCloud does its sync with desktop it finds a mismatch -- because OS X and iCloud handle CR and CR/LF differently.

So it replicates the Note.

The Note grows eternally, until everything crumps.

Nasty bug! I hope Apple figures this one out soon.

There's an option to specify a text encoding on Address Book import. I'm experimenting with exporting an archive, then reimporting as UTF-8 rather than "automatic".

Additionally, I figured a way to crash iCloud Contacts reliably. Just try deleting 1800 contacts from the web UI.

Update 6/29/12: Ongoing experiments

• I tried importing an archive into Address Book using Western (Mac OS Roman) text encoding, but that didn't help.
• When I export a VCF file for the contacts that show extra line spaces, the vCard shows \n\n\n\n where, based on the Notes display, I'd expect \n\n.
• I installed a trial version of Bento, opened a contact and inspected the text in TextWrangler then pasted it back in from TW. I also migrated to a Bento Contacts view. Then I exported a vCard again and this time it only showed \n\n. The Card synchronized to iCloud and the extra line feeds disappeared there. 
• I get the same results just using Address Book, pasting into Text Wranger, then pasting back into Address Book. The round trip through Text Wrangler solves the EOL problem. I wonder if that's something I could write an Apple script to do.
• Mac OS X hints has the best information on doing this conversion. From 2003.
• I found a Service that I could enable in Service Preferences called "Macintosh Line Endings". It may have come with Devon Technologies WordService. Highlight Address Book then applying this service removed the Windows CRLF pairs and left Mac OS X LF line feeds. It fixed the problem. So if I can figure a way to walk my Contact Notes and apply this Service the problem will be fixed.
• iCloud updates are unpredictable. Sometimes fairly fast, sometimes I give up.

Update 6/30/2012: vCard options

i've decided there's an Uncanny Valley for software; products that look almost like real software, but don't quite make it. Bento and Automator are like that.

Bento didn't turn out to be very useful, but it showed me there were 1000s of notes to clean up. Too many to brute force, even after creating kb shortcuts for the "Macintosh Line Ending" WordService.

I looked at scripting Bento, but it's not scriptable. Address Book is very scriptable, but there didn't seem to be a way to walk through a list of addresses applying the Line Ending utility to each note in turn.

vCard is more promising though. I can use TextWrangler to change \n\n to \n and remove the dangling \n at the end of many vCard Note strings. On import then Address Book will turn the \n into a LF -- no more CRLF problem.

As expected, however, Apple's Address Book import is unable to process the %^$#! CATEGORY strings that Apple Address Book Export $^@%# writes! Grrr. #$#@!$% Apple.

I tried using the 'merge' function, but, like iCloud, that merely replicated the Notes. I'm looking at ways to put the CATEGORY strings into the Notes field (Append), so perhaps I can use smart groups to recreate my Group assignments. Maybe Stefan Kelley'rs Export Address Book utility will help. I need to think on this one.

Update 7/1/2012: Hex Editor won't help

It occurred to me that I could use a hex editor to try to fix the problem. Alas, .abbu is just a Pacakage, there's no single file I can try editing.

So I think I'll be doing the vCard export/import and I'm experimenting with ways to speed group membership recreation on import. Some useful references:

• vCard import group 9/2008
• Difficulty importing Categories into 2/2009
• HT4489 Importing vCards to iCloud from... 5/2012
• VCard grayed out when attempting to... 4/2012
• Custom fields in Mac OS X Address Book « Diaries, Triumphs, Failures, and Rants
• Address Book Power Tips | Mac|Life

Update 7/2/2012: Exploring more options, with a focus on somehow retaining Group membership

Lion's Address Book 6.0 has more import/export behaviors, including restoring 10.4's option-export individual vCard. That makes its continued inability import Categories as groups, or import "custom fields", all the more exasperating.

• Address Book 6.x: Resolve dulicate cards when importing cards: After first review can apply to all.
• Address Book 6.x: Import contacts
• Address Book 6.x: Export and import vCards

And LDIF is perhaps an option ...

• Importing an ldif address book?

Some potentially useful tips ...

• 5 handy Address Book tips -- Reality Distortion: Macs, Mac OS X, and Apple stuff
• Mac Address Book tips: 5 more handy hints
• Hawk Wings - Plug-ins for Address Book (and other) (there used to be an advanced importer plugin, but it appears to be gone - Address Book importer)

I suspect vCard import/export includes the Department field. I don't often use Department, so I could probably copy the Category strings into Department, and then use Smart Groups to create subsets from which to create Groups.

Update 7/3/2012: vCard export/import won't work, but a hybrid hack looks promising

Many fields are omitted from Address Book's vCard option. So I can't do clean up and restore from there, I'd lose too much. Worse, Address Book Import is also quite weak. Instead on a test machine running Lion and Address Book 6.0 I tried this convoluted process on a copy of my Snow Leopard Address Book.

1. I selected all contacts and exported Group vCard (1820 contacts)
2. I used Bento to delete all Notes (select column, hit delete, exit Bento)
3. Working on the vCard export, I used TextWrangler to replace \n\n with \n everywhere and remove a trailing \n at the end of the NOTES string. This removed some paragraph definitions but I didn't mind that. Simpler to do.
4. I then imported the vCard file back into Address Book. Address Book said it had found 1818 duplicates and merged those in; I ended up with 1822 Cards. So there were two duplicates.

The results seemed good, I no longer found duplicate notes. The Address Book synchronized to iCloud far more quickly than before, and iCloud also appeared fine and without duplicate Notes.

I considered using Address Book merge to resolve the two duplicate Cards, but it is too aggressive and would merge addresses I wish to keep separate. So I used Contacts Cleaner. It found 3 duplicates and a (new) 5-6 duplicate addresses. Those were quickly resolved and I ended up with 1819 cards, but iCloud did not update; it still showed 1822 Cards. I suspect that because Contact Cleaner is altering records "behind Address Book's back" that it doesn't know what to sync to iCloud.

Unfortunately Contacts Cleaner still refers to MobileMe in the UI and the App Store support link goes to a 1 page contact-free FAQ that is also the entire product documentation.

I suspect the only way to force iCloud updating is one outlined for a different product; Address Book Cleaner. It requires deletion of all contacts on iCloud to enable synchronization. Since I've shown previously that Safari will crash if you try to delete 1,800 Contacts this way, you have to do it using Address Book and some Archive shuffling (if you are reading this you are enough of a geek to figure that one out on your own).

This approach, though inelegant, appears to work. I'll do some more testing before I try it on my real Address Book. (I haven't tested yet to see if Snowie will open a Lion Address Book. I suspect not, but the process I used will probably work on Snowie. As long as one backups up the Archive, it's easy to restore an Address Book.

*OOPS* Late update: Even if I delete all entries, iCloud/Address Book now has large numbers of replicated Categories/Groups. This is quite comical!

Update 7/4/2012: Every iCloud client Address Book irrevocably poisoned by metastatic Groups

This was when I gave up. I'd solved my Notes duplication issue, and I'd gotten the Contacts Cleaned. I'd even figured out how to clear out iCloud (I thought!) and how to push changes from one machine to others. Then I noticed this:

That's a portion of the hundreds of replications of the Personal Group as shown in iCloud.com. I see the same list in every Address Book synchronized with this iCloud account, even though all the Contacts/Cards have been removed.

Wait, it gets worse. On these test machines I tried removing the entire contents of Library/Application Support/AddressBook. (I found I had to quite LaunchBar to empty the trash -- it holds onto this data. Which means LaunchBar is a minor suspect in this crime.) It didn't work; they came back (I could see them being added to Library/Application Support/AddressBook/Metadata as Address Book launched).  They are immortal and they're spreading to all of my machines. These iCloud Groups are the metastatic cancer of OS X Lion.

How can this be? Clearly they don't only live in Address Book. I suspect that after one syncs with iCloud, regardless of whatever one does later including signing out of iCloud, the group definition lives in iCloud. On a disconnected machine it's somewhere in an iCloud offline cache, and they get synchronized into Address Book. To delete them one would need to clean out the iCloud local data store. There's no known way to do that (no iCloud equivalent of  reset SyncServices).), just as, incidentally, there's no way to delete an iCloud.com account. 

Of course there are still options. Given enough time, I could delete them one at a time in Address book.app or iCloud Contacts. Maybe I could track down the cache and remove it. Maybe I could see if creating a different iCloud account removed them.

I could do a lot of things ... but ... really, I'm done. I've established to my own satisfaction that iCloud and Lion are broken.

I'm done. My Address Book is in Snow Leopard, and it works -- even with the EOL bug. I sync now to my iPhone via the iTunes cable. Maybe I'll sync them to a special Google Apps account as well -- using iTunes. One that's setup just to provide access to my Contacts.

I'll try again when Mountain Lion is out. (Update: Mountain Lion helped a bit, but AppleScript was the fix. See below.)

Update 7/4/2012b: More broken iCloud (I can't resist).

I have a mac.com Apple ID. For kicks I tried creating an iCloud account from it. Turns out you can -- except iCloud hangs. So you get part of an account I think. Did Apple outsource all their iCloud development? I mean, this thing is made of cheap crystal.

Also, when I did this OS X told me that this new account wasn't my primary iCloud account. So you might ask - how does one change or configure the primary iCloud account? It's not clear, but in some experimentation involving removing accounts, logging in and out, etc. I think I did.

One more discovery. In the iCloud world data really lives on the Cloud. My local data seems like a cached version of Cloud data. So, once you make the Cloud transition, Apple owns your Contacts. I don't see a way to get them back. I have noticed that the Address Book archives are bound to a .me account, not to my machine.

Update 7/4/2012c: Postscript, in which all 1819 contacts move to a new iCloud account with the EOL problem fixed.

I couldn't quite leave it alone. In further experiments I discovered:

• Sometimes my Address Book 6.x Import option would be grayed out. Deleting everything in AddressBook fixed this. Seems to be a permissions problem related to iCloud.
• It's iCloud that's writing the Groups back into Address Book. Indeed, I discovered when iCloud is active it's impossible to truly delete all files from AddressBook. It grabs hold of some on startup. When all iCloud accounts are moved both from System Preferences and Address Book Preferences you can delete the files.
• You can drag and drop Contacts from "iCloud" to "On My Mac" and create a local set -- but you can't move or copy Groups. Groups live only on iCloud. Makes you wonder what Apple wants to do with groups.
• I was able to make my mac.com account my Primary iCloud account. It told me if I wanted email to work I'd have to pick a me.com email address. I ignored that.
• I deleted everything from iCloud on my test Mac (I've figured out how to do that). Then I copied my Address Book archive from Snowie. Then I went through the  EOL cleanup process I described above. Then I set up an iCloud connection to my mac.com identity. This time it moved everything over (so what I see on my Mac is only a cached version). With the EOL bug fixed I had no duplicate notes and no metastatic Groups, and the same count on both.

And now I wait for Mountain Lion. At least I know the EOL fix works; I'll be sure to test that before I try Mountain Lion. At some point I may take my machine to an Apple "Genius" and see if they can get the original iCloud account cleaned up.

Update 7/26/2012: I fixed the Group Replication problem - deleted the Groups with AppleScript.

Update 7/27/2012: Nigel Garvey of MacScripter writes an AppleScript to fix the line terminations (!)