Chrome broken in OS X due to certificate change: Issue 108238 - Canary asks for access to every keychain item

I got hit with this bug on my Air: Issue 108238 - chromium - Canary asks for access to every keychain item.

We had to change the certificate we use to sign Chrome.

Transition code was included in (stable channel) Chrome 19.0.1084.53 and 19.0.1084.54 to migrate your Keychain items. If you’re on the stable channel but did not launch one of these two versions, upon update to 19.0.1084.56, your Keychain items were “lost” and need to be reauthorized manually.

There is a mitigation in place for 19.0.1084.56 and newer: when Chrome automatically updates itself, the updater will perform the transition. Unfortunately, this is only effective if the updater has permission to do this, and that’s generally only the case when you’re on a “user ticket,” and not when “Set Up Automatic Updates for All Users” has been used.

The correct thing to do when such a dialog appears is to click “Always Allow”. We don’t anticipate any further Keychain changes like this for a very long time, and we hope that in the future, if we do need to make such changes, to have a longer time in which to let the transition run.

It's an interaction bug between Chrome Sync, OS X Keychain, Lion, and a certificate signing change with a short overlap period:

The at-update reauthorization is intended to handle the reauthorization for users who rarely restart Chrome and might miss out on the at-launch step during the window where Chrome is signed by the old certificate but has the new reauthorization code in place.

It hit a machine I rarely use where Chrome was probably running, so when I restarted it the old cert had expired.

There's no plausible workaround, the Google support forum thread on this topic has quite a few deleted messages. The fix seems to be to switch to Safari or Firefox. In a case like mine, where there are hundreds or even thousands of passwords in the Keychain, tapping "Always Allow" repeatedly can take a very long time. Chrome is unusable for now on my Air.

I'm curious as to why Google made this fairly urgent change to their Chrome code signing cert. My Google searches didn't turn anything up. I assume it's related to a significant security breach.  The silence is curious.

Update: Since this was not my main machine I applied the drastic "fix" of deleting my entire #$#$ user login keychain because Google was supposed to 'regenerate it'. Which it sort of did, but the process turned into a slog of bugs and issues far too tedious to list (including a password displayed in a font that rendered a zero as an o. Even after the Google sync was done I am still out many passwords on that machine that are used by other apps. I really don't recommend this approach. It's easier to switch to Firefox or Safari.

So far tonight Google and Apple are causing me similar amounts of pain.

iCloud transition went as expected - disastrously

I was sure Apple would screw up my iCloud transition.

So I did everyone else on our family account first. I backed up my Contacts. I did a final MobileMe sync on all machines. I logged out of MobileMe on all but my Lion MacBook Air (fully updated).

Then I migrated. All seemed well until the Preference Pane opened. And crashed.

Address Book synchronization failed.

My Address Book now has one entry instead of 1,824 entries.

Apple, your competence is not complete.

Happily, I have backups.

I ended up disconnecting the Air from iCloud and moving my Contacts from my iPhone.

Did I mention that iCloud doesn't render well with Chrome?

And Apple thinks they can do Maps?

Update: 

• My old MacBook, barely able to run Lion, managed to synchronize its Address Book correctly (at least counts and samples match) to iCloud.
• I signed out of iCloud completely on the MacBook Air and logged out. When I logged in again my Address Book entries were back. Looks like they weren't removed, rather the sync failure left Address Book in an unhappy state. I logged out of iCloud completely then logged back in and that restarted iCloud synchronization. It appeared to complete successfully.
• You can't enter an admin un/pw to active Find My Mac, you have to activate it from an admin account (so same iCloud settings have to be active in two accounts.)
• On the plus side, the web version of Contacts is pretty good in Safari. I'll be using that until I convert my iMac to (yech) Lion. I may see if I can wait long enough to go to Mountain Lion directly.

Update 6/27/2012

• There are 1831 contacts on the iCloud site, 1828 on my MacBook Air and MacBook, 1831 in Snow Leopard (no longer synchronizing but at this point should still be in sync) and 1829 on my iPhone. I tried removing all contacts from my iPhone and then restoring from iCloud, that gave me 1828. So there are 3 on iCloud that aren't anywhere else. 
• My MacBook Air (the one that crashed on initial configuration) was hanging on iCloud login, so I tried restarting it in safe mode then tried connecting to iCloud again. I got a curious error message, it seemed as though the machine was in a mixed state part way between MobileMe and iCloud state. From the 'Mail, Contacts and Calendars'  Preference Pane I removed a residual mail account that seemed a 'mix' of MobileMe and iCloud. Then I learned that I was entering an incorrect password, which perhaps didn't fail as it should have because of a partial MobileMe connection. After this I was able to login to MobileMe easily, and I got a System Preferences Directory Services configuration modification notice. Subsequently my Address Book slowly updated without any UI indication of ongoing synchronization. Although my MacBook is now behaving better, it still has 1828 Contacts (Cards) compared to 1831 in iCloud. So those numbers aren't changing.
• After the initial synchronization, I realized updates were not propagating to my OS X Address Books. I unsubscribed from iCloud Contacts, removed all contacts, and resubscribe. At the moment they are updating. (See Apple's very popular iCloud Contacts troubleshooting article for more on this.)
• I'm trying Contact Cleaner to look for problems on one of my Lion machines.
• Looks like I have another update for my Synchronization is Hell blog post. MobileMe, over the past few months at least, was actually working reasonably well.
• Apple Discussion on this post.

Update 6/27/2012: Disaster diagnosed

It appears that each time iCloud synchronizes it duplicates the Notes in many of my Contacts. MobileMe and Snow Leopard Address Book didn't do this.

I've abandoned iCloud. I'm now synchronizing the same way I did my old PalmPilot -- by cable. It works quite well. After some Contact Cleanup I've 1821 contacts on Snow Leopard and 1821 on my phone.

I then copied my Snowie Address Book archive to a thumb drive and found that Lion would import it. Now I have 1821 contacts on my machines. These sync to iCloud still, I'll see how badly that goes and I'll experiment with dropbox distribution.

Update 6/27/2012: I find the extra 3, and a theory on the iCloud bug

I figured out how to find the 3 extra iCloud Contacts. I deleted all the contacts from an Address Book on one of my machines, then I looked at what was left in iCloud.com. There were 3 duplicate addresses left, all dupes of a single good address. So I removed them.

My theory on this bug is that it's one of the oldest bugs in the Mac world -- the CR/LF vs. CR vs. LF bug. Many of the entries in my Address Book started in Outlook on Windows, and moved over via MobileMe Windows. They probably have a line termination issue. This was ok with MobileMe and with Snow Leopard, but iCloud can't handle it.

I'm reasonably sure this is part of the problem, because when I move my Contacts to iCloud I see extra line spaces, even when the notes aren't duplicated.

As I wrote in my Apple Discussion post:

I think I've figured out the bug.

It's the oldest bug in the book -- line termination.

My Address Book contains contacts that started out in Outlook, then went via MobileMe Windows Control Panel to live in MobileMe, and then into my desktop machine.

MobileMe could handle the LF (Mac) vs CRLF (Windows) difference. So can Address Book in Lion and in Snow Leopard.

iCloud can't.

I'm seeing extra line spaces in all of my contacts with notes that started out life on Windows.

Each time iCloud does its sync with desktop it finds a mismatch -- because OS X and iCloud handle CR and CR/LF differently.

So it replicates the Note.

The Note grows eternally, until everything crumps.

Nasty bug! I hope Apple figures this one out soon.

There's an option to specify a text encoding on Address Book import. I'm experimenting with exporting an archive, then reimporting as UTF-8 rather than "automatic".

Additionally, I figured a way to crash iCloud Contacts reliably. Just try deleting 1800 contacts from the web UI.

Update 6/29/12: Ongoing experiments

• I tried importing an archive into Address Book using Western (Mac OS Roman) text encoding, but that didn't help.
• When I export a VCF file for the contacts that show extra line spaces, the vCard shows \n\n\n\n where, based on the Notes display, I'd expect \n\n.
• I installed a trial version of Bento, opened a contact and inspected the text in TextWrangler then pasted it back in from TW. I also migrated to a Bento Contacts view. Then I exported a vCard again and this time it only showed \n\n. The Card synchronized to iCloud and the extra line feeds disappeared there. 
• I get the same results just using Address Book, pasting into Text Wranger, then pasting back into Address Book. The round trip through Text Wrangler solves the EOL problem. I wonder if that's something I could write an Apple script to do.
• Mac OS X hints has the best information on doing this conversion. From 2003.
• I found a Service that I could enable in Service Preferences called "Macintosh Line Endings". It may have come with Devon Technologies WordService. Highlight Address Book then applying this service removed the Windows CRLF pairs and left Mac OS X LF line feeds. It fixed the problem. So if I can figure a way to walk my Contact Notes and apply this Service the problem will be fixed.
• iCloud updates are unpredictable. Sometimes fairly fast, sometimes I give up.

Update 6/30/2012: vCard options

i've decided there's an Uncanny Valley for software; products that look almost like real software, but don't quite make it. Bento and Automator are like that.

Bento didn't turn out to be very useful, but it showed me there were 1000s of notes to clean up. Too many to brute force, even after creating kb shortcuts for the "Macintosh Line Ending" WordService.

I looked at scripting Bento, but it's not scriptable. Address Book is very scriptable, but there didn't seem to be a way to walk through a list of addresses applying the Line Ending utility to each note in turn.

vCard is more promising though. I can use TextWrangler to change \n\n to \n and remove the dangling \n at the end of many vCard Note strings. On import then Address Book will turn the \n into a LF -- no more CRLF problem.

As expected, however, Apple's Address Book import is unable to process the %^$#! CATEGORY strings that Apple Address Book Export $^@%# writes! Grrr. #$#@!$% Apple.

I tried using the 'merge' function, but, like iCloud, that merely replicated the Notes. I'm looking at ways to put the CATEGORY strings into the Notes field (Append), so perhaps I can use smart groups to recreate my Group assignments. Maybe Stefan Kelley'rs Export Address Book utility will help. I need to think on this one.

Update 7/1/2012: Hex Editor won't help

It occurred to me that I could use a hex editor to try to fix the problem. Alas, .abbu is just a Pacakage, there's no single file I can try editing.

So I think I'll be doing the vCard export/import and I'm experimenting with ways to speed group membership recreation on import. Some useful references:

• vCard import group 9/2008
• Difficulty importing Categories into 2/2009
• HT4489 Importing vCards to iCloud from... 5/2012
• VCard grayed out when attempting to... 4/2012
• Custom fields in Mac OS X Address Book « Diaries, Triumphs, Failures, and Rants
• Address Book Power Tips | Mac|Life

Update 7/2/2012: Exploring more options, with a focus on somehow retaining Group membership

Lion's Address Book 6.0 has more import/export behaviors, including restoring 10.4's option-export individual vCard. That makes its continued inability import Categories as groups, or import "custom fields", all the more exasperating.

• Address Book 6.x: Resolve dulicate cards when importing cards: After first review can apply to all.
• Address Book 6.x: Import contacts
• Address Book 6.x: Export and import vCards

And LDIF is perhaps an option ...

• Importing an ldif address book?

Some potentially useful tips ...

• 5 handy Address Book tips -- Reality Distortion: Macs, Mac OS X, and Apple stuff
• Mac Address Book tips: 5 more handy hints
• Hawk Wings - Plug-ins for Address Book (and other) (there used to be an advanced importer plugin, but it appears to be gone - Address Book importer)

I suspect vCard import/export includes the Department field. I don't often use Department, so I could probably copy the Category strings into Department, and then use Smart Groups to create subsets from which to create Groups.

Update 7/3/2012: vCard export/import won't work, but a hybrid hack looks promising

Many fields are omitted from Address Book's vCard option. So I can't do clean up and restore from there, I'd lose too much. Worse, Address Book Import is also quite weak. Instead on a test machine running Lion and Address Book 6.0 I tried this convoluted process on a copy of my Snow Leopard Address Book.

1. I selected all contacts and exported Group vCard (1820 contacts)
2. I used Bento to delete all Notes (select column, hit delete, exit Bento)
3. Working on the vCard export, I used TextWrangler to replace \n\n with \n everywhere and remove a trailing \n at the end of the NOTES string. This removed some paragraph definitions but I didn't mind that. Simpler to do.
4. I then imported the vCard file back into Address Book. Address Book said it had found 1818 duplicates and merged those in; I ended up with 1822 Cards. So there were two duplicates.

The results seemed good, I no longer found duplicate notes. The Address Book synchronized to iCloud far more quickly than before, and iCloud also appeared fine and without duplicate Notes.

I considered using Address Book merge to resolve the two duplicate Cards, but it is too aggressive and would merge addresses I wish to keep separate. So I used Contacts Cleaner. It found 3 duplicates and a (new) 5-6 duplicate addresses. Those were quickly resolved and I ended up with 1819 cards, but iCloud did not update; it still showed 1822 Cards. I suspect that because Contact Cleaner is altering records "behind Address Book's back" that it doesn't know what to sync to iCloud.

Unfortunately Contacts Cleaner still refers to MobileMe in the UI and the App Store support link goes to a 1 page contact-free FAQ that is also the entire product documentation.

I suspect the only way to force iCloud updating is one outlined for a different product; Address Book Cleaner. It requires deletion of all contacts on iCloud to enable synchronization. Since I've shown previously that Safari will crash if you try to delete 1,800 Contacts this way, you have to do it using Address Book and some Archive shuffling (if you are reading this you are enough of a geek to figure that one out on your own).

This approach, though inelegant, appears to work. I'll do some more testing before I try it on my real Address Book. (I haven't tested yet to see if Snowie will open a Lion Address Book. I suspect not, but the process I used will probably work on Snowie. As long as one backups up the Archive, it's easy to restore an Address Book.

*OOPS* Late update: Even if I delete all entries, iCloud/Address Book now has large numbers of replicated Categories/Groups. This is quite comical!

Update 7/4/2012: Every iCloud client Address Book irrevocably poisoned by metastatic Groups

This was when I gave up. I'd solved my Notes duplication issue, and I'd gotten the Contacts Cleaned. I'd even figured out how to clear out iCloud (I thought!) and how to push changes from one machine to others. Then I noticed this:

That's a portion of the hundreds of replications of the Personal Group as shown in iCloud.com. I see the same list in every Address Book synchronized with this iCloud account, even though all the Contacts/Cards have been removed.

Wait, it gets worse. On these test machines I tried removing the entire contents of Library/Application Support/AddressBook. (I found I had to quite LaunchBar to empty the trash -- it holds onto this data. Which means LaunchBar is a minor suspect in this crime.) It didn't work; they came back (I could see them being added to Library/Application Support/AddressBook/Metadata as Address Book launched).  They are immortal and they're spreading to all of my machines. These iCloud Groups are the metastatic cancer of OS X Lion.

How can this be? Clearly they don't only live in Address Book. I suspect that after one syncs with iCloud, regardless of whatever one does later including signing out of iCloud, the group definition lives in iCloud. On a disconnected machine it's somewhere in an iCloud offline cache, and they get synchronized into Address Book. To delete them one would need to clean out the iCloud local data store. There's no known way to do that (no iCloud equivalent of  reset SyncServices).), just as, incidentally, there's no way to delete an iCloud.com account. 

Of course there are still options. Given enough time, I could delete them one at a time in Address book.app or iCloud Contacts. Maybe I could track down the cache and remove it. Maybe I could see if creating a different iCloud account removed them.

I could do a lot of things ... but ... really, I'm done. I've established to my own satisfaction that iCloud and Lion are broken.

I'm done. My Address Book is in Snow Leopard, and it works -- even with the EOL bug. I sync now to my iPhone via the iTunes cable. Maybe I'll sync them to a special Google Apps account as well -- using iTunes. One that's setup just to provide access to my Contacts.

I'll try again when Mountain Lion is out. (Update: Mountain Lion helped a bit, but AppleScript was the fix. See below.)

Update 7/4/2012b: More broken iCloud (I can't resist).

I have a mac.com Apple ID. For kicks I tried creating an iCloud account from it. Turns out you can -- except iCloud hangs. So you get part of an account I think. Did Apple outsource all their iCloud development? I mean, this thing is made of cheap crystal.

Also, when I did this OS X told me that this new account wasn't my primary iCloud account. So you might ask - how does one change or configure the primary iCloud account? It's not clear, but in some experimentation involving removing accounts, logging in and out, etc. I think I did.

One more discovery. In the iCloud world data really lives on the Cloud. My local data seems like a cached version of Cloud data. So, once you make the Cloud transition, Apple owns your Contacts. I don't see a way to get them back. I have noticed that the Address Book archives are bound to a .me account, not to my machine.

Update 7/4/2012c: Postscript, in which all 1819 contacts move to a new iCloud account with the EOL problem fixed.

I couldn't quite leave it alone. In further experiments I discovered:

• Sometimes my Address Book 6.x Import option would be grayed out. Deleting everything in AddressBook fixed this. Seems to be a permissions problem related to iCloud.
• It's iCloud that's writing the Groups back into Address Book. Indeed, I discovered when iCloud is active it's impossible to truly delete all files from AddressBook. It grabs hold of some on startup. When all iCloud accounts are moved both from System Preferences and Address Book Preferences you can delete the files.
• You can drag and drop Contacts from "iCloud" to "On My Mac" and create a local set -- but you can't move or copy Groups. Groups live only on iCloud. Makes you wonder what Apple wants to do with groups.
• I was able to make my mac.com account my Primary iCloud account. It told me if I wanted email to work I'd have to pick a me.com email address. I ignored that.
• I deleted everything from iCloud on my test Mac (I've figured out how to do that). Then I copied my Address Book archive from Snowie. Then I went through the  EOL cleanup process I described above. Then I set up an iCloud connection to my mac.com identity. This time it moved everything over (so what I see on my Mac is only a cached version). With the EOL bug fixed I had no duplicate notes and no metastatic Groups, and the same count on both.

And now I wait for Mountain Lion. At least I know the EOL fix works; I'll be sure to test that before I try Mountain Lion. At some point I may take my machine to an Apple "Genius" and see if they can get the original iCloud account cleaned up.

Update 7/26/2012: I fixed the Group Replication problem - deleted the Groups with AppleScript.

Update 7/27/2012: Nigel Garvey of MacScripter writes an AppleScript to fix the line terminations (!)

NOOK for Mac - Barnes and Noble is finished

I finally bought my copy of Redshirts. I got it at the iBookstore because I prefer the ePub format, and since Redshirts is DRM-free (would have bought the paperback otherwise) I can, in theory, open it in OS X ePub readers.

Alas, there are no great OS X ePub readers. A SuperUser reader suggested NOOK for Mac, so I tried it. 

Splashed open and then ...

Nothing. It's stuck at the splash screen. Here's the log file:

Platform:Mac OS X v.10.6.8
NOOK for Mac build number:3.0.0fc9774
user:

application location:/Applications/NookForMac.app
system clock speed:2147 MHz
system RAM:8192 MB
system file size:953541 MB
system free size:509230 MB
log contents:

=================== BEGIN LOG ===================
2012-06-22 08:17:23.00358718 ERROR [ClientApi.cpp:60] ==============================================
2012-06-22 08:17:23.00386206 ERROR [SQLiteDatabase.cpp:1455] select schema_version from bn_schema_version where id = 0 ... fail
2012-06-22 08:17:23.00432055 ERROR [SQLiteDatabase.cpp:1426] sqlite3_exec ERROR: duplicate column name: lend_message
2012-06-22 08:17:23.00459153 ERROR [SQLiteDatabase.cpp:1426] sqlite3_exec ERROR: cannot commit - no transaction is active
2012-06-22 08:17:23.00459185 ERROR [SQLiteDatabase.cpp:1426] sqlite3_exec ERROR: cannot commit - no transaction is active
2012-06-22 08:18:16.00984787 ERROR [ClientApi.cpp:60] ==============================================

=================== END LOG ===================

I got the same thing from my Admin account. I'm using Snowie, B&N's web site says: "Mac OS X 10.6 or later".

B&N's Community site has reports of this problem from 9/2011. A developer wrote:

In the /var/log/system.log, I see this error:
Mar 1 22:24:17 ISC-krhodes BNeReader[77695]: wrote:<?xml version="1.0" encoding="utf-8" ?>^M\n<eReader>^M\n</eReader>^M\n to path:/Users/keithrhodes/Library/Application Support/Barnes & Noble/BNDesktopReader/config.xml
Mar 1 22:24:17 ISC-krhodes BNeReader[77695]: BNeReader(77695,0xa0c54540) malloc: *** error for object 0x1512f40: double free\n*** set a breakpoint in malloc_error_break to debug

There's a workaround if you're desperate: "Once the app fails at the start screen, right-click anywhere on the grey area and click on reload. This will get you to your library."

This is not a serious piece of software. When companies ship mock software, the end is in the rear view mirror.

Google supports sharing MULTIPLE Calendars in single embedded calendar web view

I was visiting the S&I Framework embedded Google Calendar today and I saw this:

 

Huh?! I didn't know it was possible to create an embedded web calendar with a multiple calendar display options!

Turns out this is not a new feature, I found a 2008 article on it. The UI is seriously weird, which is why I've somehow blocked it out.

When you customize the Google code for embedded calendars the customization screen lists, on bottom left somewhat out of sight, all your Google Calendars. You can check the box to include multiple calendars.

What's weird about that? Well, as far as I know, the only way to get to this UI is through the calendar setting control for a single calendar. So you view settings for a single calendar, then click customize embedded code, and you can share multiple. That's a hack.

Once I knew how to do it I could find the official documentation:

Embed on your website - Google Calendar Help

... To display events from multiple calendars in the same embedded calendar, simply follow the instructions above to access the Google Embeddable Calendar Helper. Then, select the checkbox next to each calendar you wish to display, in the section Calendars to Display...

Am I the only geek who didn't know this was possible?

Pinboard and IFTTT - credential outage

I was quite pleased with my jury rigged replacement for my long lost Google Reader Shares. (Few remember them now, the net years have passed like leaves in the wind ...)

I had IFTTT tracking Pinboard posts and creating Tweets and posts to an archival WordPress blog. The Pinboard posts were largely created from the essential iOS Reeder.app and, less often, from Google Reader and Reeder.app for OS X.

It worked. I was happy. I started composing the celebratory blog post and explanation when...

Yes, you can guess. It fell apart.

I don't know why. Pinboard had blocked Superfeedr around that timefor aggressive crawling, but this turned out to be a red herring. IFTTT doesn't use Superfeedr.

IFTTT hasn't gotten back to me, but they're a free service so support is not predictable.

Well, I did know this was going to be a fragile and shortlived solution! I particularly didn't like that IFTTT is "free". I'm now looking for an alternate solution I can build around Reeder.app and Google Reader with Twitter and WordPress as outputs.


Update 6/19/2012: I didn't hear back from IFTTT support, but Pinboard convinced me to dig deeper there. I tried creating a new task but that also failed. The clue was in a screen I'd not visited, the Channels screen.

There I found this icon:

I edited it, reentered my credentials, and it's no longer offline. Now I'll see if it starts working again.

What happened? Well, there are two possibilities, and without help from IFTTT I don't know which is true. One is that Pinboard was unresponsive and IFTTT took the channel offline, but then never restarted it.

The other, which is at least as likely, is that I changed my password to Pinboard (I do that sort of thing) and that any email notification of subsequent credential failure was either misread, mislabeled, or filtered out. IFTTT doesn't use OAuth or similar service with Pinboard, so it needs my credentials to work. If that was the problem then IFTTT might be able to come up with a better way to notify users. For example; there was nothing on the IFTTT task screen to tell me a channel was offline, and when I manually triggered my tasks there was no error message.

Update 6/19/2012b: Fixing the Channels problem did the trick.

Thinking this over, why does IFTTT need my Pinboard credentials anyway? They are reading from a public feed, not writing to my account.

Update 6/22/2012: The latest IFTTT rev has much improved diagnostics. 

Mac OS X Mail: Import Mailboxes from a different user account or computer - and Old Netscape email import

It's been a long time since I've been impressed by Apple desktop software.

Partly that's because I have high expectations, but it's also that I'm not Apple's target demographic. I prefer capability over ease of use. I'm one of the very few people who likes iTunes for example -- because of it's very powerful playlist query language. I'm not looking forward to Apple's overhaul -- because I'm certain they'll remove the features I like.

Been a long time, but today I was impressed. I'd just completed a Eudora to Mail.app conversion in a user account I created to test the import process. 

The process went well, so I planned to repeat it on my primary account. I wondered though, if there was a way to simply import from the folder in ~Library/Mail. I couldn't find much documentation or discussion on Mail.app's merging or important of another Mail.app database, but the UI is simple enough. I updated my backup and give it a try.

First I moved the 'Import Folder' in ~Library/Mail to the Shared folder for my primary machine so I could do the import from my primary account. Then I selected that folder from the Import UI. Mail.app scanned the folder tree and returned a list of a few hundred folders I could import from. I told it to go ahead.

Fifteen minutes later I had my archive. Unfortunately there was no way to know if I'd gotten all 55,000+ messages; all I had were unread message counts and those didn't match up (but I didn't trust them anyway).

Happily OS X Mail has a very clean message structure, one message is one .emlx file (attachments are in a separate folder) and files are organized by a Finder folder corresponding to a Mail.app folder.

So I used Get Info from the Finder:

• Original: 2.36GB for 84,871 items
• Import: 2.36GB for 84,405 items

That's not too bad, except I'm short about 466 items. Coincidentally, I recall from the EudoraMailboxCleaner process that I had 478 folders. By directly inspecting the folders in the original and the import I could see that most of my folders were missing a .plist file that was in the original. This file didn't seem to be necessary; the messages are rendering correctly both times.

So it seems the import worked.

That's good, but what really impressed me was I noticed Netscape import option. I happened to have some old Netscape archives lying around. A few minutes later and my Netscape email was in Mail.app.

Twenty years of email in one repository; 9GB, 200,000 messages - that's impressive.

Update: I wrote too soon. The message metadata displays, but the Netscape messages are all blank. (A few Eudora emails are blank too.) OS X will preview them in the file system and I can read them with a text editor. So the data is there, but Mail.app doesn't want to render it yet.

Update b: Well, that was interesting. I tried REBUILD on one of the folders. All the emails vanished. I checked in ~/Library/Mail; they were still there and still viewable. So I tried removing the plist file and then tried restore again. This time they all came back, and they're readable. But the real weird part is that now all the Netscape emails in other folders are non-blank. I didn't touch any of those.

In praise of The Unarchiver

OS X's native archive manager won't handle zip files from the late 90s. The Unarchiver just opened one of mine from 1986.

Great app. Thank you Dag Agren.

Converting Eudora email archives -- the end is near (and Norton Commander ncmail.exe in a footnote)

I have years of email messages locked up in a 1.2 GB Eudora for Windows archive [3]. Converting them has been a pending project for about 10 years. This archive is a part of my extended memory; it has the keys to remembering people and places now forgotten.

Now time is running out. Eudora Mailbox Cleaner is one of the best tools for this project, and it's PPC only. Since Lion doesn't have Rosetta, the app won't run on 10.7 or higher. [1]

I've downloaded a copy. I think I'll delay my primary machine's Lion conversion until the Eudora conversion is done.

There are other Eudora archive conversion methods, but most run on Windows -- and I think Windows VMs on Mac will fade away as Apple locks the OS down to both block malware and enforce DRM. Time is passing. If you have old archives, you may have only a year or so to (relatively) easily convert them.

I'm planning to create a user account to test the import, then backup up my current Mail database prior to adding this one.

See also:

• Gordon's Tech: Converting Eudora email
• Gordon's Tech: Converting Email from Eudora OS X to a modern format - the TidBITS review: Adam Engst's comprehensive guide

Update

It took a few hours to do the conversion; most of that was spent waiting for a bundled AppleScript to Rebuild Mailboxes. Post conversion the database is 2.4 GB and over 55,000 messages.

Follow the directions carefully. I didn't think the nicknames were worth converting until I saw them; the connections between Groups and Members turned out to have powerful memory value. I recommend testing first with a special user and an empty address book and email account. If you import into an existing Mail database be sure to backup ~/Library/Mail first.

I didn't import Filters. They'd just be noise now.

With OS 10.4 and later there's an essential 'Mailbox Rebuild' step. My Eudora had at least 478 folders across a deep hierarchy [2]. It would have been extremely tedious to try to rebuild them all. So I tried the 10.4-10.5 rebuild AppleScript, even though I am running 10.7 Snow Leopard. 

That script runs slowly, and because of the way UI Scripting work it must own the entire machine. It will steal focus. Don't try to even switch users. It took over an hour to complete. Thank Darwin it's there; the job would have been untenable without it.

As per the documentation (mandatory read):

Your imported mailboxes will be located in a new folder called "Import" at the top-level of your local mailboxes. The new files will be located at "~/Library/Mail/Mailboxes/Import" on your hard disk...

... due to the way the Address Book works, a new group will be created for each Eudora nickname which expands to more than one email address.

I've already leveraged the archive to rebuild memories of people I worked and corresponded with in the 1990s. Some have died, some I've lost touch with. My old Eudora email archive has now been reintegrated into my extended memory.

For now the email archive and my old Address Book are sitting in a distinct user account. I will decide over the next week whether I'm going to leave them there or, more likely, repeat the process. If I do a complete merge I may create yet another user account, copy both my current email archive and this archive there, and let the process run.

OS X mail has a virtually undocumented import Apple Mail Data feature. I don't think it will be practical to import the archive using that but I'll take a look at it. (Update: It seems to have worked quite well. I imported my old Netscape emails at the same time.)

- fn -

[1] A year or two ago our 8 yo G5 iMac hard drive died. I dithered about whether to replace it or not because of the infamous iMac capacitor problem and its slowly delaminating screen. (The G5 iMac was one of Apple's most troublesome machines -- they desperately needed to move to Intel. The heat problems were extreme. It was, however, extremely easy to repair. That was even a marketing feature. Perhaps the G5 iMac is why Apple doesn't design for fixability any more.)

I did replace the drive and it's still working. It can even, if I were to install 10.4, run Classic. It's now one of my most valued machines. I think I may turn our MacBook into the kid machine and put the G5 iMac on a very light duty cycle so I can keep it available for another decade. I'll also archive my software install library onto its now capacious hard drive.

[2] My Gmail archive has essentially 2 mailboxes: In and All. Full text indexing and search changes things.

[3] Created by Steve Dorner, Eudora was the preeminent Mac email software and was dominant for a time on Windows as well. This archive, during its lifetime, moved from Mac to Windows. I found some 1996 correspondence with Steve, he was a very approachable developer with a lot of correspondence!

Before Eudora I used Norton Commander Mail (ncmail.exe) with MCIMail (pre-inernet -- 4867991@mcimail.com was my email at the time.

Writing this post inspired me to look into those archives. I did have them -- zipped up from the days when 200K was a lot of data. OS X couldn't open the old zip archives, but Unarchiver did fine. I used Automator to attach a .txt extension to each file and assigned them to TextWrangler. That was enough to convince OS X to index my old NC email messages. For example:

I actually got a bunch of public domain and shareware Windows applications running SLIP under OS/2 over my 14.4K modem ... 

... The second thing I'm excited about ...) is learning HTML and creating my own home page on the SILS Mosaic connection...

...  for my next 'HOT tip'. Check out TeleScript. Ignore "General Magic" -- it's bogus. TeleScript is the secret pearl.

I didn't remember that 14.4K modems coexisted with HTML 1.x. TeleScript is huge in an alternative reality. GM was a copy co-founded by "Bill Atkinson, Andy Hertzfeld and Marc Porat". Historic names.

The next archive to crack is a set of email written using Netscape's email app. (Update: Mail imports these directly.)

Notes on converting a MacBook Core 2 Duo dual USB to Lion

Our vintage 2006 MacBook Core 2 Duo can, in theory, run Lion [1]. I just upgraded it from Snow Leopard, which it ran reasonably well albeit with lots of fan activity.

I didn't upgrade happily. There's a reason I've waited this long. I wanted to stay with Snow Leopard, but Apple's MobileMe migration was going to cause problem for Emily's Address Book/iPhone Contact sync. Yes, that was the primary reason. Sad.

I upgraded all the apps we use, said farewell to AppleWorks and many games the kids no longer use, removed all PreferencePanes and extensions, unplugged all cables, did a safeboot cleanup and two completely independent backups (one a clone). Then I upgraded.

It seemed to go well enough once Spotlight rebuilt its indices and I refreshed everyone's account. it does take a very long time to log out; Lion is saving a lot of state information and the MacBook doesn't like that.

It went well enough, that is, until my old account. Then things got slow. Even though I'd updated VMWare Fusion to the final "Lion Compatible" 3.x version, I suspected it was the problem. I tried running my VM -- that was a disaster. After power down and a safe boot I uninstalled VMWare 3 per directions.

That helped a great deal. In fact, the laptop is quieter than it used to be. I suspect Fusion has been causing problems for a while [3].

So will I try Fusion 4?

No, not on this old machine. I really don't use my VM's very often, and neither Parallels nor Fusion are sold through the App Store. To do their magic without Apple's help they must be hacking the underlying OS; and OS X is increasingly hard to safely hack.

For now the MacBook seems to have survived Lion - albeit at the cost of a little used VM I'm probably better off without and a dozen or so older games -- and AppleWorks.

So far, better than it could have been. I am, however, regretting obeying Lion's command to update my Airport firmware [2]. 

[1] That's almost six years ago! I'd forgotten how old it really was. Maybe I shouldn't be too upset it can't run Mountain Lion. [2] Lion really wanted me to undo my Time Capsule firmware regression, so I did. Now I'm seeing more problems with losing connectivity, i saw a -1 error again, and I'm again having to rebuild Spotlight indices of the backups. I have a strong feeling I'll be reverting again. I seem to be the only one with this problem though. [3] My best Fusion experience was version 2 with a Windows 2000 VM -- on that old MacBook probably with Leopard (10.5). It's never worked as well since. 

Update 6/17/2012: Logging out and user switching is much slower. It takes about 20-30 seconds to log out and 15-20 to switch. I think it's because of all the context saving Lion does; this old machine can't handle it. There are ways to disable saving of application states, but I'm going to wait a while on this one. Otherwise things aren't too bad. The machine is quieter than it has been for years, the fan no longer roars. I suspect that's due to uninstalling Fusion 3.x, but it could be a Lion improvement.

Update 6/17/2012: I went through each user account and turned off 'save and restore windows' in system preferences. Then I logged out and unchecked the restore windows on login option shown there. No logout and login is back to Snow Leopard times. Now I have to figure out what to do about #$@$ Google Software Update. it keeps popping up in managed accounts that don't have privileges to run it.

How well does Gmail archive via IMAP really work?

My Gmail account holds 64,800 messages. Of those perhaps a few hundred are of lasting interest, but by the time AIs sort out what they are we'll be long obsolete.

So, perhaps illogically, I like to hold onto the archive. That means a local copy. 

I recently had an opportunity to test how well an IMAP archive approach works [2]. I upgraded a machine I'd not used for a while and had to catch up on a year's worth of email. It took a few separate synchronization sessions to catch up, I think I ran into Google's bandwidth limits/transaction throttling behavior.

I ended up with 64,440 in my local "All Mail" synchronized folder, so about 360 have been lost in the bowels of IMAP. Perhaps they'll trickle in with future synchronizations.

The oldest message in Gmail online is dated 2/29/2004 [1], in Mail.app (Lion) it's 2/2/2004. Yes, that's weird.

To answer the post question then, I'd say Gmail archiving via IMAP is imperfect but not pointless. A C grade.

[1] It's not obvious how you can see this. If you mouse over the message count there's a hidden option to swap newest and oldest. It's not easy to get to show; navigating large email collections in Gmail is almost impossible with the "new" UI.
[2] IMAP archive is what Google's all-but-defunct Data Liberation Front recommends. I miss that gang, but there were of the Google- era. In the Google 2012 they're an anachronism.

Sparrow — How not to do marketing

Sparrow's value proposition is that it works with Google's email much better than OS X Mail.app (even Lion OS X Mail), but it also uses the OS X Address Book.

It's a great solution to problems I face.

They don't market this though. If they had, I'd have tested it out long ago.

Aperture 3.3 is iPhoto Pro. At last. Sort of.

After years of false claims of iPhoto to Aperture migration support, and four months after I realized iPhoto was going down the wrong road for me, and three months after I began my extremely painful iPhoto to Aperture migration, Apple has released iPhoto Pro (aka Aperture 3.3).

Imagine my joy.

The only saving grace is that I might have saved a few people by advising them to wait for Aperture 4, which has now come in the form of iPhoto Pro 1.0/Aperture 3.3.

According to Apple's marketing claims, the latest (Lion-only) versions of Aperture and iPhoto  share a single database model. So, in theory, both apps can work on the database.

I doubt it works as advertised, but it has to be an improvement on my experience! I'd strongly advise waiting until September before doing a major iPhoto to Aperture migration. Now that you know the end is in sight the wait should be tolerable.

Personally, I'm looking forward to, at the least, using iPhoto's Picasa uploader and iPhoto's superior UI for Event/Project and many common image management operations.

Assuming the inevitable bugs get sorted out, this is an extraordinary conclusion to what must have been a formidable software effort. At one point Apple had two completely incompatible photo management products. One was a natural Mac app with an elegant UI and some infuriating limitations. The other looked like a port from NeXTStep [1]. They had almost nothing in common.

Slowly, painfully, Apple turned these two disparate products into iPhoto and iPhoto Pro. To do that they had to reconcile very different functional models and data models. It would have been a very hard, very long, glamor-free slog. I hope the team was at least paid well.

Now, if you'll excuse me, I'm going to go cry in my Scotch.

[1] I was never able to learn where the heck Aperture came from. It's neither Windows nor Mac.

Update: I remembered my underused MacBook Air runs Lion, so I tested the new Aperture and iPhoto together. Unsurprisingly, they failed my first test.

First -- the good news. Annotations on an iPhoto Event are now visible in Aperture.

The bad news -- Album annotations (descriptions) are still not viewable or editable in Aperture.

Maybe in another year or two?

iTunes Store: "Error (-50)" - try GoogleDNS?

A few weeks after an intractable -100000 error iTunes error that only resolved when I uses SSH tunneling to download from a California server I ran into a iTunes Store: "Error (-50)" when downloading purchased content with a TV show my son wanted.

I cleared the download cache and deleted the partial download but I was stuck with the problem. I'd already ruled out all the other issues Apple mentions. I certainly didn't use a 'web accelerator' ...

But wait. "Web accelerator" gave me an idea. That suggests a networking issue...

So I used OS X Location switching to swap my CenturyLink DNS services for GoogleDNS (I had both configured). That did it. My download streamed normally.

I wonder if DNS issues with iTunes will get more common; after all, iTunes is a competitor for many DNS providers (though not, I think, CenturyLink).

Perhaps more likely, I wonder if this is related to Apples IPv6 transition.

I'm leaving GoogleDNS running for now.

Access 2003: Join expression not supported with VBA type conversion

One of the stranger bits of my life is that I often do data manipulation using Microsoft Access 2003 in a Windows 2003 server VM.

 

It's a long story. I suspect I'm not completely alone however. Access 2003 is unbeatable for certain kinds of data wrangling across disparate data sources. Access 2007 was a quite severe regression (post-2000, software  regressions are increasingly common).

 

So perhaps it's worth sharing the weirdest bug I've seen in a while. I was unable to do a "left" join on two subqueries where one of the keys involved a type conversion (string/long using VBA CStr() or CLng() - I tested both ways). I got a meaningless "Join expression not supported" error message.

 

I think this has worked fine for years. I wonder if a forced security update broke something. I found if I switched the query from 'Dynaset' to 'Dynaset (Inconsistent Updates)' it worked. Snapshot didn't work.

 

Now you know.

 

For me, it's a sign that I either need to switch to Access 2007/2010 or switch jobs. (I don't think 2010 is 64bit, so I still face the problems with ODBC drivers and bitness. I might be better to try Access 2007 in an XP VM.)

Update: I'm seeing other issues. It feels like something very bad has happened, probably as a result of a Microsoft security update, but I can't tell what.

 

Update 7/4/2012: I'm not quite sure what's going on, still having some issues. It might be an ODBC issue. I think the error message is a red herring.