Saturday, February 14, 2009

Work home contact integration: Outlook to Google to OS X Address Book

One of the secrets learned on the wrinkly path is that simple solutions are very hard.

Obviously, not hard to describe or apply. They're hard to invent, because there are so many solutions, simple and complex alike, and most of them aren't good.

Lately, I've been starting to tackle a gnarly problem - managing my Contacts (address book). It's a big problem ...
Gordon's Tech: Project Contacts: Integration across iPhone, Google and whatever

... There are a lot of variables in the mix, from endpoints to transition tools. Gmail Contacts, Google Apps domain shared contacts, Google Contacts API and extended Contact Kind, OS X Address Book [1], iPhone Contacts.app, Microsoft Exchange Server, ActiveSync, SpanningSync, FileMaker Pro, Bento, MobileMe, gSyncIt, iTunes sync, Google iPhone Sync, vCal, hCard, GData, OpenSocial, Windows Live, Facebook and Outlook among others. (Yes, several others.)...

... Yes, this one is the big Kahuna. Big enough that, for this project, I’ve created a new Gordon’s Tech Label of “Contacts” to help me as I work through all the options.

In the meantime, there’s one big bit of good news. Contacts don’t change as much as Calendaring events. I can get by with manual export/import for quite some time...

It took me quite a bit of thinking and playing with the variables to come up with that last sentence. I explored a lot of very complex approaches, only to be stymied by problems with data structures and a lack of adequate OS X desktop tools.

Eventually I decided to scope the problem strictly to creating a pool of Contacts in OS X Address Book that would sync to my iPhone, and a slightly different pool of Contacts in my Gmail account. Once I did that, the problem became a lot simpler. I now have 1,600 Contacts on my iPhone.

A diagram would help here, but I don't have time to make one up, so here's a skeleton of what I'm doing. It's all manual, but I only do it every few weeks, because (this is the key), Contacts don't change as much as most things.
  • Exchange Server/Outlook -> Export CSV file -> Google Gmail import into a "group" called "Work Contacts". Since this is a simple one way export I clean out this group before the import. I do this every few weeks.
  • Gmail export -> vCal file -> OS X Address Book. When you do this Address Book adds items to the main group, but it highlights those just imported. You can then carefully drag them into a "group" called "Work Contacts". Incidentally, OS X Address Book duplicate management is very sophisticated -- the best I've yet seen.
I think I'm going to discover some other interesting nuances. In particular Google Apps now provide an intriguing pooled Contacts collection that may facilitate a family Contacts pool.

Still, this is substantial progress for me.

Update: BTW, the default vCal behavior for OS X Address Book does not export photos or notes. You must turn these on in options. The vCal from Google does export notes and photos. Address Book imports the notes, but not the photos -- or at least not during a merge/update (this may be a bug).

Update 2/18/09: Oops. Corporate exchange servers often use x.500 email addresses (aka EX format, for "exchange") internally. Those aren't useful to Google or my iPhone. I recall older Palm sync solutions substituted net-standard email addresses during sync; I need something like that. So there's still some work to do. I'm not the only one dealing with this ...

Update 2/19/09: Inspecting the output file I see that Outlook 2007 exports CSV file emails in this form:
E-mail address E-mail type E-Mail Display Name
fredflintstone@bedrock.zz SMTP Fred Flintstone
/o=Megacorp/ou=North America/cn=Recipients/cn=xdf415 EX Barney Rubble (Barnie.Rubble@Megacorp.com)
So the x.500 (EX) addresses contain an employee identifier ('cn') that maps to a current email address which is stored in parentheses in the E-Mail Display Name field. The SMTP style addresses do not display the email address in the E-Mail Display Name field.

So really, we want Google to be smarter about importing. It ought to "know" to pull the email address out of the [E-Mail Display Name] column when [E-mail type] = "SMTP".

This is rather annoying.

Someone must have written a utility that reads in this export file and post-processes it to extract the SMTP address and store it in the E-mail address column for EX type rows. I'll have to look for it. Meanwhile I'll ask Google to fix their import processing.

Update 2/19/09: I tried linking from Access 2007 to Outlook 2007, but this produces a very peculiar view of email addresses. The old data model problem no doubt -- the UI for doing this looks very old, so I suspect this function is really obsolete. I then exported as Access 2003 from Outlook 2007 and got something similar to the CSV file export. On inspection, however, it became obvious that Outlook's storage of SMTP equivalents for EX email addresses is pretty erratic. Indeed, it looks like even the EX addresses stored in Outlook (vs. on Exchange server) are flaky. So I'm beginning to understand why Google doesn't want to try to get clever with processing the CSV files -- the email addresses are unreliable.

I feel so sorry for programmers who try to work with this stuff. Outlook is really decrepit.

I'll need to think about this a bit more ...

Update 5/19/09: Major progress. Among other things, I learn that if you copy .OST Contacts to a .PST file Outlook resolves the X.500 EX addresses to SMTP addresses.

Project Contacts: Integration across iPhone, Google and whatever

I was just about ready to turn my iPhone into cyber-mulch when, unexpectedly, Google saved it. Calendar nerdvana has come to my iPhone, albeit at some cost to my iPhone battery. Again, I have hope. (Yes, Andrew, I am obsessed.)

I need hope. My PIM/PDA world is still a post-Palm mess. At the moment Google is my source of Calendar Truth, my iPhone is a sync client, and iCal is not involved. I’d like to have a copy of Calendar data that I “own”, so I’ll eventually re-integrate iCal. I’m no fan of Apple’s calendaring monstrosity, so there’s not a great rush. Sometime this year though.

In any event, it’s time for a refresh of my Palm Migration Status page:

image

Things have changed quite a bit over the past few months. It’s been a lot harder than I ever expected, and there’s still a way to go. (I hope the Palm Pre is a smash success, if only for vicarious revenge upon Apple for MobileMess.)

Today my old Palm Classic PIM data is scattered across my iPhone, Google, OS X desktop, ToodleDo, Evernote and Jott. That has to get simpler! I’ve made a “strategic” family and personal commitment to Google, so if/when they integrate their inadequate Tasks with Google Calendar (here’s how to do it) I’ll drop ToodleDo from the mix.

I could drop Jott fairly easily as well, and Evernote is improving nicely, so for now my focus is Contacts. This isn’t as critical as Calendar integration, but I’d sure like to get all my work and home Contacts on my phone.

Unfortunately, Contacts are hard, much harder than Appointments, Notes, Tasks, etc. (Ok, so Appointments have their own special non-technical problems too.)

There are a lot of variables in the mix, from endpoints to transition tools. Gmail Contacts, Google Apps domain shared contacts, Google Contacts API and extended Contact Kind, OS X Address Book [1] OS X Address Book Google Sync [2], OS X Address Book Exchange Sync [3], iPhone Contacts.app, Microsoft Exchange Server, ActiveSync, SpanningSync, FileMaker Pro, Bento, MobileMe, gSyncIt, iTunes Google sync, iTunes Address Book Sync, Google iPhone Sync, vCal, hCard, GData, OpenSocial, Windows Live, Facebook and Outlook among others. (Yes, several others.)

Bad enough, but “Contact” information is a big part of vendor lock-in strategy. Yes, Data Lock, big time. We’re talking Google, Microsoft and FaceBook fighting over ownership of contact relationship networks, and everyone fighting with banks, Amazon and the phone companies for identity ownership.

Complexity, corporate combat, rapidly shifting implementations, vendors focused until recently on Calendaring …

Yes, this one is the big Kahuna. Big enough that, for this project, I’ve created a new Gordon’s Tech Label of “Contacts” to help me as I work through all the options.

In the meantime, there’s one big bit of good news. Contacts don’t change as much as Calendaring events. I can get by with manual export/import for quite some time.

In the meanwhile, I’ll be experimenting carefully. Follow the Contacts thread forward to learn what I learn …

[1] OS X iCal is miserable, but OS X "Address Book" isn't all bad. If Bento weren't so crummy I'd put the OS X "Address Book" plus Bento at the core of my strategy.

[2] Update: Since my original posting, I discovered through this thread that OS X 10.5 Address Book has its own options to synchronize with Exchange, Google, and Yahoo -- in addition to the Phone to Google synchronization built into iTunes/OS X (available for 10.4 and 10.5). The mind boggles. So one could synchronize the OS X Address Book with Google Contacts directly, or through Google's Active Sync service ...

Update 5/19/09: Major progress - I have work and home integrated and options for Google too. The cost is I may have to subscribe to MobileMe.

Friday, February 13, 2009

Adobe Reader installer quits after ODBC Setup

Adobe is the most incompetent large software company on the planet Earth.

I really can't understand why they're still in business, or why anyone pays any attention to them.

I just ran into this bug -- it was well understood in August 2008 ...

Adobe Forums - AdobeReader 9 installer quits after ODBC Setup on Vista

... In my case this was caused by the presence of a MS ODBC setup.exe file programfiles\SAP\FrontEnd\SAPgui\FILC\odbc. This file was in PATH and thus triggered, I renamed the file temporarily and the install works. Probably goes for any file named setup.exe in PATH...

No other XP installer has this kind of problem.

Why is Adobe still in business? Won't someone rid me of this company?!

Thursday, February 12, 2009

Applications for moving data out of OS X Address Book

After many years in which my address book more or less lived in a FileMaker database and Outlook/Palm I migrated it to OS X Address Book.

Now that Google is providing Exchange services for my iPhone it's time to move it again -- at least a part of it will live in Google.

This is a surprisingly gnarly task. I'm not sure what tools I'll use; I think I might end up doing some reconciliation work using Outlook 2007 and Microsoft Access 2007 because of the limitations of Address Book and FileMaker.

As I think this through, here's a list of tools I've come across for moving data from OS X Address Book (it exports VCF files).

Wednesday, February 11, 2009

Gmail contacts - now search works

It took a surprisingly long time for a search company to enable search across contacts:
Official Gmail Blog: Four changes to Gmail contacts

...Search across all contact fields
We've heard you loud and clear, and contact search now works much better: instead of just searching contact names and email addresses, it now includes phone numbers, notes fields, and mailing addresses as well. So, if you're visiting the Bay Area and looking for friends to catch up with, you could try typing '650' or '415' in the contact manager search box...
On the iPhone you can still only search by name and business, on the original PalmPilot you could search across all fields.

Google has made several changes to Contacts. In Google Apps you can now enable a "contacts" app that moves Contacts out of Gmail into their own context. Of course there's long been a secret URL to do the same thing.

I'm a bit disappointed Google has never restored the wonderful simplicity of adding email addresses in version 1 of Gmail. I recall you could post lots of addresses into a text field and Gmail would parse them.

Even so, it's good to see Contacts getting attention.

Access OS X Address Book records using OpenOffice database?

While idly browsing FileMaker forums contemplating the futility of Bento and the tragedy of FileMaker 10 and the theory that Product Managers hate me I came across a remarkable post:
Does FMP 10 have full integration with Address Book? - Using FileMaker Pro - FileMaker Forum

... I looked at addressbookmanipulator but it seems to be a development plugin. It costs $74 via credit card. It doesn't transform FMP in Bento with respect to address book integration :-( I would love to be able to select the address book as a data source in FMP.

Another promising option is OpenOffice 3.0. Its database wizard offers to connect to an existing database and lists Mac OS X Address Book as an option. When using it you get what seems a complete table for each Address Book group and includes the UID. The downside is that you can't edit the data ...
OpenOffice?

Well, yes. For example (read only, no edits):
I'll have to give OpenOffice a try! It's sad that OO can do what FM can't. Along with the A to G Address book utility this may be another tool to use in my quest to create a unified work/home Google Contacts collection that will sync via Google's ActiveSync Exchange service to my iPhone.

Soon, victory will be mine (queue insane, evil, laughter)...

Update: Dang! OpenOffice 3 is still Intel only! If you try to download you get a "bouncer" error. On the other hand, you can download a pre-release 3.01 rc2 PPC version from a dev site (this is seriously weird, and not encouraging).

On the other hand, the Java based NeoOffice 2.x and 3.x preview both have PPC versions.

Update: Wow, OpenOffice 3.0 PPC is pretty spartan. Lots of redraw artifacts on my G5. It did open the data, but I haven't yet figured out how to export from OO Database in any format. A useful tool though.

Synchronizing iPhone and Gmail contacts: getting OS X Address Book into Gmail

Google recommends this utility for transforming an OS X Address book into the (yuch) CSV format Gmail can handle: A to G The easy way to get your Mac OS X address book into Gmail. This is one possible path on the way to using Google's ActiveSync implementation to sync contacts from iPhone to Google and gSyncIt to get them from Outlook to Google.

I'm approaching the problem of integrating my work, Gmail and OS X contacts with great care. It feels like juggling antimatter -- the data models are very different. There's a lot of potential arrow-of-time entropy risk -- data loss that can't be reversed.

One big confounder is the absurd persistence of CSV as a format for moving data around. That's was nice in 1970s. We should be using an XML transaction that ought to at least preserve line feeds in text notes. Unfortunately we should also be using object-oriented personal databases too, but the last of those died in the 1990s. (Anyone remember "Arrange" for MacOS?)