Converting OS X Mail.app from ISP IMAP to Google Apps IMAP

A few weeks ago I wrote about Google Gmail gets IMAP services. I'd already been using Gmail POP services on my accounts, but IMAP was a better choice for my wife. [1] I figured I'd migrate her from our no longer competent local ISP email service once things settled down a bit.

Today I migrated my wife's email from our newly annoying local ISP (visi.com) to our family private domain Google App domain.

It worked, but it's a bit tricky. I'll outline the process and some key things I learned, but I won't replicate the directions available elsewhere.

But first ...

A cautionary tale about IMAP hell - IMAP and local spam filters

Emily has been using IMAP on 3 OS X machines at our home for years. This seemed to work, but in retrospect there were severe problems I didn't recognize.

1. Each client was doing spam filtering. She never tuned the local spam filters and was used to checking her ISP's spam filter, not the OS X Mail.app spam filter. The result was a lot of email was being captured in the local junk folder of each machine -- then being deleted after 30 days. She never knew about these. She lost quite a few messages this way, including some from family and business.
   
   
2. It gets worse. She could easily have 2 Mail.app clients running simultaneously. Each polled the server every few minutes. Depending on which one got there first, email that was flagged by Mail.app as spam would get pulled off the IMAP server and saved in one or the other of the local Junk folders. This hit me when I was testing her new email. A message I expected to appear never showed up -- because the downstairs machine had pulled it from the IMAP server and stored it in the local Junk folder!
   
   
3. Sent mail, of course, piled up locally. (She cc'd herself on key sent messages, so this was more annoying that anything else)

Now on to the conversion ...

Converting Mail.app from VISI IMAP to our family domain Google IMAP.

1. Print out Google's Mail.app directions and their recommended client settings [1, see below!]
2. Scan the Google Gmail IMAP Access document. Especially read known issues.
3. Read the ThirtyOne discussion for the key advice on "Properly sort Drafts, Deleted, and Sent mail in Apple Mail"
   

I did the conversion on 10.3.9 and 10.4.11 Mail.app. It worked similarly on both versions. A few tips from the experience:

1. If you have a huge number of emails on an IMAP server this may take a very long time or your client may crash. This wasn't an issue for us as we'd been archiving emails locally (limited capacity on the old service).
2. If you're changing from a previous IMAP service, just edit your old IMAP entry. Don't create a new one. You need to edit to the new settings. Then exit. Then start again. I had to do that on 10.4 and 10.3 on 3 different machines. It only worked when I would exit and restart. Actually, most IMAP configuration changes required a Mail.app restart to complete properly.
   
3. Turn off spam filtering in Mail.app. Remember our email Hell? Don't let that happen to you. Let Google do the spam filtering.
4. The process created some empty local machine folders with names like "Junk" and "Draft". This was in addition to the IMAP folders. I confirmed they were empty and deleted them. They did not reappear on restart or with IMAP sync, they really were accidental junk.
5. Follow the "ThirtyOne" directions about "Use This Mailbox For" so the Google Apps Sent folder vanishes and you see only the one "Sent" folder. Much nicer.

There's still one more thing. Google says Mail.app "Move deleted messages to the Trash mailbox" should be unchecked. This is fine when you're strictly doing IMAP. There's one Trash that appears and that's Google's. But, what if you're using storing email locally in an archival store? If you delete from that store, you may want a Trash.

On the family iMac, which is backed up, Emily does have archival email stores (see image above). On that machine only, I did check Mail.app "Move deleted messages to the Trash mailbox" and I set "use this mailbox for" so that Gmail Trash and Local Trash appear as "children" of the Mail.app iconic Trash mailbox. (See ThirtyOne folder link, above.)

The screenshot to the top left of this post shows you the 10.4 Mail.app view with both a set of local folders (we archive locally, drag and drop from the server to the local folders such as Save and "flylady") and the IMAP folders. [Gmail]/Sent has been mapped to the standard Mail.app Sent folder, but "Starred" and "In" remain as folders under [Gmail].

It seems to be working very well. The process is strictly for geeks however.

PS. When Blogger says they only accept JPEG and GIF uploads, they lie. They accept PNG too.

[1] Google maps tags to IMAP folders. Since I've used tags with my email this is a bit annoying, so I'm staying with POP for now. My wife has never used Gmail or Google Apps Mail, so this isn't an issue for her.

[2] Google is not joking about the best way to set up Mail.app. Follow this advice:

Drafts:
Store draft messages on the server > checked

Sent:
Store sent messages on the server > do NOT check

Junk:
Store junk messages on the server > checked
Delete junk messages when > Never

Trash:
Move deleted messages to the Trash mailbox > do NOT check [jf: see "one more thing, above"]
Store deleted messages on the server > do NOT check

Update 11/19/07: I've posted on Gmail Groups: we're seeing multiple versions of messages saved to the Trash.

Update 12/21/07: Either Google changed something or a new router might have broken connectivity from one of my IMAP OS X Mail.app clients. I tried following Google's OS X Mail.app configuration settings and got nowhere. The fix is to disregard all their lovely client-specific instructions and use the bare bones geek-friendly directions provided for "other clients". The settings there differ from the OS X Mail settings.

Needless to say, Google is in its usual confused state.

Google's link search behavior and a Firefox request

It's annoying to have so many text entry fields at the top of a browser. There's one for a URL (which may act as a search field), one for search (which may allow specification of the search engine) and one embedded in the Google or WL toolbar (the latter can be set to search Google.

It's no wonder users get confused:

Google Tries to Fix Broken Links

....So people type URLs in the search box. Google changed the way it treats those queries last year: instead of showing information about a URL, it returns standard search results and the top result should be the page that corresponds to your query. This is a great way to avoid typosquatting sites, like flicker.com or twiter.com.

But what happens when Google doesn't find a page from a site included in the index? Until recently, it returned the standard 'Your search did not match any documents.' Now it returns more helpful information: results from websites with similar addresses, suggestions for queries and a way to identify the page by restricting the search to the domain/subdomain from my initial query.

This is a great improvement that should be emulated. It's one more step in the "brute force" approach to broken links. We're stuck with "brute force" because the web never evolved a true directory service.

Now I'd like Google (or Firefox) to add an option such that when the URL matches a well ranked site then we just go to the site. We'd only see search results when the URL was off, that would be the current behavior.

This might have to be an advanced option since the variant behavior would be confusing, but it would let me live with a single top text field. One that also needs to mapped to command-L (OS X) or Ctrl-L (Win).

iPhone UI unsuited to podcast navigation

My SONY car radio is pretty good with AAC and MP3 tunes, but it fails miserably with podcasts. The controls are designed for navigating a 3 minute tune, not a 60 minute podcast. At the very least they needed to have the fast forward accelerate over time.

I was sure Apple wouldn't mess up like that.

Wrong.

graphpaper.com - Scrubbing the iPhone Scrubber

.... the new scrubber bar is almost useless, especially for long tracks like podcasts where it’s impossible to move the playhead any less than a few minutes per hop...

At least there's a chance Apple will fix their dumb mistake in a future update.

Fixing Leopard stacks (via Japan)

I won't be walking the Leopard until 10.5.3, by which time Stacks might actually work out of the box, but I'm still going to make note of this hack:

TidBITS Problem Solving: A Simple Hack To Fix Leopard's Stacks:

...Zen balance is now restored to my Desktop and Dock, and I find myself once again using Stacks to organize my current projects. For smaller folders the single-click access to the stack is surprisingly more convenient than right-clicking was under Tiger. And sorting by date modified makes even large directories useful as stacks, since I usually want to access the most recent files anyway.

It's a really beautiful hack, and to me one of the neatest aspects is that it comes to us from a Japanese Mac fan.

Rumor has it that the Mac is starting to get a following in Japan. If true this will bring a new rush of invention and creativity to the platform.

Google Desktop Search bites the dust

I've posted frequently since January 2005 about various approaches to full text search in Windows XP in general and Outlook 2003 in particular.

My test environment is harsh. GBs of Outlook PST files. GBs of documents. Machine never rests. Hard drive always on the edge of meltdown.

The best solution, by an order of magnitude, was Lookout for Outlook. Fast, stable, and the bugs are manageable. Alas, it conflicted with some other Outlook Add-Ins (that environment reminds me of DOS TSR hell). Lookout also doesn't work in Outlook 2007 and it's definitely not supported anywhere, so I thought I needed to change. That was probably a mistake.

I used Yahoo Desktop Search for a while, but it was increasingly buggy and Yahoo finally abandoned it. It reverted back to X1 but it was even buggier when I tried their version.

Recently I decided Windows Desktop Search was my only option. It was slow and sucked performance, but it did all I needed and it incorporated Lookout's search syntax (see the help file). Unfortunately, my work machine has been possessed lately.

I'd removed about everything I could think of to try to stabilize my system, so it was time to remove WDS and try the only remaining contender Google Desktop Search.

Yech. This is what I wrote when I uninstalled it:

... I don't have enough control over indexing behavior. I don't have enough search syntax control - esp. for outlook tasks vs. email search. It was WAY slow to do a search against my multi-GB Outlook archives. It doesn't treat folders as first class search objects...

So now what do I do? I could buy a MacBook Pro and run XP in a VM, but Spotlight won't search PST files.

I need full text search - esp. of PST files.

I'm going to try Lookout again; I've gotten rid of most of my Outlook plug-ins / add-ins anyway.

Gee whiz, I miss my home OS X machines. I know people complain about Vista, but I'm not a great fan of any Microsoft products at the moment (though my problems, to be fair, may be hardware related). Even my beloved Windows Live Writer is buggier at the 1.0 release than it was in the last two beta versions!

Update 11/16/07: Joe Cheng (see comment) noted I hadn't submitted my WLW bug reports. They're really not awful, but they stood out relative to the perfection of the last beta. One is certainly a bug -- the category/tag names can be truncated randomly (last few characters are missing). The other two I need to prove are really bugs, which is a bit tough since my Dell laptop is definitely flaky.

I'm going to be more careful going forward about what I call a bug ...

Update 11/16/07b: The regressions (bugs) will soon be history. In the meantime don't use the Post to Weblog as Draft feature with Blogger.

The real iMovie '08 is Final Cut Express 4

iMove '08 should have been called iClip 1.0. It clearly wasn't an upgrade from iMovie HD, it was a new product.

At the time I wrote:

Gordon's Tech: Apple's iClip 1.0 (iMovie '08) shaft: what they should have done

...The honorable thing for Apple to have done would have been to provide iMovie HD users with an upgrade path to Final Cut Express (which might also require some updates to FCE, I'm not sure how serious Apple is about that product)...

Well, Apple didn't do the honorable thing for iMovie HD users, but the new Final Cut Express is out. Users of iMovie HD should probably forego the iLife 2008 update and buy FCE 4 instead.

I'd give them points if they were to provide a $100 upgrade path for users of iMovie HD, but I admit that would pretty hard to police. The new product is $200, so heavy users of iMovie HD will probably pay up.

Safari 3: very cool tech

Safari 3 is even more impressive than I'd realized: Surfin’ Safari: Ten New Things in WebKit 3.

Faster - esp. JavaScript. Less memory. More capabilities. Fewer bugs. SVG support (not fully optimized).

I'm going to test it with Blogger and Google Docs, Safari 3 beta didn't work too well there.