Google IMAP and the intractable duplicate IMAP email problem

I've only recently returned to using Mail.app; for a couple of years I was almost pure Google. Since I integrated my personal Contacts across OS X Address Book/MobileMe/iPhone/Google Contacts however I've returned to enjoying the power of a dedicated email client.

Alas, there's a catch. In my time away I'd forgotten the Gmail IMAP to Mail.app sync duplicate email problem. It's not really a Mail.app specific problem, every IMAP client has the same problem.

This is an intractable problem. The standard model for email is that that every message belongs to exactly one folder. Folders can contain folders. It's very much like traditional directories since the days of DOS and well before. (Tree data structure)

Google has a very different model. Google's mail "folders" are an illusion, in reality all Gmail messages are in a single repository. Each message can have many tags, and a single tag can be used for many messages. Each tag is treated in the UI like a "folder", but a message belongs equally to each "folder' (they're just tags).

This is more than a mere "physics" problem -- it's a math problem. 7 != 3, and a tree-type folder hierarchy cannot represent a tag collection.

There are two approaches to "synchronizing" a traditional email application with Gmail. You could make the first tag of an message the "folder" tag, and ignore the other tags. The other approach, which Google took, is to replicate emails. So a single message in Gmail with 4 tags becomes 4 messages in Mail.app in 4 folders.

Of course this wastes space, but space is cheap. Much worse, however, is that it clogs up searches.

I don't see this problem going away - unless Google admits defeat and regresses to using standard folders. For now, however, I'd love to see a program that would go through my Mail.app database and remove all the duplicate emails - even if it randomly assigned them to a single folder.

See also:

• TidBITS Networking: Achieving Email Bliss with IMAP, Gmail, and Apple Mail. I don't think he's anywhere near bliss, but there are lots of details. Alas, if you use tags in Gmail, all of this explication won't help.

Why are my Canon images names with a _MG_ prefix instead of IMG_?

This happened to me a while ago.

My Digital Rebel images started having the prefix _MG_ instead of IMG_.

I figured I'd run by some strange numbering threshhold. It was odd though. Sometimes I'd see IMG_ again ...

It bugged me.

I should have googled it long ago. Photo.net explains - this is what Canon does when you switch from sRGB to Adobe 1998 RGB color space. It's a JEITA specification mandate. I probably see IMG_ when I shoot JPEG instead of RAW.

Thanks photo.net.

Blogger lives - mobile templates in draft

Last June I put Google's Blogger on the Dead list, in part because of the lack of a mobile template. This morning Blogger in draft offered me a mobile template.

Blogger isn't dead after all.

Google apps for our family: now with email monitoring (delegation)

It wasn't designed to support parental control and supervision, but Google's new email delegation function is very helpful (emphases mine) ...

Google Apps update alerts: Email delegation now available for all Google Apps customers

.... Administrators must first enable mail delegation by checking the 'Mail Delegation' checkbox under 'Email Settings' in the administrator control panel.

To enter a delegate, users can select the 'Accounts' tab under 'Settings' in Gmail and click 'Add another account' to enter their delegate's email address.

Once the delegate is signed into their own own Gmail account, they can then access the other person's account from the account selection menu at the top of Gmail....

You can only add delegation accounts that are a part of the Google Apps domain, you can't add external Gmail accounts.

It takes a while for the "account selection" menu to appear. About fifteen minutes after I set up delegation on my son's account a small arrow appeared to the right of my bold email address on the top right of my Google Apps Gmail page.

In fact currently my son does not directly use Gmail, it's just an IMAP service for OS X Mail.app. Only I know his Gmail password. So this doesn't let me do anything I couldn't already do, but it's much easier to monitor his account.

Troubleshooting MacBook wake from sleep problems

Our 3 yo Dual USB MacBook (Intel dual core) is having intermittent wake from sleep problems. It's on a WiFi network and running Leopard - 10.5. On lifting the lid it spins up then hangs with a faint blue screen. There's no cursor.

I haven't found a fix yet, but here's a list of what I've tried [4]. They're listed in the order I'd suggest. Of course the first thing to do with any ill-behaved machine is to check that your backups are good...

1. Run Safe Mode start, (aka Safe Boot) then restart [2]
2. Remove share mounts
3. Login items - remove all
4. Clean out list of remembered WiFi networks - Network:AirPort:Advanced. [5]
5. Do not inactivate the ethernet port - even if you don't use it. Delete all Locales. [6]
6. Disable Bluetooth
7. Update Flash (or consider removing it forever)
8. Input Managers - remove all. They are Satan's tool.
9. StartupItems - remove all from Library\StartupItems and ~\Library\StartupItems if it exists. Do not mess with System\Library.
10. Onyx: check corrupted preferences, check disk, delete caches including font caches
11. Onyx: Repair Permissions :-) [1]
12. Install disk, run disk utility check disk
13. Reset SMC [3]
14. Zap PRAM [2]
15. Run hardware test in loop mode on install DVD
16. Remove or update any low level software
    1. Look in /System/Library/Extensions for ideas, but dont' mess with /System. Look in Library/Extensions.
    2. Cisco VPN for example - uninstall requires terminal
    3. Retrospect Client [7]
    4. Fusion is another - it loads early. Update it.
17. Console.app: Look for messages on startup. Unfortunately Console captures a vast number of warnings and error messages even in a "healthy" system.
18. Disable Safe Sleep (more a Leopard than Snow Leopard option?)
19. Look for more ideas here: Tutorial: Dealing with Wake-from-sleep issues | MacFixIt - 2006
20. Reinstall OS X (desperation move). If you're on Leopard, might as well upgrade to Snow Leopard.
21. Send out for repairs - likely a hardware issue. Bad memory can do this and problems may not show up in the hardware test.

I'll update this post with the eventual outcome. As of this moment I'm trying Permissions [1] and SMC reset (more likely to help) and I'll try run the hardware test tonight in loop mode.

Update 12/18/10: The hardware test worked. The problem is much better. I wouldn't say it is perfect, but clearly some of the above measures helped. The two I'd most favor are "clean out list of remembered WiFi networks" and "reset SMC". I suspect this problem is less common with 10.6 and that, at least with 10.5, there are many contributing factors to "wake from sleep" failure. The fixes I made reduced the frequency, but it will still occur.

By way of comparison, there were NEVER any issues with the pre-hibernate sleep mode of 10.3 and MacOS Classic, but XP sleep mode is completely unusable. (I've no experience with Win 7 sleep.)

Update 6/17/11: This problem resolved. I think it was finally fixed in 10.6

See also:

• Gordon's Tech: Things to do if your Mac doesn't boot (2/2005)
• Gordon's Tech: My 10.6 iMac is crashing - a debugging exercise (2/2010)
• Inquirer confirms apple macbook pros have nvidia bad bump material (12/9/2008)
• Can’t Wake MacBook Pro from Sleep (5/2008)
• How to Turn Off Safe Sleep
• Set newer portable Macs' sleep mode (return to the good old days - 10/2006)
• Tutorial: Dealing with Wake-from-sleep issues | MacFixIt - CNET Reviews (3/2005 - problems were more common then)
• Solving Macbook Wake from Sleep issue | Tancredi
• Computer sleep vs hibernate | Scott Gruby’s Blog 12/13/10

-- footnotes

[1] OS X has reams of permission related issues. I have never, however, seen repair permissions help with any of them, much less anything else. It always finds things to "repair", but the "repairs" fix nothing. Repair Permissions is the OS X equivalent of a disconnected thermostat. It's there to distract the customer. Still, when all else fails, I suppose it's worth trying. Onyx will conveniently run Repair Permissions.

[2] Safe Boot, loop mode hardware test and zap PRAM all need a wired keyboard. There are terminal workarounds for Safe Boot.

[3] An Apple identified cause of wake from sleep issues - Intel-based Macs: Resetting the System Management Controller (SMC): "A portable Mac doesn't appear to respond properly when you close or open the lid." I have noted that sometimes the green light on my magsafe connector doesn't come on ...

[4] An infamous Discussion thread on MacBook Pro problems is a good source of ideas.

[5] I had at least 20-30 in my machine. I removed every one.

[6] I have done this in the past, but not on this machine.

[7] I had a leftover version from when I used a Win XP Retrospect Pro backup server. I had to find an old installer to safely uninstall this low level app that runs on startup.

Picasa web uploader - Google Data Freedom scores

Credit to the data freedom team at Google - the new OS X web uploader can also download albums and even ALL albums - including video.

Does any other free image service allow such easy download of "all albums"? (I pay for extended storage, but most don't)

Onyx finds a directory problem

Joel Barriere's Onyx troubleshooting utility for OS X is free. It comes well recommended, so when my main machine was misbehaving I tried it again.

This time the Onyx startup test told me I had a drive directory structure problem. I was a bit surprised; It's been years since I ran into something like that. Onyx recommended I run disk utility from my 10.6.2 installer disk.

I was lazy, so I tried safe boot instead. Safe boot won't work with a wireless kb, so I plugged in a wired kb and held the shift key down on restart. Surprised again -- it didn't work! I thought safe boot would run fdisk if it found an error, but I guess not.

So I gave in, found my OS X installer disk, and ran Disk Utility from there. It found a missing directory item and fixed it. The behavior has changed over the years, now if Disk utiliy fixes an error it runs a second time to look for uncovered problems. It was fine, but I ran it a third time because I'm that kind of geek.

I don't know if this had any relationship to my problems. My iMac's screensaver uses images hosted on a Time Capsule external share; for some reason it's been dropping the wireless share and failing to reconnect. This is a pain to debug, because there could be multiple causes and because, even at the best of times, OS X doesn't recover from dropped shares very well.

Still, I'm glad to have found something fixable. I made a donation to Titanium Software. I meant to give $10, but I didn't pay attention to the denomination so I gave 10 Euro (about $13 or so).

I wonder if Onyx will be sold in the OS X App store come January. Since it's always been donation-ware (free) that would probably cause some complaints, but this is a venerable and well regarded utility. Barriere surely deserves some revenue.

Aperture Projects descriptions and iPhoto Event descriptions - what happens on iPhoto migration?

One of the problems with migrating from iPhoto to Aperture has been that all of the descriptions associated with iPhoto events and albums are lost.

In Aperture 3, however, projects now have descriptions ...

Working With the Project Info Window | Aperture Users Network

The Enter description field is a place to add descriptive text to add to your Project. This is a great concept, which we’ll talk about more in other tutorials, but essentially it’s designed to hold text to describe a whole Project as opposed to individual images in the Project. You can include any information you’d like here, and it’s searchable. For example, let’s say that you were doing a shoot for a client, but you didn’t want to add the client’s name to the metadata of your photos. You could add that to the Project description field and search for that later on.

Aperture 3 projects now behave like iPhoto events, they can be created automatically (fixed 1 day interval, less flexible than iPhoto). There's a good description of the project description field in this support thread. Along the way rwboyer mentions using a one page book template with a large text area for notes associated with project.

This is encouraging. So has Apple finally addressed a major issue with migrating from iPhoto to Aperture? Will event descriptions be saved in project descriptions?

I couldn't find any documentation on this, but it was easy to experiment. I created an iPhoto 8.1.2 Library with an annotated Event and imported that Library into Aperture. The iPhoto Library became an Aperture folder, and the iPhoto Event became an Aperture Project.

After I did this, the Aperture Project description field was ...

...

empty.

I would love to see Apple sued for false advertising when they claim Aperture 3 is a natural migration path for iPhoto users.

Why can't OS X restore dropped network shares?

This was written in 2004 ...

Dropping Network Connection In Sleep Mode - Mac-Forums.com

... I am running OSX 10.3.4 and really want to know if there is any way to put my computer to sleep without dropping the network connection"...

Actually, I remember that 10.3 wasn't too bad at holding onto network connections. 10.6.5, on the other hand, couldn't hold onto a network share if its life depended on it.

It's been such a longstanding problem few geeks have the energy to complain about it any more.

Maybe the OS X network team should study how Windows 95 did things ....

MarsEdit 3.1.3 - Still not Windows Live Writer, but definitely worth the money

I've been evaluating the MarsEdit OS X blog writing tool since November 2004. In 2007 I gave it a good try, but it fell short. Finally, in May 2010 I bought it, but by July 2010 I threw it out. It didn't measure up to my lost love - Windows Live Writer [1].

Six years is a long time to evaluate a product, but not as long as I've been fighting with Google's incompetence. Evidently billions of dollars aren't enough to create a rich text editor that knows the difference between <div></br> and <p></p>. This past October Blogger broke me. While I researched alternative hosting solutions, I decided to give MarsEdit another try. By this time it was at version 3.1.2.

MarsEdit is now good enough. Try it, buy it ($40).

No, it's still not the equal of Microsoft's free Windows Live Writer, but WLW was one of the finest pieces of consumer software created in the past five years on any platform. WLW is one hell of a high standard. MarsEdit is now about 60% as good as WLW, and that's more than good enough.

The key to succeeding with MarsEdit is not to mix MarsEdit posts with any Blogger product. Don't use Google's BlogThis! to create a draft post, use the "Post with MarsEdit"/Blog This bookmarklet that comes with MarsEdit. Don't touch your posts with Bloggers pustulent editor, open them with MarsEdit [2]. When you paste text, always use "Paste and Match Style" (sure wish that didn't strip out URLs though).

You can alternate editing a post with MarsEdit and WLW by the way. The post formatting will not be injured.

There are several bugs with the rich text editor portion of WLW. I don't see them very often because I've learned workarounds. Sometimes you just have to switch to the HTML editor to remove <div> tags that seem to confuse MarsEdit -- though if you stick with WLW (Windows) and MarsEdit (OS X) you won't run into this problem. There are bugs with <blockquote>, it helps to include an extra line before the paragraph.

The good news is that you can learn to work around the bugs, and there's an end to them. Honestly, I rarely notice them any more. I notice more the rich text editor's lack of keyboard shortcuts or menu bar icons for commonly used commands. Daniel Jalkut is actively working on MarsEdit though, and I think they'll be there in the next release.

MarsEdit has made my life better. It's not perfect, but it improves. Within a year or two I bet it's 75% of WLW, and that's way more than good enough.

[1] Lost because I'm pure OS X at home, and because Microsoft has abandoned the product. Since it was about perfect though, it will be a fine tool for years to come. They've relabeled a "2011" version, but really there have been no changes.

[2] Ok, so I do open quite old posts with Blogger's editor.

Shazam pricing surprise - an unwanted upgrade to worthless

Contrary to what's written here, my free version of Shazam was upgraded to a pay version ...

Shazam set to limit free iPhone app to 5 songs per month | Edible Apple

... until recently, Shazam was free and allowed users to identify any number of songs they wanted.  But now, the development company behind Shazam is saying that new downloaders of the app will only be able to tag 5 songs a month.  Users who want to get more use out of Shazam can opt to pay for a $5 pro version of the app, which will enable them to tag an unlimited number of songs in addition to providing them with new features such as music recommendations and enhanced search features. If you’ve already downloaded Shazam, then you can rest easy as the new pricing structure won’t affect you...

Except it did affect me.

I wouldn't mind the transition if iOS app updates were less automatic. I'd simply stay with the old version. The new version is really just a demo app.

Unfortunately iOS App updates are almost automatic, so there's a lot of room for these unwanted contract changes.

We're moving to software rental fairly quickly.

Facebook: What happened to hide all posts from this game?

Facebook now allows us to download all of our personal information in one zip archive (Account:Account Settings: Download Your Information). That's good.

On the other hand, I realized today I can't block game posts any more.

Until recently when I moused over a Facebook "app" generated game posts generated by a friend's game use, I would see a small "X" to the right of the post. Clicking on that produced a list of options like:

• Hide this post
• Hide all by Joe
• Hide all posts from this application (roughly)

Now I only see the first two. I can either hide a single post (pointless), or I can hide all from "Joe" (not what I want). I can't hide all posts from, say, "farmville".

Now I'm seeing Facebook App posts in my news stream - for the first time in a year.

Games are how Facebook makes its money. Even so, this surprises me. I wonder if it's a technical glitch ...

Update: Some game related posts from today still have the "Hide All ..." option. I wonder if some app vendors are trying to bypass FB's "Hide All" feature.

Update 11/21/10: Per comments this has been going on for weeks. I don't think it's a glitch. Strange that there's been abundant coverage of FB minor events, but none of this much bigger change. I'm going to have to start hiding all posts from my friends who use FB Apps that publish activity.

Why you may want to wait on the new Google Apps services - identity collisions

Google is making most of the apps that were once available only to gmail accounts available to Google Apps accounts.

Behind the scenes, what this means is that they're consolidating identities across the Google enterprise. They'll force the transition in early 2011, but at the moment it's optional.

Sounds great, but there's a catch for some users. The catch is that for some users there will be an identity collision, and reconciling that collision won't be pretty. These users should let others go first.

The users who are in trouble are those who used their Google Apps email address to register for Google services. For example, they used jgordon@kateva.org as the email address to create a Google Account. (Yes, it's confusing. Not all Google Accounts are associated with Gmail addresses.)

After doing this, it's possible to use Blogger (for example) with the user name "jgordon@kateva.org". The password may or may not the same as the Google Apps password associated with jgordon@kateva.org; until this transition the accounts were completely separate.

Next year though the accounts will merge. In this case "jgordon@kateva.org" will be belong to only one identity, an identity managed through Google Apps.

Merging identities is difficult. For example ...

Early adopter phase FAQ - Google Apps Help

...If your users used Picasa Web Albums with personal Google Accounts, they will not be able to reuse their old Picasa Web Album display usernames. They will have to sign up for new display usernames....

... The following Google products don't work with Google Apps accounts that have transitioned:

• Google Extra Storage
  
• Health
• PowerMeter
• Profiles

Review the options for transferring data between accounts.

At this time, these options are limited. Your users can transfer data in some applications from their personal Google Accounts to their Google Apps accounts after the conversion. There are no administrative controls for data transfer at this time. Each user will have to decide what to transfer and initiate the process. Learn more about transferring data in the Google Accounts Help Center.

Learn how to access two accounts at the same time in the same browser.

Users will no longer be able to access multiple Google accounts in the same browser unless they add an optional feature called multiple sign-in. See Using multiple accounts in the Google Accounts Help Center for more information.

Make sure your users complete the readiness checklist for users before they transition their accounts.

Note: After the transition, sign in to your admin control panel using your Google Apps account (example@domain.com). You might also need to use a separate browser. See Signing in to your control panel for more information.

and

Resolve conflicting accounts - Google Apps Help

Users with conflicting Google Accounts can easily resolve their conflicts by renaming their personal Google Accounts, and the data in their personal accounts will remain safe and accessible to them. Here’s how a user can rename their personal Google Account:

• Step 1: Visit www.google.com/accounts and sign in with your personal Google Account
• Step 2: Click ‘Change email’ under ‘Personal Settings’
• Step 3: Enter a different email address where you can receive mail, enter your password, and click ‘Save email address’
• Step 4: Check your other email address and click the link in the verification message from Google to confirm your change

Yikes. This smells real bad. I remember how badly Google botched the transition from "Pages" to "Sites". Reading through this list I can see all kinds of bad news.

I will start the "renaming" process with the some of the problem accounts my family has, but we'll be going slow.

The loathesome Apple Magic Mouse and the MagicPrefs salve

I have only one dysfunctional relationship.

It's with my Apple Magic Mouse. I hate it, but I can't leave it. I don't swear, but today my son heard me drop the F bomb. Thanks to the Magic Mouse from Hell.

There are two parts to the curse of the Magic Mouse. One is that Apple, the company that's never made a decent mouse, was too clever with their latest failure. The other is that Apple, the company that hates its geeks, has steadfastly ignored our screams. All we want is an option to make scrolling require two fingers, but instead we get accidental scrolls that wipe work and lose context.

I have an order for a $35 Logitech Bluetooth mouse in my Amazon cart, but before I pull the trigger I'm trying the last refuge of Magic Mouse users -- the MagicPrefs menubar and preference pane. (Better Touch Tool is a similar product, but seems to have less recent development).

I don't like to install this kind of tool -- they're usually playing in illegal APIs. Indeed, I've seen rumor that MagicPrefs has trouble with 10.6.5. I've done it though, which shows my desperation.

So far all I've created one preset. All I've done is reduce touch sensitivity a bit, and change scrolling behavior:

• disable one finger scrolling (also disables scroll momentum - I miss it)
• two finger scrolling: vertical axis only
• three finger scrolling: horizontal access only
• four finger scrolling: disabled

I'll see how this works before I add more features or gestures. I have seem some atypical clicking and scrolling behaviors and I want to see if those settle down.

If MagicPrefs saves my Magic Mouse relationship I'll forgive it some minor system glitches.

Now if only Apple would fix its own $##% Magic Mouse Mess.

Instapaper: editor's picks and more

I've succumbed to the elegant web page reformatter and offline reader Instapaper. I was a late adopter, but I've drunk the Kook Aid now.

It's the technology, it's the service, it's the quality, it's the versatility, it's the content and it's the developer(s). Classy.

Instapaper works beautifully with over 130 iPhone apps including the best Google Reader client: Reeder.app .

You don't need an iPhone to use Instapaper (currently only the iPhone app generates revenue, that may change). It's great on the desktop too - it makes the most ad laden, Flash infested, page broken content a pleasure to read (and breaks business models too, but that's another story).

It also solves a problem that you might have too.

You know you do it. Sitting in a boring phone meeting**. Browsing web sites on your corporate box (or iPhone* or iPad). All's fine until you see something you really want to read. That's crossing the line though. I can't read something good and pay attention to a dull meeting. Worse, I might continue when the meeting is done -- when I should be doing real work. (I assume everything I do on a corporate machine is monitored of course.)

Instapaper solves the problem. I click the Read Later bookmarklet and put the browser down. I'll read the article, elegantly formatted, on my iPhone later. If you don't have an iPhone, you can read it with any web client through your free Instapaper account.

Instapaper is $5 on the app store with a fine free version if you, like me, distrust reviews.

Oh, and don't miss the Editor's Picks. I'd like to know where the #$! they come from. Great list of readings, and anyone can browse this list -- even if they never use Instapaper.

*iOS browser bookmarklet integration of Instapaper is painful. Apple needs to fix this one, the developer can't. It's simple to use the mail article feature though.

** My meetings are not boring. For one thing I get 60 minute meetings done in 15, for another I round-robin attendees for comments q5-8 minutes.

PS. Instapaper got top grades on every cloud service test I use.