Pinboard and IFTTT - credential outage

I was quite pleased with my jury rigged replacement for my long lost Google Reader Shares. (Few remember them now, the net years have passed like leaves in the wind ...)

I had IFTTT tracking Pinboard posts and creating Tweets and posts to an archival WordPress blog. The Pinboard posts were largely created from the essential iOS Reeder.app and, less often, from Google Reader and Reeder.app for OS X.

It worked. I was happy. I started composing the celebratory blog post and explanation when...

Yes, you can guess. It fell apart.

I don't know why. Pinboard had blocked Superfeedr around that timefor aggressive crawling, but this turned out to be a red herring. IFTTT doesn't use Superfeedr.

IFTTT hasn't gotten back to me, but they're a free service so support is not predictable.

Well, I did know this was going to be a fragile and shortlived solution! I particularly didn't like that IFTTT is "free". I'm now looking for an alternate solution I can build around Reeder.app and Google Reader with Twitter and WordPress as outputs.


Update 6/19/2012: I didn't hear back from IFTTT support, but Pinboard convinced me to dig deeper there. I tried creating a new task but that also failed. The clue was in a screen I'd not visited, the Channels screen.

There I found this icon:

I edited it, reentered my credentials, and it's no longer offline. Now I'll see if it starts working again.

What happened? Well, there are two possibilities, and without help from IFTTT I don't know which is true. One is that Pinboard was unresponsive and IFTTT took the channel offline, but then never restarted it.

The other, which is at least as likely, is that I changed my password to Pinboard (I do that sort of thing) and that any email notification of subsequent credential failure was either misread, mislabeled, or filtered out. IFTTT doesn't use OAuth or similar service with Pinboard, so it needs my credentials to work. If that was the problem then IFTTT might be able to come up with a better way to notify users. For example; there was nothing on the IFTTT task screen to tell me a channel was offline, and when I manually triggered my tasks there was no error message.

Update 6/19/2012b: Fixing the Channels problem did the trick.

Thinking this over, why does IFTTT need my Pinboard credentials anyway? They are reading from a public feed, not writing to my account.

Update 6/22/2012: The latest IFTTT rev has much improved diagnostics. 

Mac OS X Mail: Import Mailboxes from a different user account or computer - and Old Netscape email import

It's been a long time since I've been impressed by Apple desktop software.

Partly that's because I have high expectations, but it's also that I'm not Apple's target demographic. I prefer capability over ease of use. I'm one of the very few people who likes iTunes for example -- because of it's very powerful playlist query language. I'm not looking forward to Apple's overhaul -- because I'm certain they'll remove the features I like.

Been a long time, but today I was impressed. I'd just completed a Eudora to Mail.app conversion in a user account I created to test the import process. 

The process went well, so I planned to repeat it on my primary account. I wondered though, if there was a way to simply import from the folder in ~Library/Mail. I couldn't find much documentation or discussion on Mail.app's merging or important of another Mail.app database, but the UI is simple enough. I updated my backup and give it a try.

First I moved the 'Import Folder' in ~Library/Mail to the Shared folder for my primary machine so I could do the import from my primary account. Then I selected that folder from the Import UI. Mail.app scanned the folder tree and returned a list of a few hundred folders I could import from. I told it to go ahead.

Fifteen minutes later I had my archive. Unfortunately there was no way to know if I'd gotten all 55,000+ messages; all I had were unread message counts and those didn't match up (but I didn't trust them anyway).

Happily OS X Mail has a very clean message structure, one message is one .emlx file (attachments are in a separate folder) and files are organized by a Finder folder corresponding to a Mail.app folder.

So I used Get Info from the Finder:

• Original: 2.36GB for 84,871 items
• Import: 2.36GB for 84,405 items

That's not too bad, except I'm short about 466 items. Coincidentally, I recall from the EudoraMailboxCleaner process that I had 478 folders. By directly inspecting the folders in the original and the import I could see that most of my folders were missing a .plist file that was in the original. This file didn't seem to be necessary; the messages are rendering correctly both times.

So it seems the import worked.

That's good, but what really impressed me was I noticed Netscape import option. I happened to have some old Netscape archives lying around. A few minutes later and my Netscape email was in Mail.app.

Twenty years of email in one repository; 9GB, 200,000 messages - that's impressive.

Update: I wrote too soon. The message metadata displays, but the Netscape messages are all blank. (A few Eudora emails are blank too.) OS X will preview them in the file system and I can read them with a text editor. So the data is there, but Mail.app doesn't want to render it yet.

Update b: Well, that was interesting. I tried REBUILD on one of the folders. All the emails vanished. I checked in ~/Library/Mail; they were still there and still viewable. So I tried removing the plist file and then tried restore again. This time they all came back, and they're readable. But the real weird part is that now all the Netscape emails in other folders are non-blank. I didn't touch any of those.

In praise of The Unarchiver

OS X's native archive manager won't handle zip files from the late 90s. The Unarchiver just opened one of mine from 1986.

Great app. Thank you Dag Agren.

Converting Eudora email archives -- the end is near (and Norton Commander ncmail.exe in a footnote)

I have years of email messages locked up in a 1.2 GB Eudora for Windows archive [3]. Converting them has been a pending project for about 10 years. This archive is a part of my extended memory; it has the keys to remembering people and places now forgotten.

Now time is running out. Eudora Mailbox Cleaner is one of the best tools for this project, and it's PPC only. Since Lion doesn't have Rosetta, the app won't run on 10.7 or higher. [1]

I've downloaded a copy. I think I'll delay my primary machine's Lion conversion until the Eudora conversion is done.

There are other Eudora archive conversion methods, but most run on Windows -- and I think Windows VMs on Mac will fade away as Apple locks the OS down to both block malware and enforce DRM. Time is passing. If you have old archives, you may have only a year or so to (relatively) easily convert them.

I'm planning to create a user account to test the import, then backup up my current Mail database prior to adding this one.

See also:

• Gordon's Tech: Converting Eudora email
• Gordon's Tech: Converting Email from Eudora OS X to a modern format - the TidBITS review: Adam Engst's comprehensive guide

Update

It took a few hours to do the conversion; most of that was spent waiting for a bundled AppleScript to Rebuild Mailboxes. Post conversion the database is 2.4 GB and over 55,000 messages.

Follow the directions carefully. I didn't think the nicknames were worth converting until I saw them; the connections between Groups and Members turned out to have powerful memory value. I recommend testing first with a special user and an empty address book and email account. If you import into an existing Mail database be sure to backup ~/Library/Mail first.

I didn't import Filters. They'd just be noise now.

With OS 10.4 and later there's an essential 'Mailbox Rebuild' step. My Eudora had at least 478 folders across a deep hierarchy [2]. It would have been extremely tedious to try to rebuild them all. So I tried the 10.4-10.5 rebuild AppleScript, even though I am running 10.7 Snow Leopard. 

That script runs slowly, and because of the way UI Scripting work it must own the entire machine. It will steal focus. Don't try to even switch users. It took over an hour to complete. Thank Darwin it's there; the job would have been untenable without it.

As per the documentation (mandatory read):

Your imported mailboxes will be located in a new folder called "Import" at the top-level of your local mailboxes. The new files will be located at "~/Library/Mail/Mailboxes/Import" on your hard disk...

... due to the way the Address Book works, a new group will be created for each Eudora nickname which expands to more than one email address.

I've already leveraged the archive to rebuild memories of people I worked and corresponded with in the 1990s. Some have died, some I've lost touch with. My old Eudora email archive has now been reintegrated into my extended memory.

For now the email archive and my old Address Book are sitting in a distinct user account. I will decide over the next week whether I'm going to leave them there or, more likely, repeat the process. If I do a complete merge I may create yet another user account, copy both my current email archive and this archive there, and let the process run.

OS X mail has a virtually undocumented import Apple Mail Data feature. I don't think it will be practical to import the archive using that but I'll take a look at it. (Update: It seems to have worked quite well. I imported my old Netscape emails at the same time.)

- fn -

[1] A year or two ago our 8 yo G5 iMac hard drive died. I dithered about whether to replace it or not because of the infamous iMac capacitor problem and its slowly delaminating screen. (The G5 iMac was one of Apple's most troublesome machines -- they desperately needed to move to Intel. The heat problems were extreme. It was, however, extremely easy to repair. That was even a marketing feature. Perhaps the G5 iMac is why Apple doesn't design for fixability any more.)

I did replace the drive and it's still working. It can even, if I were to install 10.4, run Classic. It's now one of my most valued machines. I think I may turn our MacBook into the kid machine and put the G5 iMac on a very light duty cycle so I can keep it available for another decade. I'll also archive my software install library onto its now capacious hard drive.

[2] My Gmail archive has essentially 2 mailboxes: In and All. Full text indexing and search changes things.

[3] Created by Steve Dorner, Eudora was the preeminent Mac email software and was dominant for a time on Windows as well. This archive, during its lifetime, moved from Mac to Windows. I found some 1996 correspondence with Steve, he was a very approachable developer with a lot of correspondence!

Before Eudora I used Norton Commander Mail (ncmail.exe) with MCIMail (pre-inernet -- 4867991@mcimail.com was my email at the time.

Writing this post inspired me to look into those archives. I did have them -- zipped up from the days when 200K was a lot of data. OS X couldn't open the old zip archives, but Unarchiver did fine. I used Automator to attach a .txt extension to each file and assigned them to TextWrangler. That was enough to convince OS X to index my old NC email messages. For example:

I actually got a bunch of public domain and shareware Windows applications running SLIP under OS/2 over my 14.4K modem ... 

... The second thing I'm excited about ...) is learning HTML and creating my own home page on the SILS Mosaic connection...

...  for my next 'HOT tip'. Check out TeleScript. Ignore "General Magic" -- it's bogus. TeleScript is the secret pearl.

I didn't remember that 14.4K modems coexisted with HTML 1.x. TeleScript is huge in an alternative reality. GM was a copy co-founded by "Bill Atkinson, Andy Hertzfeld and Marc Porat". Historic names.

The next archive to crack is a set of email written using Netscape's email app. (Update: Mail imports these directly.)

Notes on converting a MacBook Core 2 Duo dual USB to Lion

Our vintage 2006 MacBook Core 2 Duo can, in theory, run Lion [1]. I just upgraded it from Snow Leopard, which it ran reasonably well albeit with lots of fan activity.

I didn't upgrade happily. There's a reason I've waited this long. I wanted to stay with Snow Leopard, but Apple's MobileMe migration was going to cause problem for Emily's Address Book/iPhone Contact sync. Yes, that was the primary reason. Sad.

I upgraded all the apps we use, said farewell to AppleWorks and many games the kids no longer use, removed all PreferencePanes and extensions, unplugged all cables, did a safeboot cleanup and two completely independent backups (one a clone). Then I upgraded.

It seemed to go well enough once Spotlight rebuilt its indices and I refreshed everyone's account. it does take a very long time to log out; Lion is saving a lot of state information and the MacBook doesn't like that.

It went well enough, that is, until my old account. Then things got slow. Even though I'd updated VMWare Fusion to the final "Lion Compatible" 3.x version, I suspected it was the problem. I tried running my VM -- that was a disaster. After power down and a safe boot I uninstalled VMWare 3 per directions.

That helped a great deal. In fact, the laptop is quieter than it used to be. I suspect Fusion has been causing problems for a while [3].

So will I try Fusion 4?

No, not on this old machine. I really don't use my VM's very often, and neither Parallels nor Fusion are sold through the App Store. To do their magic without Apple's help they must be hacking the underlying OS; and OS X is increasingly hard to safely hack.

For now the MacBook seems to have survived Lion - albeit at the cost of a little used VM I'm probably better off without and a dozen or so older games -- and AppleWorks.

So far, better than it could have been. I am, however, regretting obeying Lion's command to update my Airport firmware [2]. 

[1] That's almost six years ago! I'd forgotten how old it really was. Maybe I shouldn't be too upset it can't run Mountain Lion. [2] Lion really wanted me to undo my Time Capsule firmware regression, so I did. Now I'm seeing more problems with losing connectivity, i saw a -1 error again, and I'm again having to rebuild Spotlight indices of the backups. I have a strong feeling I'll be reverting again. I seem to be the only one with this problem though. [3] My best Fusion experience was version 2 with a Windows 2000 VM -- on that old MacBook probably with Leopard (10.5). It's never worked as well since. 

Update 6/17/2012: Logging out and user switching is much slower. It takes about 20-30 seconds to log out and 15-20 to switch. I think it's because of all the context saving Lion does; this old machine can't handle it. There are ways to disable saving of application states, but I'm going to wait a while on this one. Otherwise things aren't too bad. The machine is quieter than it has been for years, the fan no longer roars. I suspect that's due to uninstalling Fusion 3.x, but it could be a Lion improvement.

Update 6/17/2012: I went through each user account and turned off 'save and restore windows' in system preferences. Then I logged out and unchecked the restore windows on login option shown there. No logout and login is back to Snow Leopard times. Now I have to figure out what to do about #$@$ Google Software Update. it keeps popping up in managed accounts that don't have privileges to run it.

How well does Gmail archive via IMAP really work?

My Gmail account holds 64,800 messages. Of those perhaps a few hundred are of lasting interest, but by the time AIs sort out what they are we'll be long obsolete.

So, perhaps illogically, I like to hold onto the archive. That means a local copy. 

I recently had an opportunity to test how well an IMAP archive approach works [2]. I upgraded a machine I'd not used for a while and had to catch up on a year's worth of email. It took a few separate synchronization sessions to catch up, I think I ran into Google's bandwidth limits/transaction throttling behavior.

I ended up with 64,440 in my local "All Mail" synchronized folder, so about 360 have been lost in the bowels of IMAP. Perhaps they'll trickle in with future synchronizations.

The oldest message in Gmail online is dated 2/29/2004 [1], in Mail.app (Lion) it's 2/2/2004. Yes, that's weird.

To answer the post question then, I'd say Gmail archiving via IMAP is imperfect but not pointless. A C grade.

[1] It's not obvious how you can see this. If you mouse over the message count there's a hidden option to swap newest and oldest. It's not easy to get to show; navigating large email collections in Gmail is almost impossible with the "new" UI.
[2] IMAP archive is what Google's all-but-defunct Data Liberation Front recommends. I miss that gang, but there were of the Google- era. In the Google 2012 they're an anachronism.

Sparrow — How not to do marketing

Sparrow's value proposition is that it works with Google's email much better than OS X Mail.app (even Lion OS X Mail), but it also uses the OS X Address Book.

It's a great solution to problems I face.

They don't market this though. If they had, I'd have tested it out long ago.