Using SiteSucker to backup my Blogger blogs - and my extended memory

For several years I've used Teleport Pro to create local searchable and browsable copies of my Blogger blogs.

That way, if Google falls to the The Dapocalypse I'll at least have my own copy of my extended cybernetic memory. More recently Google has added the ability to export one's blog in a google-readable format, so I do that as well.

Recently Teleport Pro ran out of gas. I hit a 65K limit for its URL database. TP has great support, and the author referred me to a $165 upgrade to their professional web spider. I've been very pleased with TPP, so if I weren't (with occasional regrets) primarily an OS X shop these days I'd pay for the upgrade.

Instead I decided to re-evaluate an OS X spider I'd tested years ago: SiteSucker for OS X. It's donationware (Paypal, sigh) and a quick download with no nasty system side-effects. I'd used it years ago, but even back then my much smaller blogs broke it. I had to set it aside.

I used it to download the site that broke Teleport Pro. It's not nearly as fast as TP Pro, and it wasn't able to handle blogger's tag links (I need to contact the author) but, overnight, it completed the download of over 15,000 separate files related to about 4,000 posts occupying 560MB of disk space (clearly the actual text is the least of the content). The download doesn't include any images, they're included by reference since I constrained the spider to my blogger path.

The first time I did the download I forgot to localize my links, so I couldn't navigate internally. The localization seems to work for some links, but not, as mentioned earlier, for the tag links.

I suspect Teleport Pro is a more robust solution -- but it's XP only and it can no longer handle my blog. Site Sucker looks very promising. I'm going to try tuning it and corresponding with the author about the tag links. If it passes my further tests I'll add configuration notes to this post and I'll be making a donation (much as I dislike using PayPal!).

Entourage perspective on OS X Sync Services

Ths article by a Program Manager in Microsoft's Mac Business Unit gave me some new insights into OS X sync services. It illustrates how very hard it is to do synchronization of any sort between disparate products. Perfection is impossible, the best vendors can do is reduce the outrage rate.

Synchronization is Hell.

Andy Ruff includes a discrete link to a section of Apple's developer documentation on the Sync Services Truth Database. To the bloodied sync veteran every word in these paragraphs could be written in blood (Entourage, according to Andy Ruff, defined a schema for Notes since Apple didn't have one. I find Apple's approach to schema definitions interesting.) ...

... The truth database contains an aggregate of all the client’s records. Consequently, the truth database uses a canonical schema that is an aggregate of all the schemas used by all the clients.

A sync schema is based on an entity-relationship model similar to that used by other Cocoa technologies. Read Cocoa Design Patterns in Cocoa Fundamentals Guide to learn more about entity-relationship models and terms such as entity, property, attribute, relationship, to-one, and to-many.

You can use one of the existing sync schemas—for example, for contacts, calendars, and bookmarks— extend one of these schemas or create your own. If you extend a schema or create your own, then you need to create an entity model for your custom objects and save it in a schema format that Sync Services understands.

This format, called sync schema, is a property list that specifies details about the entities in your model. For entities, you might specify the name of the entity and names of its attributes and relationships (collectively referred to as its properties). For attributes, you might specify the name of the attribute and its data type. For relationships, you might specify the name, destination entity name, cardinality, and delete rule. See “Creating a Sync Schema” for a complete description of the sync schema property list.

A sync schema defines a template for records stored in a database whose records are of a particular type (records belong to an entity) and may have relationships to other records. Records stored in the truth database are dictionary objects with key-value pairs, one for each property defined in the entity. Each record dictionary also has an entity name property and an associated unique record identifier. The record identifier is not stored with the record but is instead used by the client and sync engine when referring to a record. The truth database can also store custom fields in a record that are not defined in the schema. For example, these fields can be used to store client information added by a device.

The truth database doesn’t store arbitrary key-value coding-compliant objects—it stores record dictionaries. Therefore, unless all your entities are dictionaries, you typically transform records back and forth between the sync engine’s record representation and your client’s object representation. However, when fast syncing, you can apply changes only to properties—you don’t need to push and pull entire records when only a few property values changed.

Because the truth database is an aggregate of all the client schemas, it can contain a lot of information that your application doesn’t care about. Your client can filter the records that it pushes and pulls in several different ways...

The more I suffer with Apple's approach to Calendars, Tasks (phhhtt!) and Notes/Memos the more I'm interested in Entourage. The problem is there's no way for a 3rd party to sync directly with the iPhone Calendar. Microsoft would have to create a complete Entourage-friendly iPhone PIM suite, and I rather doubt they're going to do that.

Update: Macintouch has a great, meaning despair-filled, thread on Entourage to iPhone synchronization. Synchronizing disparate data models is not a soluble problem. Even a deity-equivalent AI can't perfectly reconcile disparate data models.

Chapura KeyTasks for the iPhone

Chapura is offering an iPhone Task management application tied to an online "cloud" service.

If you're an Outlook user you can sync your tasks from Outlook to the "MyChapura" cloud service, and from the cloud service to your iPhone/iTouch -- even if you sync Outlook with Exchange Server. (Unlike MobileMe, which has been changed to no longer sync with Outlook when exchange server is involved.)

As far as I can tell the MyChapura cloud doesn't include any user interface for manipulating the Task data, it's strictly a store that gets around all the issues with syncing an iPhone to multiple machines and the inability to sync over the Apple calbe.

Grrr. Maybe I should have left all my PIM data in Outlook -- rather than moving it to the pathetically inferior OS X environment. (Only Address Book is in any way competitive to the Outlook PIM equivalent.)

So is Chapura going to go all the way, and add Calender, Contact and Memo/Notes support to MyChapura, with complementary apps on the iPhone? That would be a large project, but it's very much like their Palm KeySuite application.

I'm definitely interested; it's a shame Chapura doesn't have a blog I could track. Tasks synchronization alone isn't worth the risk of installing software into Outlook/XP (very unstable environments these days), but if Chapura goes further ...

Modern software development on the web

This is a well done essay on model-view-controller development that happens to also talk about the web: Advogato: Blurring of MVC lines: Programming the Web Browser.

If you want to catch up a bit on current programming styles (I'm a bit behind myself) the Wikipedia article on RESTful development is a good complement. This bit of the article is particularly helpful, especially if you know that "RPC" is often used to refer to SOAP services, that Dave Winer was a champion of SOAP and RSS alike, and that Atom was championed by Google over RSS (phew!) ...

... It is possible to claim an enormous number of RESTful applications on the Web (just about everything accessible through an HTTP GET request or updateable through HTTP POST). Taken more narrowly, in its sense as an alternative to both Web Services generally and the RPC style specifically, REST can be found in a number of places on the public Web:

• The “blogosphere” — the universe of weblogs — is mostly REST-based, since it involves downloading XML files (in RSS or Atom format) which contain lists of links to other resources;
• The Atom Publishing Protocol for publishing to blogs is considered a canonical RESTful protocol;
• Various websites and web applications offer REST-like developer interfaces to data (e.g. Flickr or Amazon S3).

Note that WSDL version 2.0 now offers support for binding to all the HTTP request methods (not only GET and POST as in version 1.1).^[6]

I think of Jon Udell as a godfather of the RESTful world, but I had trouble finding a review essay of his on the topic; his early discussions are fairly brief. I've got a very good reference somewhere, I'll add it here when I find it....

Missing JPEG preview icons restored by Onyx rebuild LaunchServices

This is probably not a new 10.5 bug, now that I know the fix I can find mentions of similar problem from five years ago.

OS X 10.5 Leopard will, like XP's thumbnail view, show a preview of a JPG, PDF or other image file in place of the large icon.

In theory.

In practice it's prone to losing the preview and just showing the generic icon. As per this discusison thread and an older thread that has the real advice the OS X launchServices database is fragile and often broken. (For example.)

One fix is to run a fairly complex terminal command that rebuilds that too fragile data source. Another is to download Onyx and run the rebuild launchServices and cleanup scripts (remove caches) then restart.

That restored my icon previews.

Another few hours lost to complexity collapse.

I'd really like to know why Apple hasn't either fixed this ancient bug or included a launchServices rebuild in the monthly maintenance script OS X runs.

I sent Onyx a $10US donation. I've used this app long enough to contribute something.

Using my business card as iPhone wallpaper

I have the world's dullest iPhone wallpaper -- my business card with my phone number and personal email written in big ink letters.

I used to have a photo of the dog (that way the kids don't accuse me of favoritism), but I realized that if the phone was lost there was nothing to point to me. I set the phone to lock after a few minutes without use, so anyone finding it can't get through to my phone list (for example).

Palm devices, and probably most devices, have a security option to show a user-defined text screen when locked. Not so the iPhone! [1]

So I used my Griffin Clarifi macro lens and took an iPhone picture of my business card, with annotations. Then I set the business card as wallpaper. I framed it so the key information would be easy to see.

So if my phone is lost, it's now easy for someone to return it to me.

I'll try making a better picture with my dSLR, but really the card picture is quite readable even in the wallpaper version.

[1] Yes, I too would like to lock the iPhone team in a room with a Palm Vx for a month until the good bits of that device/software package were permanently burned into their brains.

Has iPhone 2.2 created a new class of application update problems? Remote is broken now.

At first we thought 2.2 had fixed an old problem ...

Gordon's Tech: iPhone 2.1 - unknown application 0xE800002E - A FIX!

...Update 11/22/08: The Nov iTunes and iPhone releases may have squashed this one. I can't speak from personal experience since I long ago fixed my problem, but I'm seeing positive reports....

Now I've had to manually remove and reinstall 2 applications, MPR and Remote, to get them to work - partly.

Just like the old days, except now there's no application error.

Remote is no longer showing me controls for my remote airport express speakers. That's a slight improvement, before the forced restore it was just giving me an "unknown error" message when I tried to use it. I think that's tied to the airport express problem.

This is not looking so good.

If it is a return of the old horrible bug with application updates, I suspect the return may have root in Apple's broken identity management system. It broke with the .mac to .me transition for those who had been original .mac users but were not users at the time of the .me transition ...

Update: I remain suspicious that there's a new bug, or a twist on an old bug, with propagating app updates to the iPhone. On the other hand, the loss of remote speakers was due to my Airport Express going offline. When I power cycled it I again saw the remote speaker option.

Why did the AE go offline for the first time in a year or so? That's another question, maybe related to an update to my Airport Extreme, maybe coincidental.

Update: Yep, I can't sync my iPod now...

Update: Deauthorizing and reauthorizing my Mac had no effect. I found a slightly relevant discussion group thread. I rebooted the iPod to no effect, but then restarted my Mac and was able to sync all my devices.

Update 11/24/08: Rebooting the Mac solved the problem. Alas, that was too easy a solution for this to be entirely an Apple error. I have a bad feeling that the error was intracranial, that is, my user error.

For reasons too complex to bother explaining it's just barely possible that I was running a user account that points to a shared multi-user Tunes library.

I've also verified that at least one iPhone App Store update worked as expected and functioned without my having to remove and restore the app.