Gordon's Tech

Obscure bug: Reeder share to Facebook will post to last used Page Identity

2012-02-04T21:38:00.001-06:00

This is an oddball bug. I suspect it's not fixable.

I used iOS Reeder to share a Facebook post.

Problem was, when I last used Facebook I used it as one of the Pages I maintain. That's how Facebook Pages work; you assume the Page identity while you post to it. Then you flip back to your own identity.

So Reeder shared my personal post on my public Page. It wasn't too embarrassing, but it was clearly wrong for the Page (org).

It's a weird bug; I suspect the flaw is in Facebook's API. It probably isn't designed for this use case. I also wonder if this is a side-effect of Facebook introducing the Timeline to my personal account.

Wiki impressions: XWiki

2012-02-04T18:46:00.001-06:00

After a disastrous SharePoint 2007 to 2010 migration [1] it was clearly time to replace my team's use of SharePoint wiki. The mangled conversion was a red-flag-and-air-raid-siren declaration that SP was going to be a longtime world of hurt.

We looked around at the wiki options, and considered TWiki, XWiki and Confluence (WikiMatrix). Atlassican Confluence is the best of breed, but for corporate users like us it's expensive. Hardware we have, platform licenses we have, networks and backups we have, and expertise we have. What we don't have is a budget.

TWiki is the easiest of the three to configure and it has the fewest technical requirements. XWiki looked like more work, but we liked the rich text editor and (limited) table support. We're now starting up on XWiki enterprise. Here are a few impressions from the process that may be of interest to other wiki searchers ...

We started out on a Linux server but couldn't get XWiki working. We're not Linux experts. We switched to Windows and our engineer resource had XWiki setup within a day. I wasn't impressed with the ease of installation.
XWiki requires a Java Servlet Container. That rules out Dreamhost and many other hosting options. (TWiki can install with Dreamhost.)
XWiki includes a blog service but there's no support for Blogging APIs. So you can't use MarsEdit or Windows LiveWriter to write.
The documentation is weak. For example, how do you delete a Space? Turns out it's easy if you have privileges, but the documentation claimed we needed an extension.
XWiki Enterprise is a geek platform, though much of the complexity can be hidden.
I like the approach to URLs and page titles. You can change a page title, but the page URL is fixed. XWiki provides an effective UI for looking up local pages during link creation. (SocialText retitling creates a redirect page with the old title/url and a link to the new title/url. Sharepoint changes the URL and title, but updates intra-liki links. Foreign links break.)
The rich text editor is simple but works fine for my purposes.
IE 9 doesn't work with the controls on gadgets, but seems ok for non-admin users.
XWiki has the universal wiki curse -- no ability to migrate posts between systems. I wonder if people who complain about being unable to move their medical records between providers see the similarity. OTOH, when our Linux install failed it was easy to move to Windows.

I'm reasonably optimistic that XWiki will work for us. I'm glad to be free of Sharepoint 2010.

See also:

[1] With SharePoint it's not possible to separate software issues from implementation. Maybe Microsoft provides wiki conversion tools that our IT department didn't use. I have discovered that if I copy/paste of our 500+ SP 2007 wiki created pages into Word, then remove all styles, then past back into SP 2010, I can edit them again.

Time Machine backups of Aperture are not reliable?!

2012-01-24T21:18:00.001-06:00

I've restored my Aperture Library from backup twice in the past few months. Two months ago Aperture crashed when a bad block corrupted a file. I restored from Carbon Copy Cloner. A month later I figured a dying iMac drive was the root cause, and my backups weren't as robust as I expected.

After a mildly painful Apple service call i had to restore all my data from backup. I backup the server nightly to Carbon Copy Cloner and hourly to Time Capsule, so I had to pick a backup source. I decided to restore from Carbon Copy Cloner, though I did consider Time Machine. It seems to have worked.

Today I learned that I might have dodged a bullet. I was wise to choose CCC over Time Machine ...

Macintouch Backup
Derek L
...Thanks to Antonio Tejada for his note about Time Machine and Aperture. The combination remains unreliable for me, as I'd described in my comment on this topic last June: not all items in my Aperture library get included in my Time Machine backup. Although I've been able to temporarily repair it via forcing a "deep traversal", the problem recurred in fresh TM backup sets, on multiple disks, and through several point releases of Snow Leopard and Aperture 3 (I've never used any other version, and I also haven't migrated to Lion). I gave up on it and instead depend on Aperture Vaults.
Skot Nelson
... However, after having Aperture randomly lose some of my masters -- old photos from a concert that I hadn't touched in month and just happened to click on as part -- I no longer trust my Aperture library's integrity...
Richard Tench
... Seeing the warnings here about Aperture and Time Machine, I decided to do a test recovery (to my desktop). It failed due to permissions on the Time Capsule. Though one would think that's an easy problem to fix, it isn't. At the end of 2.5 hours on an AppleCare call, they were unable to help me. I was told that Aperture doesn't work well with Time Machine...

Sigh. I miss Dantz Retrospect. I'm grateful for CCC, but old Retrospect combined the reliability of CCC with the features of a true backup.

It's been noted that in 2012 Apple OS X development is the equivalent of Siberian exile. Time Machine work must be reserved for unwanted interns.

See also:

Backups - why you need two methods and abundant paranoia 12/2011
Freeing up Time Capsule space – and documentation for Time Machine and Time Capsule 12/2009 (Time Machine is flaky)
Lessons from Apple Store out-of-warranty repair of a Seagate drive 1/2012
Lion continues to disappoint: the Duplicate/Save problem 1/2012
7 facets of a good Mac backup strategy « Haystack Software Blog 1/2012 - advice from the developer of Arq (backup software for S2) -- but absolutely correct. I do this and I'll evaluate Arq.

SharePoint 2010: migration of a SP 2007 wiki

2012-01-23T15:01:00.001-06:00

My business group has maintained a large SharePoint 2007 wiki.

Recently we had to migrate to SP 2010. The wiki went along for the ride.

I've seen some bad outcomes in software over the past 20 years. I mean -- really bad products.

I've never seen anything like this.

Wiki pages that look like they were edited by intoxicated baboons. Pages that can't be edited in IE 9, but can be edited in Chrome. Pages that can't be edited in Chrome but can be edited in IE 9. Pages that can't be edited anywhere, save as HTML.

SharePoint Designer is an option. I tried it. I shed tears for the lost spirit of FrontPage and Vermeer.

I despair, sometimes, when I see what Apple's interns are doing to iPhoto or OS X. Even they, even they, have never done anything like this.

This is the worst.

Microsoft, astonishingly, is truly finished. We really are in the twilight of the personal computer.

My MacBook fan was roaring. Again. Time Capsule edition.

2012-01-22T16:25:00.001-06:00

My MacBook fan was roaring. Again.

This time, however, activity monitor didn't show much going on. It wasn't a Flash ad running in another user account. I didn't have a print job stuck in the bowels of Apple's dysfunctional printing framework.

It had been doing it for weeks. Sluggish performance, slow fan pinup, then continuous running. Something was draining performance and making heat -- and it wasn't showing up in Activity Monitor.

Or, at least, it wasn't obvious in Activity Monitor. I did see something called fsck_hfs using up 10% or so CPU.

To make a long story short - the problem was a defective Time Capsule backup. My MacBook was running fsck_hfs, a utility that "verifies and repairs standard HFS and HFS+ file systems". When I saw this I thought there was something wrong with my system drive - but it tested out fine.

A SuperUser tip clued me in. My MacBook was trying to verify the integrity of my 150GB backup -- over a WiFi connection. This is a singularly ineffective strategy, it would have taken days to complete. The laptop never ran continuously for that long; and there is something about this process that makes a MacBook run hot [3]

If I'd known what to look [1] for I'd have seen something like this:

Except it would have shown 2-4%. It only got to 92% when I connected the laptop to my TC via wired ethernet. It got to 92% then stuck there. The backup was toast.

I then disconnected all users and tried deleting the .img file for my MacBook from TC. That failed because the Finder can't handle TB scale image files [2]. I then used AirPort Utility's very confusing Time Capsule UI to erase the entire drive and started the slow, painful, recreation of my backups.

My MacBook is quiet and responsive again, and fsck_hfs free.

That was painful.

Backup is an unsolved problem.

See also:

My MacBook fan was roaring. Why? 5/2008.
Obnoxious old Epson Scan bug: EPSON Scan cannot be started 2/2008
OS X 10.5.5: CPU pegging with Firefox 9/2008.
When Safari locks up - Kill Flash process
iMac G5: fans running more often 10/2006
osx - Why has my Mac been running fsck_hfs for two days now? - Super User - great discussion of this problem
Time Machine. Backup failed with error...: Apple Support Communities 9/2008

[1] I wasn't so direct, instead I wandered about searching Console error messages. There I found:

search console for com.apple.backupd (all messages)

Attempting to mount network destination using URL: afp://;AUTH=No%20User%20Authent@Molly.local/Molly_Internal
Backup failed with error: 21
Error writing Time Machine Information file: /Volumes/Molly_Internal/Stanford_MacBook_0017f2f04828.sparsebundle/com.apple.TimeMachine.MachineID.plist
Error writing to backup log. NSFileHandleOperationException:*** -[NSConcreteFileHandle writeData:]: Input/output error
Event store UUIDs don't match for volume: Escanaba

While plumbing Console I discovered an unrelated error that distracted me for a while - a pile of errors like this:

Jul 20 18:27:55 localhost com.apple.launchd[1] (com.apple.SystemStarter): Failed to count the number of files in "/System/Library/StartupItems": No such file or directory

Googling on this one I discovered this is an old OS X bug, one some systems a routine update deleted this folder. I recreated it using terminal:

cd /System/Library
sudo mkdir StartupItems

That cleared up a bunch of Console error messages, but it didn't fix my real problem.

[2] Kind of amazing, but there you go. Apparently they can be deleted via terminal, but erasing through TC is safer. You can only erase the internal drive, if you want to erase a TC mounted external drive you need to move it to a Mac and erase it there.

[3] The MacBook runs hot whenever it does an 802.11n TC backup. I wonder how much of this is heat output from the WiFi/encryption systems.

Organizing kid school accounts with OS X: Chrome to the rescue

2012-01-22T12:52:00.001-06:00

In the twilight of the general purpose computer, I struggle to balance OS X and Apple tech, Google services, parental obligations, and getting work done.

Our iOS and OS X devices are parental controlled -- at least as far as they can be. Among other things, that means Google services are unavailable on child accounts. [1].

Schools, however, make increasing use of Google Apps [2]. This is how I reconcile that use case with our general approach to home computing:

You need the username and password for the school Google Apps account. Example: kid_name@school.mn.us.
Create a single non-controlled "homework" account on the primary homework machine.
Use Google Chrome, not Safari, for this account.
In Chrome create a user account for each child. For each account, from Chrome Preferences, choose to sync Google. You will be asked for the school user name and password.
Add gmail, docs and so on to the toolbar.

Each child uses this single OS X account with their own Chrome identity. Use of this account requires direct parental supervision. It is used only for homework. On personal OS X accounts our kids don't directly access our Family Google Apps domain, they use OS X Mail.app, for example, to get email. They don't know their Family Google Apps passwords.

[1] Partly by design and partly due to market disinterest, Google services are not compatible with OS Parental Controls.
[2] Alas, this transition occurred even as Google's Hyde crushed its Jeckyl.

OS X shared printing: Is it just me?

2012-01-22T11:45:00.001-06:00

Is it just me, or is OS X shared printing broken beyond all repair?

I spend an inordinate amount of time dealing with print failures on Leopard and/or Snow Leopard shared Brother USB printers using Apple provided drivers. The only reliable printing I've had with OS X was with an ethernet connected network printer. [1]

Yet another example of why general purpose computers are dying (and I've a lot more to say on that ... later).

[1] The very best, most reliable, printing I ever experienced was using Mac Classic, AppleTalk, and Apple's LaserWriter Select (not sure of product name) in the early 1990s. I'm planning to donate our Brother HL-2140 and buy a Brother networked printer.

Lessons from Apple Store out-of-warranty repair of a Seagate drive

2012-01-21T14:36:00.001-06:00

Mysterious application crashes, including corruption of Aperture file data, exposed bad blocks on a slowly dying 2 yo iMac drive. Modern drives aren't supposed to show their bad blocks, and of course they should never corrupt a file. So I knew the drive needed to be replaced.

The 27" iMac 11 (i5) is not user-serviceable. It can be done, but it's hard to keep dust out of the display. So I compared repair costs at FirstTech in Minneapolis to the Roseville MN Apple store. FirstTech's service costs were about $200, but Apple's service fee is a flat $40 (offsets the high cost of their proprietary designs). Both charge far more than retail for a hard drive. Apple dinged me $226 to replace a 1TB Seagate 7200.12.

The replacement drive is an ST31000528ASQ revision AP24 S/N 5VP9Z4TC. I was told it had a 90 day warranty. A comparable new drive from Amazon is $134, so Apple adds a $100 markup on the drive. (Of course I'd have preferred a 2TB replacement, but Apple out-of-warranty repairs follow warranty rules -- so no upgrades.)

Overall the Roseville MN Apple store experience was mediocre. Some of this is because of Apple policies, but mostly this store is bursting at the seams at this time of year. Apple needs more retail.

If I were to do it again I'd order replacement parts from a quality Mac after-market vendor and I'd do the repair myself, or I'd order the part and pay FirstTech to do it. When an out-of-warranty Apple repair makes sense, I'd look for a quieter Apple store.

Here's what I learned for future reference:

Apple wanted to keep my old drive. They can probably get money from Seagate. This was a problem because I hadn't secure-wiped the drive, and although my passwords are on an encrypted image I prefer not to have our family data floating around. I also wanted to reformat and stress test it myself, and decide if I could use it as an emergency store. Store management wrote that "I needed the drive for data recovery" and that let me keep it. (Not true - I had 2+ complete backups.)
I thought I was told the drive would cost about $160, but in retrospect that was probably my mistake. $160 would be low markup. In any case, get estimates in writing if possible.
When I went for an estimate I was told to expect a 3 day turnaround. In fact it took about 9 calendar days -- including 3 days to repeat the drive test and confirm the bad blocks. It took so long it ran into a business trip; the store was grumpy about storing it until I returned.
They expect to have a guest or maintenance account available for testing. I had none on this machine. They were able to test anyway of course, but this is worth knowing.
The machine had additional memory added, but the repair receipt listed the original memory. This was mildly worrisome but it came back with its bits.
The installed drive was formatted with the same OS as the old one - Snow Leopard.

I tested the drive with Tech Tools Pro. The SMART check on the prior drive showed no problems except out-of-range temperature variation. The SMART check on the new drive also shows out-of-range temperature variation! I hope this is a quirk of TTP and the iMac's thermal regulation system rather than a drive problem. The block scan passed 1,953,525,168 blocks, 0 bad. (Although no bad blocks should ever be exposed, the good block total will fall over time.) The overall SMART test also shows a tendency to read errors, though still within normal limits (Hardware ECC recovered, Raw Read Error Rate).

Lastly, I chose to do my restore from a bootable Carbon Copy Cloner backup rather than my Time Capsule backup (I trust CCC more). I'd never done a full clone restore on a Mac; it worked well but there were a few quirks...

During my initial testing I'd created a user account on the new disk. The default CCC restore would have left those files in place -- which is an abnormal install state. Also, the default restore seems to leave "more recent" files untouched, which might produce a mixed version system.
I set the restore to overwrite "more recent" files and to move non-matches to an archive folder.
After the install completed I deleted CCC_archive on the target system
After the restore it appears TC is backing up my entire system. That will take a week over the home wireless.

For extra insurance I'm going to leave my CCC backup untouched and rotate it offsite. My routine backups will be to an older CCC backup and to Time Capsule. In my initial testing however the restored data appears fine. I also have my original drive which is fully readable, I'll wait a month or so before I wipe that drive and stress test it.

Update: Time Capsule is trying to backup 340GB over WiFi; it's doing a full backup. This will take weeks. I've set my TC backup to omit all but Users, tomorrow I'll bring it to the computer and connect by ethernet.

Lion continues to disappoint: the Duplicate/Save problem

2012-01-16T09:13:00.001-06:00

I have a document open in Pages.

It's saved somewhere.

What is its name? Where has it been saved? If I "Duplicate", how do I find the "Duplicate"? What is the name of the "Duplicate"?

Apple's change to Lion's file management behavior is a significant hassle. Macintouch has a detailed list of complaints from numerous Lion victims.

It's worse than a hassle though. It's a scary sign of incompetence. If Apple's developers are capable of this level of judgment failure, what else might they do?

Lion to Vista comparisons are well deserved. I may keep Snow Leopard on our household machines indefinitely, which means I won't be doing much with iCloud. It also means I'll delay upgrading our hardware as long as possible.

NYT's Tedeschi misses the iOS Porn story

2012-01-15T07:51:00.001-06:00

Sadly, Bob Tedeschi, who should know better, missed the big story in his NYT article on Porn and iOS parental controls.

When he wrote that it was "an hour's work" to secure an iOS device I almost snorted coffee out my nose.

This is the article I wrote him:

re: Safeguarding a Child’s Mobile Device From Pornography
Bob, I'm surprised you missed the truck-sized loophole in Apple's iOS Parental Controls. Alas, by missing it you came to precisely the wrong conclusion.
It is not 1 hour's work to secure an iOS device. It is almost impossible.
The loophole is embedded WebKit. Disabling Safari does not disable WebKit.
Almost all free Apps, and many commercial apps, include links that will, when clicked, bring up an embedded WebKit browser. From that browser it is often only a few clicks to anything.
For example, my 15yo showed me how he could use the links on this travel app to bring up wikipedia, and from there Google.
Almost all iAds, and all Google platform ads, use Webkit.
This problem is common in apps that are rated for children.
The solution is simple. Apple should provide an option to block Webkit use as well as Safari use. They haven't done this because they aren't feeling any pressure, and their ad platform is already doing poorly.
I am sorry you missed a golden opportunity to put some pressure on Apple, but I hope in a follow-up article you might mention this.

You can write him too.

Update: I was amazed and impressed to get a personal response to my email - on a Sunday night! He's verified the issue and is now researching it. Wow. I am a fan.

iPhoto 9.2.1 to Aperture 3.2.1 - it doesn't actually work

2012-01-12T20:38:00.000-06:00

Apple promotes Aperture's seamless import of iPhoto content.

I've been skeptical, but I gave it a try on a plane flight. I added 21 images to a brand new iPhoto 9.2.1 Library and I created albums and events. I then gave descriptions to images and to both albums and events. Then I imported the images into Aperture.

The iPhoto.Events became Aperture.Projects. The iPhoto.Albums became Aperture.Albums. iPhoto.Folders became Aperture.Folders.

Eventually. At first the iPhoto.Albums were missing. They showed up minutes later on reopening Aperture. This took so long it feels like a bug.

Image metadata seems to have been preserved - titles, captions, etc. I've written about this previously for iPhoto, Aperture and Picasa Web Albums.

That's the end of the good news. All of the descriptions I added to Albums/Events were lost. Aperture Projects/Albums can't have annotations. So that description you wrote in iPhoto about the family reunion? It's toast.

Aperture's iPhoto import is feeble - and Apple's marketing of Aperture's iPhoto import is deceptive.

Apple does stuff like this though. I'm not surprised they did a crappy job on iPhoto import.

What's truly weird however, is that nobody besides me seems to care. That means Apple isn't going to fix this.

There are times when I know I live in the Twilight Zone. This is one of those times.

See also:

Gordon's Notes: Lessons in Software: Aperture 3 and iPhoto 11 10/2010
iPhoto '08: don't embed metadata in JPEG originals 12/2007
Stuck in Apple's photo management Limbo 5/2011 - still stuck today
Apple Aperture: iPhoto replacement?: 10/2005 - seven years later, it's still not a full replacement
Migration of metadata from Aperture to iPhoto and Google's Picasa web albums

How to cancel or reschedule an Apple Genius Bar Reservation

2012-01-10T13:39:00.000-06:00

I don't think you can currently cancel or reschedule a Genius Bar Reservation from Apple's GB reservation site.

If you have iOS though, you can cancel or reschedule using Apple's Apple Store.app. It's not obvious how ...

Tap Stores
Find store where you have reservation.
Click Store then Genius Bar. From here you can create a new appointment, or cancel or reschedule an existing appointment.

I don't think there's an Apple Store.app equivalent for the desktop, so, like iMessenger, this is strictly an iOS service.

OS X opens Aperture every time I start

2012-01-06T21:38:00.001-06:00

Every time I logged into my Lion machine, Aperture started up.

I checked the Login items option on my user account. Nothing there.

Then I figured it was a bug with OS X 10.7 Lion resume. I deleted all the saved states, including Aperture's (Delete Specific Application Saved States from Mac OS X 10.7 Lion Resume).

Didn't help.

Finally, something clued me. This wasn't a new Lion problem, it was the old 'launch Aperture when iPhone connected' bug. Same thing can happen with iPhoto or Image Capture or Preview or "Auto Importer". This particular machine is connected to a USB hub that had some iPhones attached.

I don't know the proper place to control this peculiar OS X behavior, but I do know it can be controlled through Image Capture. I opened that app, and clicked on the iPhones icons on the left side. For each one I set 'Connecting this iPhone' to 'No application'.

Problem solved.

10.7 Lion: Automation and AppleScript

2012-01-03T12:33:00.000-06:00

Surprise! Via Macintouch we learn that Mac OS X 10.7 Lion has Automation features.

It is weird that Apple's official Automation documentation is hosted on macosxautomation.com which is "not hosted by Apple". Even though it's seemingly an Apple site (with a broken icon on page 1).

Even weirder for all of us who figured AppleScript and Automator were dead, is that Lion has a lot of AS/Automator features. Some show up on Apple's Lion Features page, but many do not. Given rumors about Apple's new focus on textbooks and iPads, it's noteworthy that Automator has many new ePub support features.

Apple is eccentric.

WordPress doesn't have a built-in table editor

2012-01-01T21:29:00.001-06:00

The WordPress visual editor doesn't include tables. Neither does Blogger of course, nor, for that matter, MarsEdit.

FrontPage had terrific table support in 1995. WindowsLive Writer has decent support now. Otherwise, web tables doesn't get a lot of love. RapidWeaver doesn't do tables. Sandvox isn't any better and neither is Apple's abandoned iWeb.

SeaMonkey inherits the table technology built into Netscape Composer in the 1990s. TextEdit does tables (!) and (unlike Pages) will export HTML. It's hardly a web page editor though.

DreamWeaver does tables - and costs $400 (though I qualify for the $150 teacher edition).

It's too bad. HTML tables are really brilliant. I'm guessing implementation is very expensive especially when tables are combined with CSS; the market doesn't support this level of complexity.

Parental Controls on iOS and OS X: what we do now

2012-01-01T19:49:00.001-06:00

A year or two ago I wrote about how Google and Apple have both failed Parental controls. Since then things have not gotten much better.

In response to a comment on an old post, this is the compromise I use for the children's accounts on iOS and OS X.

Google is blocked. I find Bing searches easier to track and control because it doesn't use https.
Children get our family Google Apps domain email through mail.app IMAP, not through Gmail.
Children access our family Calendars from their iPhones, not from the desktop. (I could use iCal on the desktop, but iCal is one of the worst pieces of software garbage ever produced.)
A 'Family and Learning' account can be accessed at any time. It has very limited net access, has WorldBook, has apps, iTunes, etc.
Each child has their own account. Parental control is set to 'automatic' with a few domains specifically allowed. I was never able to get domain specific filtering to work. After they are on the computer I review their browser history with them. They could of course delete specific browser pages, but I don't believe they have (the computer is very visible and public). I stopped reviewing log files because Apple's log file review UI is almost as crappy as iCal.
Because iOS apps have so many back doors to webkit, particularly via ads, we don't use any 'free' apps. Safari is disabled. For now we allow iTunes despite the content it provides -- the boys are getting older.

This works for us, but Apple's Parental Control support is lazy and incompetent. They simply don't care.

Android/Google, as best I can tell, are worse. Note that Google Gmail explicitly states all US users must be 14 or over (COPPA partly, but really this is a Google copout). i don't think Android OS includes any default parental controls.

I don't know how Windows 7 does. I suspect it's a bit better. I can't find anything about parental controls in Metro/Windows Mobile.

See also:

Manipulating JSON data in a traditional relational database (Microsoft Access, FileMaker Pro, Converters)

2011-12-31T10:34:00.001-06:00

While I wait to see if Pinboard can fix their Google Reader JSON (JavaScript Object Notation) import, and while I consider Google Reader Share JSON import into WordPress, I'm also exploring JSON import/export tools. If, for example, I could import JSON into FileMaker Pro or other data management tool I might be able to manipulate the archive and produce a more useful WordPress import.

StackOverflow and its kin have a good set of references on this topic. Note that CSV can manage only very simple JSON; we really want native importers similar to what Microsoft Access tried with XML [1]. I suspect one approach might be to convert JSON to XML then use Microsoft Access 2010 import.

Incidentally, this topic veered off unexpectedly into something that's actually relevant to my work life and a Strata conference I'm attending in a few weeks.

As of today here are some of my pointers ...

What is “JSON” and what does it have to do with distributed computing? « DivConq: great reference (see below for DivConq series on Access)
JSON - Wikipedia, the free encyclopedia: I liked the comparison to XML [1]
JSON.org: points to libraries
18.2. json — JSON encoder and decoder — Python v2.7.2 documentation: If I have to use a programming library to manipulate Google Reader json files my choices are probably Python or Javascript and I prefer Python.
conversion - Convert JSON format to CSV format (for MS Excel) - Stack Overflow: JavaScript
convert from json to csv using python - Stack Overflow
Download JSON data and convert it to CSV using Python - Stack Overflow
python - Convert JSON to CSV - Stack Overflow
javascript - Python - convert csv file to JSON - Stack Overflow

For me this DivConq series was particularly useful because it placed JSON nosql processing in a familiar context - Microsoft Access.

Maybe I should start using Apache Cassandra to manipulate my Google Reader JSON archive and prepare it for WordPress processing. For example ... Cassandra Development Environment in Mac OS Snow Leopard « BigDiver.

[1] I doubt JSON has truly significant advantages over XML as a data interchange format (see JSON Example and wikipedia xml/json). Alas, nobody asked me. Fashion is more powerful than geeks imagine.

Pinboard imports Google Reader JSON exports

2011-12-28T10:07:00.000-06:00

Pinboard is the first service I know of that will import a Google Reader Social (shared item) JSON file:

Pinboard: howto page

Google Reader Click the gear icon in the upper right corner of the page. Select the Import/Export tab. Choose either items you have starred or items you have shared and click the Reader JSON link (the rightmost column)

When I stumble unexpectedly over something I've been looking for, I look for who else found it. Then I add them to my reading list. Google gave me only these references:

Google Reader Shared Items -> Pinboard, FeedDemon, Send to Pinboard… « cmiles – blog 11/5/2011
Pinboard Blog - Nov 2011 - "... Added importers from Firefox JSON format and Google Reader. Wrote scripts for tarring up user archives in human-readable form so people can download them. Removed Google Reader support (RIP, Google Reader, I'll mourn you til I join you)...

Pinboard has a feed, I don't know if importing will trigger feed actions (probably not)

See also:

Update: I paid my $10 and imported by Google Reader shared item JSON file. I have 3 days to cancel. I used Amazon payments.

Here are the results; as of today the most recent post is 7 weeks old. I may also try importing the JSON for my Reader shared items, which may produce some duplicates.

http://pinboard.in/u:jgordon - my pinboard collection - really my Google Reader shared items. Note my user name is a part of the URL, so it's nice that 'jgordon' was available. Posts show a title, a bookmark, and an excerpt. I think my GR annotations precede the excerpt. It's more like Google Reader Social than I'd expected.
http://feeds.pinboard.in/rss/u:jgordon/ - the public feed for my collection. I viewed this in Google Reader; gave me a real sense of deja vu. Alas, GR only pulled in 44 items.

I'm still studying the results. So far Pinboard is only showing a fraction of the JSON file, there are not tags, and every item shows with date of '9 weeks ago'. I don't see a convenient way to navigate across the entire collection.

Update 12/31/11: Pinboard has now imported 2 months of Reader shares - about 1100 items or roughly 1% of the total.

Sharing and annotation: Instapaper's supporting apps

2011-12-28T09:53:00.000-06:00

I haven't found a replacement for the rough annotation-share-feed ecosystem that had grown up around Google Reader Social (RIP). I've given up, for example, on using Twitter as a Reader Social replacement.

Yes, I miss Google 1.0. I even miss Microsoft these days.

So I'm continuing to explore the pieces of the post-Google world; trying to see where this micro-market may go. This is poorly tracked territory, but today I came across an unexpected guide in the Instapaper: Supporting iPhone and iPad Apps page.

Instapaper has an ecosystem, and although it doesn't have a feed, it will post to Tumblr, Twitter and Pinboard. Tumblr has a feed (barely), Twitter can be turned into a feed (awkwardly) and Pinboard has a feed (and, mercifully, it's not free).

So what can I do with these pieces? Can I archive the output of Pinboard as WordPress posts?

I'll find out.

See also:

Gordon's Tech: Pinboard imports Google Reader JSON exports

Update

I tried Instapaper's bookmarklet, but it hangs in Chrome with a "saving" status in the tab.

Recovering Shared Reader items: JSON import into Wordpress

2011-12-23T10:55:00.001-06:00

Google amputated a portion of my distributed memory, but they left me a frozen json remnant.

(Yes, we are living in a cyberpunk novel. Sigh.)

Across the net there are unanswered questions about what to do with these json archives. Google has been silent. I believe the Google humans who might help are ashamed or demoralized or fearful. Google 2.0 is not a happy place for them.

I want to represent my JSON archives as posts in a WordPress blog, perhaps with some kind of synthetic title. Then they will be available to search and link. Eventually I hope to add new annotations and shares to that archive, though there will be a gap of several months that will be difficult to fill.

This feels doable, but so far Google (the search engine) hasn't told me how. This is what I have found so far. When I do find an answer, I'm going to answer some of the dangling questions across the net ...

Dariel: wrote a Python script to import shares into ReadItLater
What are the good ways to import Google Reader Notes (.json) to Evernote? - Quora - an unanswered question
battis.net » Transmogrifying those Google Reader JSON dumps into something useful - Seth Battis wrote a "PHP script that converts (at least my) Google Reader JSON dump into an XML file that WordPress can import as a list of posts. With the tags and annotations converted over. In fact, with all of the data in the JSON dump embedded in the XML file (although WordPress doesn’t read all of it)."
Importing .json archives (Wordpress) - Stack Overflow - an unanswered question
osx - Json viewer for mac - Super User: a way to at least view JSON documents.
How can I Import Google Reader JSON archives into WordPress - Super User: I ask on SuperUser
A public question I asked on Google+: I ask on G+

I'll update this post as I learn more. Seth's contribution suggests a fix is close; he needs to tweak some of his code.

Update 12/25/2011: Seth writes that he won't have time to work on this further but he recommends downloading his php file from his linked zip. I'll have to learn how to run PHP scripts from my Dreamhost account, but I don't think that's too hard.

The cost of repairing a Mac is less than expected

2011-12-22T21:30:00.001-06:00

When my iMac 11,1's 2yo 1TB Seagate drive developed metastatic blockitis I was most unhappy.

It wasn't that the drive is dying young, just two weeks after my AMEX extended warranty lapsed. Two years is short for a drive, but this machine runs all the time and goes through two full disk backups every single night. The drive has had a hard life.

The fact Apple's diagnostics missed my drive's unmasked bad blocks is annoying. There's no magic to a disk scan; I shouldn't have had to buy TechTools Pro to make a diagnosis. Windows diagnostics have managed this for twenty years.

Worse though, is the cost of the repair. FirstTech, a well regarded local shop, gave me a $625 quote to install a 2TB replacement. (They can't get 1TB drives.). I was amazed, I'm used to paying $150 or so for a drive and doing my own installation. That's what I did when my old fully serviceable G5 drive died.

The problem is that the lovely 27" iMac is not user serviceable. Elegant quite design with special thermal sensor cables turns out to have a high post-purchase price. That's why I wrote ...

Gordon's Tech: Mac drive diagnostics: TechTools Pro and Drive Genius find problems OS X missed
... When you consider that iMac 27" hard drives are NOT user serviceable, the iMac is more expensive than it seems. The iMac G5 was entirely user serviceable. Design has its price....

I was wrong though. Today i checked what the cost would be for an Apple store repair. They quoted me about $200 for a 1TB drive replacement. (They don't do upgrades, only like-for-like replacements.)

How do they do that? Apple has a flat $40 service fee, regardless of the complexity of the repair. Apple offsets the ownership cost of their elegant designs by subsidizing repair. (In this case they have another advantage -- they have an inventory of 1TB drives with bundled thermal cables even as the world runs short of hard drives.)

I still prefer my G5 iMac's design -- but that was a hot and noisy machine. My 11,1 (i5, 27" 2009) iMac is quiet and cool most of the time. Apple's subsidy of post-warrany repairs makes that tradeoff more palatable - at least if you live near an Apple store!

Update 1/6/12: One warning: they will want to keep the hard drive. This fits with their out-of-warrantee repair following their in warrantee process. Apple should make this better known. It means if you bring your machine in for an Apple Store repair, you need to do a secure wipe first. Some additional tips:

Create an admin account with no password that Apple can use for testing. I didn't think of this, and my machine has guest account disabled.
They will want to recreate the problem -- even though I have to pay for the repair. Again, their out-of-warranty repair is basically in-warranty with a parts-charge and subsidized labor.

Mac drive diagnostics: TechTools Pro and Drive Genius find problems OS X missed

2011-12-20T21:19:00.001-06:00

I've been having suspicious application crashes lately. In the Mac world, that suggests a hardware problem. (Once upon a time software was the cause, but these days it's hardware.)

I knew from some backup issues that I had 3 unreadable files. That suggests my 24+ month old 1TB iMac drive is dying youngish. It was time for some diagnostics, so I plugged in my old Apple Keyboard and mouse. (Most non-trivial diagnostic work requires a wired keyboard and mouse; Apple's bluetooth keyboard/mouse drivers may be unavailable when needed.)

After I deleted the bad (non-critical happily) files I ran Disk Utility - but the drive passed. Then I ran my Apple Hardware Test - extended, and loop mode. Still no problems.

I didn't believe it. Something had to be wrong. So I checked out Disk Warrior, Disk Genius and TechTool Pro - 3 reputable diagnostic apps. They're all $100. Disk Warrior has a good reputation, but Disk Genius has a trial version. It found about 58 bad blocks -- out of 1.8 billion. That seems a modest number, but DG said I needed to replace the drive. (Incidentally, Disk Genius has a built in uninstall feature -- very nice. Yes, the Mac needs an OS level uninstaller.)

I decided to get a second opinion. Andy M clued me to a MacUpdate bundle, so I got TechTool Pro 6 for $50 (plust a bunch of other apps I don't care about). It one-upped Disk Genius; as it found bad blocks it told me which of them had files (none in this case).

TTP also found bad blocks - 56 (so two less than Disk Genius, but I don't make much of that either way).

I wasn't sure what to make of this. After all, 56 out of 1.8 billion is minuscule. Unfortunately, a modern SATA drive shouldn't have any bad blocks. The excellent TTP manual explains why ...

... TechTool Pro should not normally report bad blocks for these types of drives. The drive controller in them automatically tries to map out bad blocks as they are encountered. It will do this unless either the bad block is in a critical area that cannot be mapped out at the moment or the bad block table is full. If this occurs, TechTool Pro will report a bad block and you will ultimately need to do a low level reinitialization of the drive. When the drive is reinitialized, the entire platter is accessible so that bad blocks can be mapped out if possible no matter where they occur...

.. You can use Apple's Disk Utility to reinitialize your drive. Be sure to choose the Security Option to "zero out data." Choosing this option will map out bad blocks, if possible, during the reinitialization. This may take several hours (depending on the size of your drive). If the reinitialization is successful, the drive should be fine at that point. We suggest, however, that you do a Surface Scan a few times in the next month or two just to be sure no new bad blocks are developing. If they are, then the drive is probably failing and you should consider replacing it. If a low level reinitialization fails, this indicates the drive is faulty and needs to be replaced...

So the bad blocks I see now are probably a small fraction of the number that have already been mapped out. I'm seeing the overflow, including blocks that went bad after they'd been written to.

My i5 iMac is 24 months and 2 weeks old - so it's past even my AMEX extended warranty (by two weeks!). If the drive were user serviceable (like my old G5 iMac!) I'd simply replace it. Since it's not a user serviceable I'll probably bring a new drive and the machine to FirstTech in Minneapolis for a $200 24 hour turnaround replacement. I'll make a bootable clone before I do that. (My usual Carbon Copy Cloner backups are to an encrypted image for offsite transfer, so not bootable.)

~~If it doesn't I'll try the reinitialization~~. (See Update - this drive is on death row.)

Update 12/21/2011: Various notes and reflections the day after ...

16 months is a short lifespan for a hard drive. I bought this machine early in its lifecycle, I wonder if there will be more failures in this product line.
Modern drives don't write to bad blocks. Based on the dates of the files that were involved the involved blocks went bad in the past month. That fits with Carbon Copy Cloner not complaining until recently. (See my backup issue post for a twist to this story.)
I'm glad I bought TechTools Pro - I think I'll get good use of it. From what I know now though, I didn't really need it. In a modern drive a single bad block in a file, especially a relatively recently written file, means replacement. Carbon Copy Cloner told me 3 files were bad.
The TTP manual suggests reformatting. I suspect that might work if there was an initial formatting problem, but in this case I know existing blocks are going bad. This drive is on death row.
Carbon Copy Cloner complained about bad sectors in files during backup, but Time Machine didn't. That may be because Time Machine only reads files that have changed?
Most of the bad sectors are in unused parts of the drive. I suspect they were randomly distributed but were hidden by the drive OS as they were discovered in the parts of the drive that have been used (about 500GB of 1TB).
Drive Utility and Hardware Test didn't find any problems, but both Drive Genius 3 and Tech Tools Pro failed the drive and Carbon Copy Cloner complained too. The SMART diagnostics still pass the drive - even today! I'm a bit surprised; this isn't rocket science. Apple could do better. TechTools Pro gives more SMART diagnostics than Disk Utility -- my drive was complaining about heat (it's cold in this room!)
It's clearly worth running TechTools Pro or equivalent drive scan on a new drive then every few months. (11/23/11: TTP just crashed during a routine drive scan. I'm not impressed.)
I think Windows scandisk/chkdsk are superior diagnostic tools to Disk Utility.
TechTools Pro DVD includes an image for burning a PPC DVD. Nice touch. I still have an old PPC.
It's good to know the drive is dying before it dies. I have time to do extra backups and to move selected files to other machines -- including my Aperture and iPhoto Libraries and perhaps iTunes.
When you consider that iMac 27" hard drives are NOT user serviceable, the iMac is more expensive than it seems. The iMac G5 was entirely user serviceable. Design has its price.
Since I know the drive is dying I've disconnected my clone backup. It's my known good repository. I'll take it to my office then create a new clone, then disconnect that clone. I won't be saving data to this machine, I'll be treating it as a "guest" machine until I get it serviced. I may turn off Time Machine too.
OWC (Other World Computing - great Mac shop) showed me how to find my Model Identifier (System Profiler), it's iMac11,1. I can only go to 2TB of storage. I'm not confident that their options are correct however.

See also:

How do you upgrade the hard drive in the "Late 2009," "Mid-2010," "Mid-2011" and "Late 2011" (21.5-Inch and 27-Inch) Aluminum iMac models? - I'm glad I didn't consider doing this myself. I wonder if this is almost an Apple-only replacement.
Proprietary Cable can put the brakes on upgrading Late '09 iMacs. | Other World Computing Blog: I have an ST31000528ASQ - so Seagate. OWC says these drives will fit my computer's thermal cable:
- 1.0TB Seagate Barracuda 7200.12
- 1.5TB Seagate Barracuda 7200.11
- 2.0TB Seagate Barracuda LP

Backups - why you need two methods and abundant paranoia

2011-12-20T06:11:00.001-06:00

I can't say I feel good about my backups. I believe data wants to die; it wants to be free of the burden of order. Against the despair of data, even the best backup is barely adequate.

Consider tonight, when everything almost failed - Time Capsule and Carbon Copy Cloner alike.

The Time Capsule serves all the machines in our home over a wireless network. I was surprised at first that backup would work over wireless, but it does. Each machine has its own unencrypted disk image; one on the TC's old internal 500 GB drive, two others have images on an external 2TB drive. The TC sits in a closet upstairs; it's unlikely to be stolen but fire would destroy it. I have done 1-2 file Time Machine restores from that image, so I know it can work. The only test of a backup, of course, is a restore.

I don't trust Time Machine as much as old-time Dantz Retrospect, but it seems Apple has gotten most of the bugs out.

I trust Carbon Copy Cloner [3] more. Each day it clones my server, on which all the important data lives. It's more than a cloner; CCC keeps copies of changed or deleted files in "_CCC Archives". I've configured CCC to use an encrypted image it automatically mounts every night. Since that backup is encrypted I can take it offsite, which I do every few weeks. Ok, every month or two. Offsite rotation relies on me, so it's prone to failure. Still, even if the house burns, I am unlikely to lose more than a month of images and videos. I can live with that.

So I have two backup methods, both fully automated, both relatively independent [2]. If each is 95% reliable each day, then the chance both fail on a given day is 1/400. If the daily chance of a server drive failure is 1/1000, the odds of all three failing on the same day are about 1/400,000 [2], [4]

Tonight though, my data got within a few miles of the cliff it wants to meet.

My server has been having worrisome memory exception (EXC_BAD_ACCESS) crashes, and a TV show I recently downloaded had a file error [1]. There's something wrong on my 2yo i5 iMac; I need to run Apple Hardware Test (again). So I know my server data is at risk.

Time Capsule has had problems too -- it's reporting a "communications error" periodically. I think that error message is a scarlet herring related to the iMac issues, but clearly I can't trust that backup.

Happily there's good old CCC -- but when I restarted my server for the first time in weeks it reported a problem. The backup drive didn't mount. That was easy to diagnose -- I'd unplugged it. Probably when I was debugging my Aperture crash 3 weeks ago. Why didn't CCC report the error? Maybe it had crashed.

I wasn't that close to data loss -- but I was in a bad neighborhood. As paranoid as I am, I'm almost not paranoid enough.

It's good to have two fully automatic and completely independent backup methods. Data wants to die, and backup is still an unsolved problem.

-fn-

[1] Incidentally, you can't easily report a purchase problem to Apple until they process a charge, and to reduce transaction costs they wait a few days before they process. This is very annoying! Also, the UI for reporting a purchase problem is suspiciously clumsy. More on that experience when I see what they do.
[2] In reality they common failure points of course - me, computer memory, etc. There is the older offsite backup though, so complete and total data loss is probably less than 1/1,000,000.
[3] Donationware. I donated. I wish donation ware apps would let us set a 'reminder' so I could donate yearly. I suppose I should just make donationware donations every year on my birthday against the apps I use.
[4] I'd love to have automated offsite backup too, but I've never found an offsite vendor I trusted and I expect ISPs to eventually charge for bandwidth use.

See also:

CrashPlan (or JungleDisk) instead of Retrospect 8? (1/2010)
Online backup – the security problem (it’s not the encryption)
backup (50 or so posts on backup)
Retrospect 8.2 for OS X fails my latest review -- because it's been abandoned (11/2010 - I checked under the latest owner but it felt the same)
Freeing up Time Capsule space – and documentation for Time Machine and Time Capsule (12/2009 - TM/TC are better quality now, but the documentation still sucks).
Time Machine, Time Capsule and offsite backup (2/2010)
My 10.6 iMac is crashing - a debugging exercise (2/2010 - using hardware test)

Update 12/21/2011: I was closer to the cliff than I realized.

Twitter to WordPress via ifttt - limitations

2011-12-13T20:34:00.001-06:00

Weeks after Google's Day of Infamy i'm still failing to fully replace Google Reader Shares. Recently I gave up on Tumblr, Posterous, the zombie version of Google Reader, and some screen-scraping attempts to turn G+ streams into feeds.

Lately I've been focusing on my @jgordonshare tweets and tonight I tried using ifttt to create a WordPress feed-equipped archive of tweets.

It was easy to setup the ifttt task to turn the tweets into WP posts. I used a "1 button install" Dreamhost [1] WordPress instance I've been testing. I had to turn on XML-RPC publishing (used by Windows Live Writer, MarsEdit, etc) and provide a WordPress username and password [2].

The ifttt doesn't trigger immediately after tweet creation. I assume it checks the Twitter stream every 15-30 minutes. I manually triggered a check from the ifttt dashboard.

Here's an example of what I got

Just testing iftt tweeting to wp (sorry). http://t.co/GbR8Qdud

... Just testing iftt tweeting to wp (sorry). http://t.co/GbR8Qdud...

Yeah, not to impressive. The problem is a tweet is simply a string, it has no special structure, no way to distinguish URL from my commentary from page title from annotation (not that there's room for all that). Tweets are much simpler entities than old-style Google Reader shares.

The experiment did work, but the result isn't terribly interesting.

So the quest goes on ...

[1] Use the code "KATEVA" or this link and you are supposed to get 50% off your 1st year costs and I get an equal saving as credit.
[2] Obviously you should create a user for this purpose and create a unique password. IFTTT has to know your credentials.

See also:

Dreamhost - apparently the kickbacks work 6/2010
Google reader shared items to Facebook 3/2009 - Google's glory days - before the Fall
Loving Google Reader - Shared post feed 9/2008 - Google's apex
My reader shares are back for now - thanks to Keakon 11/2011 - sweet hack, but a bit of a security risk
Social wrestling: Google Reader, Twitterfeed, Blogger and Facebook 4/2010 - twitterfeed is still around
Filtering my Google Reader Share with Yahoo Pipes 9/2008 - never quite worked, but an interesting example
Using Bloggers undocumented label (category) feeds and Yahoo Pipes to create a tech opinion feed out of Gordon's Notes - 2/09 - fiddling
Tweeting Google Reader Shares and Notes via feedburner 10/2010 - we assume Google will axe feedburner in 2012
FreeMyFeed - Getting Twitter feed to Google Reader - turning twitter into rss 4/2009
Microblogging and Google Reader: Tumblr Fails 11/2011
After the fall of Google Reader: Posterous, Tumblr and Zootool with Twitter on the side 11/2011. I haven't tried Zootool yet.

How to learn what your current AT&T mobile contracted services are

2011-12-09T08:29:00.001-06:00

AT&T has the lowest customer service rating of American mobile phone companies. Of course that's like asking what's worse - Ebola or Rabies?

This is not because of their retail staff. They must give them powerful drugs, because, despite working for a moderately evil corporation, they're remarkably cheerful.

Their web site though, that's part of what makes them "Rabies" rather than just "Ebola".

For example, for the past few days I've been trying to follow up on some extensive bill slashing changes. In particular I've been trying to find a current contract summary for our family plan. I think I've found the best that ATT offers, but they have one of the worst web sites I've ever come across. [1]

As of Nov 2011 try this. Don't click on the tabs, but mouse over to see the substructures. Note you may have to authenticate repeatedly.

Go to your AT&T mobile account page. Look at the top menu structure. It will say myAT&T, with "tabs" like "Overview", "Bill & Payments" and so on. Depending on the services you use some are not useful, but they will still appear. The tabs that are useful for a mobile-only customer are
- Bill & Payments: see current bill statement
- Wireless: usage and recent activity
- Profile: user information
From user information look for "Contract Information". Click Customer Service Summary and Contract. Now you get a popup window. In there you find several options including two that, despite their names, both show a similar PDF (these links may actually work as shortcuts once you're authenticated):
- Wireless customer agreement: This is the real deal. CSS plus six pages that summarize your true contract [2]
- Customer service summary (wireless): Just the CSS
- Alternatively, AT&T's "Your Phone CSS" email provides this link with goes to a screen I can't find when I navigate the site, in fact it seems to be a outside of the tabs they define and possibly a separate web site: https://www.wireless.att.com/olam/loginAction.olamexecute?target=CSS. There's a trick here. Unless you read carefully, you'll hit the "continue" button -- that will just take you back to the main site. Instead, look for the link under the wireless number drop down and click that. You get the PDF contract summary.

Note that under the Wireless tab is a "Rate Plan" link, but it only shows voice plan.

So, in summary, to determine your actually currently contracted services for a family plan you need to print/view a PDF for each individual family member and do the sums to produce an integrated view.

I wonder if there's a medical term for the psychosis induced by dealing with AT&T?

Next up: How to track SMS use.

[1] The primary flaw is that AT&T mixes marketing with service. There are other reasons, but that's the primary dysfunction. What I want to know is mixed with what they want to sell me. A secondary reason, is that they choose not to invest in areas that allow customers to see what their contracts are. Those investments have a low return on investment and will not contribute to someone making VP. It's just a happy accident that it works this way.

[2] Update: AT different times I got different PDFs for different users. Only my wife, a secondary number, produced the full six page report when I requested a "wireless customer agreement".

Update 12/11/11: I've studied the PDFs in more detail. They are quite hard to interpret, particularly for a family plan, but, with some study, they do show the current contract. I think the "wireless customer agreement" is a copy of the original contract, but the "customer service summary" is the current contract.