iMovie 11 does not completely import iMovie HD projects

I was infuriated by the data loss during iPhoto to Aperture migration, but that's nothing compared to iMovie HD to iMovie 11 migration [1]. All titles and effects are lost.

This happened years before the better known debacle; Final Cut Pro X does not import Final Cut Pro 7 projects.

How does Apple get away with this?

It's not just a rhetorical question.

[1] Prior versions don't import at all, though you can dig the .DV files out of the Packages and import them by hand. Or open prior projects in iMovie HD, then save them then import.

MPEG Streamclip video converter as Compressor alternative

I came across this reading a well written response on the FCP discussion group:

Squared 5 - MPEG Streamclip video converter for Mac and Windows

... MPEG Streamclip is a powerful free video converter, player, editor for Mac and Windows. It can play many movie files, not only MPEGs; it can convert MPEG files between muxed/demuxed formats for authoring; it can encode movies to many formats, including iPod; it can cut, trim and join movies. MPEG Streamclip can also download videos from YouTube and Google by entering the page URL....

Those are impressive features. The post claimed conversion quality equalled that of Apple's Compressor app.

Reading the release notes it's typical video software - somewhat flaky and tricky to use. Some functionality requires companion apps. It's not fully Lion compatible and OS updates may require app reinstallation. Some details on functionality (emphases mine)...

MPEG Streamclip lets you play and edit QuickTime, DV, AVI, MPEG-4, MPEG-1; MPEG-2 or VOB files or transport streams with MPEG, PCM, or AC3 audio (MPEG-2 playback component required); DivX (with DivX 6) and WMV (with Flip4Mac WMV Player). MPEG Streamclip can export all these formats to QuickTime, DV/DV50, AVI/DivX and MPEG-4 with high quality encoding and even uncompressed or HD video.

Video conversion is performed in the YUV color space; you can choose to scale video to any frame size using a professional 2D-FIR scaler (better than bicubic) but you can also leave it unscaled. Other optional video processing features include a powerful motion-adaptive deinterlacer, a field dominance converter, a chroma reinterlacer and an option to perform interlaced scaling instead of progressive scaling. Audio can be converted to uncompressed or to IMA, AAC, MP2 or AMR using the high-quality MP1/MP2/AC3/PCM built-in decoders of MPEG Streamclip; it is also kept in perfect sync with video using a timekeeping system.

MPEG Streamclip can save edited movies as MOV files, and (when possible) as AVI or MP4 files. Edited MPEG-1 or MPEG-2 files are saved as MPEG or TS files.

MPEG Streamclip (with or without the MPEG-2 Playback Component) can also convert MPEG-2 transport streams into muxed MPEG-2 files, for immediate burning at full quality with Toast 6 or 7 and Sizzle; it can also demux MPEG-1 and MPEG-2 files and transport streams with MPEG, AC3, PCM audio to M2V and AIFF (or M1A or AC3) files, for immediate burning at full quality with DVD Studio Pro or Toast 6 and 7. A special demuxing option is available for Final Cut Pro 4/5: this application does not work well with M2V files, but MPEG Streamclip can write a special "unscaled" M2V file that preserves full video quality when imported into Final Cut Pro.

MPEG Streamclip can handle files and streams larger than 4 GB, split in any number of segments, or with multiple audio tracks, and can also optionally handle timecode breaks. It is compatible with MPEG-1 and MPEG-2 video, MPEG layer 1/2 (MP1/MP2) audio, AC3/A52 audio, and PCM audio.

The player included in MPEG Streamclip lets you preview the files and transport streams before doing the conversion; it also lets you visually set the In and Out points for the conversion so you can convert just the part of the file you are interested in, and also cut commercials and other unwanted parts, or edit the stream and join two streams with Cut/Copy/Paste.

MPEG Streamclip supports batch processing: just drag some files in the batch list, choose a conversion and a folder, click the Go button and MPEG Streamclip will automatically convert all your files.

Reading between the lines I suspect it's sometimes used for non-approved editing of commercial products.

Video: The state of compression is not good. Blame it on Congress.

The more I study analog to digital video conversion and export compression the more I see I've left civilization.

This is a domain of arcane knowledge and of unknown unknowns -- "experts" who really don't know what they're talking about. I'm sure there are real experts, but they're bored silly by the level of net discussion -- including Apple's own technical notes. It reminds me of my 1981 medical school lectures in renal physiology [1], or my @2002 research into color profiles.

In the absence of publicly accessible expertise I hope I'll have time to do my own experiments, such as creating a range of QuickTime 10.x exports with a span of export parameters. I'm afraid, however, that I don't have that much time.

Which means I probably need to give up on my idea of using .mp4 files the way I use .JPEG -- as an archival storage format [2].

So that means I need to buy more storage -- 2TB should do it. From that point on, given the amount of video we capture, I will just need to upgrade every 2-3 years (4, 8, 16, 32....). And I need to invest in either firewire 800 or thunderbolt.

So now I'm looking at the 1995 era ".dv" format [3]. Should be straightforward. Right?

Yeah, that's what I thought until I read this wikipedia article, which, as is usual for wikipedia, reads like a mixture of a true expert and some confusing amateurs [5] (but is still better than the alternative...):

DV - Wikipedia, the free encyclopedia

... The DV codec was launched in 1995 with joint efforts of leading producers of video camcorders...

... DV video employs interlaced video scanning with the luminance sampling frequency of 13.5 MH...

... One video frame is formed from either 10 or 12 such sequences, depending on scanning rate, which results in a data rate of about 25 Mbit/s for video [4] and an additional 1.5 Mbit/s for audio. This results in a compression rate of 5 to 1...

... All DV variants except for DVCPRO Progressive are recorded to tape within interlaced video stream...

... When video is captured onto a computer it is stored in a container file, which can be either raw DV stream, AVI, WMV or Quicktime. Whichever container is used, the video itself is not re-encoded and represents a complete digital copy of what has been recorded onto tape. If needed, the video can be recorded back to tape to obtain full and lossless copy of the original footage....

So even .DV covers a multitude of sins -- and is a significant compression from the original stream.

Even so, a mixture of .DV and Apple's proprietary professional formats seems the only way to go for now. I hope within a decade we'll have the standards and software to do better.

Or perhaps not. It's worth asking why progress has been so slow over the past decade -- and why the "standards" for still and video images date from the early 1990s. The answer seems to be something Congress and the USPTO did in 1996 ...

USPTO issued Final Computer Related Examination Guidelines stating that "A practical application of a computer-related invention is statutory subject matter. This requirement can be discerned from the variously phrased prohibitions against the patenting of abstract ideas, laws of nature or natural phenomena".

We may need legal reform before we have an archival video format. This is not a technology problem.

[1] After weeks of confident and contradictory statements from researchers and textbooks, I wondered whether they knew they were faking it, or if they'd fooled themselves. I never did find out.
[2] JPEG itself is a non-specific marketing term that hides a very wide range of codec tweaks and metadata "standards". Still, it's a recognized library of congress standard (TIFF is even less standard.). There isn't an image format today that has a comparable archival lifespan, so I use 99% (minimally compressed) JPEG and delete the RAW images -- holding my nose the whole time. Yes, I wish JPEG-2000 had succeeded. Our IP (patent) framework is intensely dysfunctional. 
[4] Now I know what the data rate setting in iMovie 11's QuickTime export parameter means.
[5] In particular there's confusion about DV handling during capture vs. how it's represented during editing/playback and confusion about what containers are. Of course, as a non-expert, I corrected that section.

iMovie 11 and analog to digital videotape conversion by passthrough

it's been six years since I've had much good to say about Apple's OS X software (we miss you Avie). So I'm mildly surprised to say something good about iMovie '11.

I haven't paid much attention to iMovie since the iMovie HD debacle, but after Final Cut Pro X and QuickTime Pro failed me I turned to my free copy of iMovie [1]. Short of Adobe Premiere, it was the last tool to use for my family video consolidation project ...

Gordon's Tech: How should I transcode my home analog video?:

... Emily picked this project for her Valentine gift, so we're off. I found a 300GB drive and LaCie firewire enclosure I'd forgotten about. So far video capture seems to be proceeding well from a 12 yo Hi8 analog tape; there's a thin sliver of artifact at base; I think capture frame is bit taller than the image frame. Setup at the moment:

• SONY Hi-8 TRV65 (1997) with S-video and RCA audio out to ...
• Canon Optura 50 (2004) with firewire out via secondary port on LaCie to ...
• iMac i5 2.7GHz, 8GB RAM
• Final Cut Pro X (trial version. There's no more academic pricing for Apple products distributed through app store. I expect I'll buy full version. So far this is the big expense) saving project to ...
• External firewire LaCie drive (300GB)

I started with FCPX, but it only seemed to be capturing video. Video streamed to FCPX, but was not recorded. Overall, FCPX feels like it's still a year from being ready.

So, after reading about 10GB import limits for QuickTime Pro, I tried iMovie 8. These were the steps I followed:

1. Connected the Hi-8 camcorder via S-Video and RCA to the Optura
2. Set Optura to Passthrough mode (most people now would use a Canopus device or the like to do A/D conversion - workflow is very similar)
3. Turned off Time Machine -- I don't want the 30 GB videos going to my backup drive [2].
4. Start iMovie. It's safest to create the Event (will hold the video, analogous to Events in iPhoto) first so you can choose what drive the event will go to (there's a small icon that lets you see how Events are distributed over media).
5. Start Import in Manual mode and direct to the Event desired. Push Play on camera.
6. iMovie will give you a few seconds to get the video in. It shows the Canon as no tape and displays 'Ready to Record'. If the stream doesn't start iMovie will quit.
7. If recording is working the counter will increment and a small red dot will show. The 'Ready to Record' message never changes.
8. When recording is done iMovie stops (1 minute delay)
9. Then thumbnails are supposed to be created (but see below) ...

I start the when I get home from work then let it run. A 1.5 hour tape produced a 25GB .DV file and 1.5 GB thumbnails.

To export to .m4v (MP4 variant) you first create a Project. I edit my .DV by selecting the parts I don't want and tapping the 'R' (remove) key. Use Shift-Command-A to undo select all; it's easy to select all and not clear how to undo. Study the keyboard shortcuts, I don't like iMovie's standard or context menus. Editing was easier than I'd expected, even for a project this size. I'm just getting rid of the obvious junk.

When done select all and drag to Project window. Then add a title slide.

After the Project is done choose Share and large size. On my iMac compression to 960x540 took about 1 hour and produced a 1.5 G .m4v (mpeg) file; it uses "270% of CPU". I'd prefer a bit less compression, maybe a 3GB output. I'm studying my options. It's ok to save the .m4v to an external drive.

I then move use iMovie to move the entire Event to an external drive for storage (see update, these directions are wrong). Empty trash and restart Time Machine.

The Events, for now, are staying on my external store (1TB+, so should handle about 8-9 of my media. I don't want to keep the .DV files, when I'm satisfied I can produce a high quality .mpeg movie with reasonable compression I'll store that alone. Three 1.0 TB drives should manage all of our video - one onsite, one offsite backup, one rotation [3].

Of course there are bugs ....

• iMovie crashed twice (kernel exception -- it really doesn't like the cheap external drive I'm using until the better ones come along [3]
• iMovie always hangs at the end of thumbnail generation, no matter how short the event. This is a replicable bug. I've tried deleting preferences; this may also be a sign these Project are too big for iMovie. I'm hunting around for an old copy of Final Cute Pro or Final Cut Express. (By Darwin, Apple screwed the video market.)

It's early days, but I'm less displeased than I'd expected. I think if iMovie fails I'll try Adobe Premiere Elements next.

[1] When Apple introduced the App Store, they also turned to iOS style licensing. iLife '11 came with my MacBook Air, and was registered with my Apple Store ID. That meant it showed up in "purchases" on my iMac and I could download and install it.

[2] Trick is to remember to turn it on again after the iMovie Event has been moved to an external drive. Streaming capture to my old external firewire enclosure did not work. Not only were frames dropped, but iMovie crashed as well.

[3] Future post pending on the external storage/backup system I'm going to use.

Update: Possible export options (ref)

• MPEG-4
• frame size: largest possible (when I set to be the source size however, the output still omits about 3% of the frame)
• key frames: <10
• data rate: DV data rate is 25Mbit/s for video and 1.5 Mbit/s for audio. The default on .mp4 is about 4Mbit/s.

There's surprisingly little quality information on these settings. Reading the Final Cut Pro 7 documentation makes me envious.

Update 2/17/2012: Apples directions for moving Event's are incorrect. If you drag and drop the Event is copied. To move you have to Cmd-Drag, just as in the file system.

Update 2/18/2012: I realized that, by accident, I was using a low end firewire 400 cable with a 400/800 converter to my external drive. That is a bad idea! I would expect it to cause issues. I switched to my LaCie 400/800 cable. I've also given up on the idea of using .MP4 as my archival format.

I've also experimented with capturing via QuickTime 10 at 'highest qualilty'. That created a .mov file that is almost exactly the same size as the .dv file that iMovie crated from the same source; at heart it's a DV file. There's a trick to get this to work with passthrough video. I had to start the stream, then click the record button. When I move these into an iMovie Event library iMovie simply moves the .mov file then creates thumbnails.

I've read FCPX will import these. I'm experimenting more with this.

Update 2/18/2012b: OK, the QuickTime method doesn't work with this particular video stream. I didn't run into any time limit, but there's no sound on playback -- even though there's sound during the capture process.

Update 2/19/2012: I was able to complete a video capture without the thumbnail hang. Two things were different. My external drive (firewire 400) was not mounted, and I stopped the stream manually rather than let it self-terminate when the video stream stopped. I have another post pending on why I gave up on the idea of .mp4 export. For now I'm collecting 30GB .DV files for each tape.

How should I transcode my home analog video?

Twelve years ago I was keen to digitize my home video tapes - on a 400 MHz Celeron system.

Eight years ago I wrote "The thought of losing the kids' taped videos is not comforting. I need to do this sooner rather than later.". At that time I experiment with transcoding through a Canon digital camera using iMovie and a G3 iMac. I figured my G5 would do the job in 2005, maybe with its capacious 40GB drive and a stack of DVDs.

Now I have a slightly aged 2.7GHz i5, 8GB RAM, and several TB drives -- and my analog tapes are 8 years older. Maybe I'll do it this year, even though I hate digital video [1].

I'm trying to warm to the idea. The big difference over 8 years ago isn't processing power [2], it's cost/MB and the gradual emergence of H.264/MPEG-4 Part 10 (AVC) as a relatively standard format for video. The National Archives recognizes there's no real AV standard, but their list of formats they work with includes MPEG4.

If I do finally walk though this, I don't want to bother with DVDs. My current plan is:

• Purchase two 1TB drives. Digitize everything we have to one of the drives as MPEG4 video and MPEG4 audio.
• Clone that drive and keep one drive on site and one drive offsite, repeat every few years.[3]
• Use my old Canon digital camera with passthrough to digitize my analog tapes, then compress on my iMac and store with a file naming convention to reflect date and time information. (I'll consider the Canopus though.)
• Convert my collection of digital tapes (DV) to H.264 as well

I don't know what software I'll need. Can I do what I need to do with iMovie/QuickTime Pro, or should I buy Final Cut Pro X? I'm not planning to edit, but if I were I'm pretty sure the current fork of iMovie won't do the job.

Update: Apple offers a free download for FCP/X. So I will probably try it out.

See also:

• Good advice on digitizing video (Tidbits) 12/2004
• Home Video Editing started 6/2000! (Pass through with a Canon ZR 60 in 2003)
• Digitizing Video: Tapes - extensive Macintouch archives
• BYTE Media Lab 2004 Imaging Awards: Analog to Digital Converters 4/2004
• Why I hate video: Format, codecs, DRM and m4v vs mp4 7/2011
• Amazon.com: Canopus 77010150100 ADVC110 Converter: Electronics (old tech, good reviews, $218) 2/2012
• REC.VIDEO Google groups/usenet posting: Video degradation with S-Video analog transfer vs. A/V mini-jack connection 3/2005

[1] In Jan of 2010 I discovered a 12 yo WMV formatted video was unplayable; I was able to use Windows Movie Maker to convert the 23MB WMV file to a 311MB DV AVI file. Digital image formats are a friggin' mess, but video is at least ten times worse. No metadata standards, crazy patents, non-standard containers, audio codecs, video codecs - yech.

[2] Yeah, the machines are a hundred times faster. But compression can run overnight.

[3] By 2020 I assume we'll have 100TB main system and backup drives, so storing my old video with my routine data will be trivial. By then a single snapshot will be a GB.

Update 12/14/2012: Emily picked this project for her Valentine gift, so we're off. I found a 300GB drive and LaCie firewire enclosure I'd forgotten about. So far video capture seems to be proceeding well from a 12 yo Hi8 analog tape; there's a thin sliver of artifact at base; I think capture frame is bit taller than the image frame. Setup at the moment:

• SONY Hi-8 TRV65 with S-video and RCA audio out to ...
• Canon Optura 50 with firewire out via secondary port on LaCie to ...
• iMac i5 2.7GHz, 8GB RAM
• Final Cut Pro X (trial version. There's no more academic pricing for Apple products distributed through app store. I expect I'll buy full version. So far this is the big expense) saving project to ...
• External firewire LaCie drive (300GB)

Both cameras are on external power. I needed my old manuals, had the SONY and found the Optura online. Getting passthrough working was balky, FCPX only gave me about 10 seconds to start then it turned off import due to lack of a signal. It can't operate the Optura 50 in pass-through mode, I haven't tried yet with a tape installed. Once I hit the play button fast enough I was ok.

I'll have data on file size tomorrow. FCPX imports using its default Apple ProRes format.

Operating the old SONY feels primeval. It's 15 years old, feels 50. It is weird to see the kid video. Plan is to put a library of MP4 versions (no editing really) on iTunes on a an old G5 with a largely unused drive. We can browse, play from iTunes.

Update 2/15/2012: Not so good. Nothing captured! Looks like I need to read the manual, or perhaps FCPX won't work at all. It certainly showed the video, but it didn't save the event. It also didn't stop recording when the input ended.

Update 2/15/2012b: QuickTime Pro is not an option. I was able to create a @2 hour 25GB .dv file using iMovie 11, but then iMovie 11 got stuck 'generating thumbnails'. I think that bug, however, was triggered by connecting an iPhone to my iMovie machine. Disconnecting the iPhone did not clear iMovie, I had to kill it. On restart my video appears to have been captured and iMovie seems to have stopped input about 2 minutes after the end of tape. Thumbnails are fine - all 1.5GB of them. More on the results and process in a follow-up post. I'd like to take a look at Adobe Premiere Elements.

Update 2/16/2012: Gordon's Tech: iMovie 11 and analog to digital videotape conversion by passthrough.

Update 2/17/2012: I experimented with QT 10 movie capture. The trick with passthrough conversion is to first start the video stream, the click record. Otherwise QT exits. I captured 30 min of video at 9 GB; QT Inspector says the codec inside the saved .mov file is dv.

I've read that this will import into FCPX, so I may try that again. My next experiment will be to see if I can record over 10 GB (1 hour of video). There are many reports of QT recording stopping with a 'size limit' message at varying file sizes, in my case that's what displayed after I stopped the video stream.

Warning - if you delete your G+ Profile you may turn off sharing from your Picasa albums

A friend of mine complained he was unable to download some wedding pictures I'd taken. Last summer downloads worked, but as of a few months ago they stopped working.

Just around the time I deleted my Google G+ Profile.

My friend told me the original sharing link worked, but the download button was grayed out. Sure enough, when logged out of Google I couldn't download images. It was easy to see why, image downloading was turned off in my Picasa preferences.

I don't remember turning that off. Given the way I share images, that wouldn't make sense. My best guess is that when I deleted my G+ Profile Google reset my sharing preferences to the default settings. if I'd known I could have fixed it immediately. Instead I've been sending share links out for months that weren't working.

I assume most of the people I sent shares too simply gave up in frustration, or ended up downloading a low res thumbnail.

Google Fail.

Time Machine for Lion supports encrypted backups

In the midst of tweaking a Time Capsule backup, I noticed a grayed out option to encrypt my backup.

Turns out 10.7 enables Time Machine encryption on Lion - but only for directly connected drives.

Reading the MacOS X hints commentary I think they're missing the point. A good backup strategy requires offsite backup, and taking a non-encrypted backup offsite is somewhat risky. I don't take my Time Capsule backups offsite, but I do rotate my Carbon Copy Cloner images/backups -- and they are encrypted.

This is a good feature if used wisely.

Obscure bug: Reeder share to Facebook will post to last used Page Identity

This is an oddball bug. I suspect it's not fixable.

I used iOS Reeder to share a Facebook post.

Problem was, when I last used Facebook I used it as one of the Pages I maintain. That's how Facebook Pages work; you assume the Page identity while you post to it. Then you flip back to your own identity.

So Reeder shared my personal post on my public Page. It wasn't too embarrassing, but it was clearly wrong for the Page (org).

It's a weird bug; I suspect the flaw is in Facebook's API. It probably isn't designed for this use case. I also wonder if this is a side-effect of Facebook introducing the Timeline to my personal account.

Wiki impressions: XWiki

After a disastrous SharePoint 2007 to 2010 migration [1] it was clearly time to replace my team's use of SharePoint wiki. The mangled conversion was a red-flag-and-air-raid-siren declaration that SP was going to be a longtime world of hurt.

We looked around at the wiki options, and considered TWiki, XWiki and Confluence (WikiMatrix). Atlassican Confluence is the best of breed, but for corporate users like us it's expensive. Hardware we have, platform licenses we have, networks and backups we have, and expertise we have. What we don't have is a budget.

TWiki is the easiest of the three to configure and it has the fewest technical requirements. XWiki looked like more work, but we liked the rich text editor and (limited) table support. We're now starting up on XWiki enterprise. Here are a few impressions from the process that may be of interest to other wiki searchers ...

• We started out on a Linux server but couldn't get XWiki working. We're not Linux experts. We switched to Windows and our engineer resource had XWiki setup within a day. I wasn't impressed with the ease of installation.
• XWiki requires a Java Servlet Container. That rules out Dreamhost and many other hosting options. (TWiki can install with Dreamhost.)
• XWiki includes a blog service but there's no support for Blogging APIs. So you can't use MarsEdit or Windows LiveWriter to write.
• The documentation is weak. For example, how do you delete a Space? Turns out it's easy if you have privileges, but the documentation claimed we needed an extension.
• XWiki Enterprise is a geek platform, though much of the complexity can be hidden.
• I like the approach to URLs and page titles. You can change a page title, but the page URL is fixed. XWiki provides an effective UI for looking up local pages during link creation. (SocialText retitling creates a redirect page with the old title/url and a link to the new title/url. Sharepoint changes the URL and title, but updates intra-liki links. Foreign links break.)
• The rich text editor is simple but works fine for my purposes.
• IE 9 doesn't work with the controls on gadgets, but seems ok for non-admin users.
• XWiki has the universal wiki curse -- no ability to migrate posts between systems. I wonder if people who complain about being unable to move their medical records between providers see the similarity. OTOH, when our Linux install failed it was easy to move to Windows.

I'm reasonably optimistic that XWiki will work for us. I'm glad to be free of Sharepoint 2010.

See also:

• XWiki - Open source, LGPL, WYSIWYG
• Offline Browsers (web robot) – options to review
• How can I create a local copy of a Sharepoint wiki?
• A workaround for image uploading to Microsoft’s Sharepoint Wiki.

[1] With SharePoint it's not possible to separate software issues from implementation. Maybe Microsoft provides wiki conversion tools that our IT department didn't use. I have discovered that if I copy/paste of our 500+ SP 2007 wiki created pages into Word, then remove all styles, then past back into SP 2010, I can edit them again.

Time Machine backups of Aperture are not reliable?!

I've restored my Aperture Library from backup twice in the past few months. Two months ago Aperture crashed when a bad block corrupted a file. I restored from Carbon Copy Cloner. A month later I figured a dying iMac drive was the root cause, and my backups weren't as robust as I expected.

After a mildly painful Apple service call i had to restore all my data from backup. I backup the server nightly to Carbon Copy Cloner and hourly to Time Capsule, so I had to pick a backup source. I decided to restore from Carbon Copy Cloner, though I did consider Time Machine. It seems to have worked.

Today I learned that I might have dodged a bullet. I was wise to choose CCC over Time Machine ...

Macintouch Backup

Derek L

...Thanks to Antonio Tejada for his note about Time Machine and Aperture. The combination remains unreliable for me, as I'd described in my comment on this topic last June: not all items in my Aperture library get included in my Time Machine backup. Although I've been able to temporarily repair it via forcing a "deep traversal", the problem recurred in fresh TM backup sets, on multiple disks, and through several point releases of Snow Leopard and Aperture 3 (I've never used any other version, and I also haven't migrated to Lion). I gave up on it and instead depend on Aperture Vaults.

Skot Nelson

... However, after having Aperture randomly lose some of my masters -- old photos from a concert that I hadn't touched in month and just happened to click on as part -- I no longer trust my Aperture library's integrity...

Richard Tench

... Seeing the warnings here about Aperture and Time Machine, I decided to do a test recovery (to my desktop). It failed due to permissions on the Time Capsule. Though one would think that's an easy problem to fix, it isn't. At the end of 2.5 hours on an AppleCare call, they were unable to help me. I was told that Aperture doesn't work well with Time Machine...

Sigh. I miss Dantz Retrospect. I'm grateful for CCC, but old Retrospect combined the reliability of CCC with the features of a true backup.

It's been noted that in 2012 Apple OS X development is the equivalent of Siberian exile. Time Machine work must be reserved for unwanted interns.

See also:

• Backups - why you need two methods and abundant paranoia 12/2011
• Freeing up Time Capsule space – and documentation for Time Machine and Time Capsule 12/2009 (Time Machine is flaky)
• Lessons from Apple Store out-of-warranty repair of a Seagate drive 1/2012
• Lion continues to disappoint: the Duplicate/Save problem 1/2012
• 7 facets of a good Mac backup strategy « Haystack Software Blog 1/2012 - advice from the developer of Arq (backup software for S2) -- but absolutely correct. I do this and I'll evaluate Arq.

SharePoint 2010: migration of a SP 2007 wiki

My business group has maintained a large SharePoint 2007 wiki.

Recently we had to migrate to SP 2010. The wiki went along for the ride.

I've seen some bad outcomes in software over the past 20 years. I mean -- really bad products.

I've never seen anything like this.

Wiki pages that look like they were edited by intoxicated baboons. Pages that can't be edited in IE 9, but can be edited in Chrome. Pages that can't be edited in Chrome but can be edited in IE 9. Pages that can't be edited anywhere, save as HTML.

SharePoint Designer is an option. I tried it. I shed tears for the lost spirit of FrontPage and Vermeer.

I despair, sometimes, when I see what Apple's interns are doing to iPhoto or OS X. Even they, even they, have never done anything like this.

This is the worst.

Microsoft, astonishingly, is truly finished. We really are in the twilight of the personal computer. 

My MacBook fan was roaring. Again. Time Capsule edition.

My MacBook fan was roaring. Again.

This time, however, activity monitor didn't show much going on. It wasn't a Flash ad running in another user account. I didn't have a print job stuck in the bowels of Apple's dysfunctional printing framework.

It had been doing it for weeks. Sluggish performance, slow fan pinup, then continuous running. Something was draining performance and making heat -- and it wasn't showing up in Activity Monitor.

Or, at least, it wasn't obvious in Activity Monitor. I did see something called fsck_hfs using up 10% or so CPU.

To make a long story short - the problem was a defective Time Capsule backup. My MacBook was running fsck_hfs, a utility that "verifies and repairs standard HFS and HFS+ file systems". When I saw this I thought there was something wrong with my system drive - but it tested out fine.

A SuperUser tip clued me in. My MacBook was trying to verify the integrity of my 150GB backup -- over a WiFi connection. This is a singularly ineffective strategy, it would have taken days to complete. The laptop never ran continuously for that long; and there is something about this process that makes a MacBook run hot [3]

If I'd known what to look [1] for I'd have seen something like this:

Except it would have shown 2-4%. It only got to 92% when I connected the laptop to my TC via wired ethernet. It got to 92% then stuck there. The backup was toast.

I then disconnected all users and tried deleting the .img file for my MacBook from TC. That failed because the Finder can't handle TB scale image files [2]. I then used AirPort Utility's very confusing Time Capsule UI to erase the entire drive and started the slow, painful, recreation of my backups.

My MacBook is quiet and responsive again, and fsck_hfs free.

That was painful.

Backup is an unsolved problem.

See also:

• My MacBook fan was roaring. Why? 5/2008.
• Obnoxious old Epson Scan bug: EPSON Scan cannot be started 2/2008
• OS X 10.5.5: CPU pegging with Firefox 9/2008.
• When Safari locks up - Kill Flash process
• iMac G5: fans running more often 10/2006
• osx - Why has my Mac been running fsck_hfs for two days now? - Super User - great discussion of this problem
• Time Machine. Backup failed with error...: Apple Support Communities 9/2008

[1] I wasn't so direct, instead I wandered about searching Console error messages. There I found:

search console for com.apple.backupd (all messages)

Attempting to mount network destination using URL: afp://;AUTH=No%20User%20Authent@Molly.local/Molly_Internal

Backup failed with error: 21

Error writing Time Machine Information file: /Volumes/Molly_Internal/Stanford_MacBook_0017f2f04828.sparsebundle/com.apple.TimeMachine.MachineID.plist

Error writing to backup log. NSFileHandleOperationException:*** -[NSConcreteFileHandle writeData:]: Input/output error

Event store UUIDs don't match for volume: Escanaba

While plumbing Console I discovered an unrelated error that distracted me for a while - a pile of errors like this:

Jul 20 18:27:55 localhost com.apple.launchd[1] (com.apple.SystemStarter): Failed to count the number of files in "/System/Library/StartupItems": No such file or directory

Googling on this one I discovered this is an old OS X bug, one some systems a routine update deleted this folder. I recreated it using terminal:

cd /System/Library

sudo mkdir StartupItems

That cleared up a bunch of Console error messages, but it didn't fix my real problem.

[2] Kind of amazing, but there you go. Apparently they can be deleted via terminal, but erasing through TC is safer. You can only erase the internal drive, if you want to erase a TC mounted external drive you need to move it to a Mac and erase it there.

[3] The MacBook runs hot whenever it does an 802.11n TC backup. I wonder how much of this is heat output from the WiFi/encryption systems.

Organizing kid school accounts with OS X: Chrome to the rescue

In the twilight of the general purpose computer, I struggle to balance OS X and Apple tech, Google services, parental obligations, and getting work done.

Our iOS  and OS X devices are parental controlled -- at least as far as they can be. Among other things, that means Google services are unavailable on child accounts. [1].

Schools, however, make increasing use of Google Apps [2]. This is how I reconcile that use case with our general approach to home computing:

1. You need the username and password for the school Google Apps account. Example: kid_name@school.mn.us.
2. Create a single non-controlled "homework" account on the primary homework machine.
3. Use Google Chrome, not Safari, for this account.
4. In Chrome create a user account for each child. For each account, from Chrome Preferences, choose to sync Google. You will be asked for the school user name and password.
5. Add gmail, docs and so on to the toolbar.

Each child uses this single OS X account with their own Chrome identity. Use of this account requires direct parental supervision. It is used only for homework. On personal OS X accounts our kids don't directly access our Family Google Apps domain, they use OS X Mail.app, for example, to get email. They don't know their Family Google Apps passwords.

[1] Partly by design and partly due to market disinterest, Google services are not compatible with OS Parental Controls.
[2] Alas, this transition occurred even as Google's Hyde crushed its Jeckyl.

OS X shared printing: Is it just me?

Is it just me, or is OS X shared printing broken beyond all repair?

I spend an inordinate amount of time dealing with print failures on Leopard and/or Snow Leopard shared Brother USB printers using Apple provided drivers. The only reliable printing I've had with OS X was with an ethernet connected network printer. [1]

Yet another example of why general purpose computers are dying (and I've a lot more to say on that ... later).

[1] The very best, most reliable, printing I ever experienced was using Mac Classic, AppleTalk, and Apple's LaserWriter Select (not sure of product name) in the early 1990s. I'm planning to donate our Brother HL-2140 and buy a Brother networked printer.

Lessons from Apple Store out-of-warranty repair of a Seagate drive

Mysterious application crashes, including corruption of Aperture file data, exposed bad blocks on a slowly dying 2 yo iMac drive. Modern drives aren't supposed to show their bad blocks, and of course they should never corrupt a file. So I knew the drive needed to be replaced.

The 27" iMac 11 (i5) is not user-serviceable. It can be done, but it's hard to keep dust out of the display. So I compared repair costs at FirstTech in Minneapolis to the Roseville MN Apple store. FirstTech's service costs were about $200, but Apple's service fee is a flat $40 (offsets the high cost of their proprietary designs). Both charge far more than retail for a hard drive. Apple dinged me $226 to replace a 1TB Seagate 7200.12.

The replacement drive is an ST31000528ASQ revision AP24 S/N 5VP9Z4TC. I was told it had a 90 day warranty. A comparable new drive from Amazon is $134, so Apple adds a $100 markup on the drive. (Of course I'd have preferred a 2TB replacement, but Apple out-of-warranty repairs follow warranty rules -- so no upgrades.)

Overall the Roseville MN Apple store experience was mediocre. Some of this is because of Apple policies, but mostly this store is bursting at the seams at this time of year. Apple needs more retail.

If I were to do it again I'd order replacement parts from a quality Mac after-market vendor and I'd do the repair myself, or I'd order the part and pay FirstTech to do it. When an out-of-warranty Apple repair makes sense, I'd look for a quieter Apple store.

Here's what I learned for future reference:

• Apple wanted to keep my old drive. They can probably get money from Seagate. This was a problem because I hadn't secure-wiped the drive, and although my passwords are on an encrypted image I prefer not to have our family data floating around. I also wanted to reformat and stress test it myself, and decide if I could use it as an emergency store. Store management wrote that "I needed the drive for data recovery" and that let me keep it. (Not true - I had 2+ complete backups.)
• I thought I was told the drive would cost about $160, but in retrospect that was probably my mistake. $160 would be low markup. In any case, get estimates in writing if possible.
• When I went for an estimate I was told to expect a 3 day turnaround. In fact it took about 9 calendar days -- including 3 days to repeat the drive test and confirm the bad blocks. It took so long it ran into a business trip; the store was grumpy about storing it until I returned.
• They expect to have a guest or maintenance account available for testing. I had none on this machine. They were able to test anyway of course, but this is worth knowing.
• The machine had additional memory added, but the repair receipt listed the original memory. This was mildly worrisome but it came back with its bits.
• The installed drive was formatted with the same OS as the old one - Snow Leopard.

I tested the drive with Tech Tools Pro. The SMART check on the prior drive showed no problems except out-of-range temperature variation. The SMART check on the new drive also shows out-of-range temperature variation! I hope this is a quirk of TTP and the iMac's thermal regulation system rather than a drive problem. The block scan passed 1,953,525,168 blocks, 0 bad. (Although no bad blocks should ever be exposed, the good block total will fall over time.) The overall SMART test also shows a tendency to read errors, though still within normal limits (Hardware ECC recovered, Raw Read Error Rate).

Lastly, I chose to do my restore from a bootable Carbon Copy Cloner backup rather than my Time Capsule backup (I trust CCC more). I'd never done a full clone restore on a Mac; it worked well but there were a few quirks...

• During my initial testing I'd created a user account on the new disk. The default CCC restore would have left those files in place -- which is an abnormal install state. Also, the default restore seems to leave "more recent" files untouched, which might produce a mixed version system.
• I set the restore to overwrite "more recent" files and to move non-matches to an archive folder.
• After the install completed I deleted CCC_archive on the target system
• After the restore it appears TC is backing up my entire system. That will take a week over the home wireless.

For extra insurance I'm going to leave my CCC backup untouched and rotate it offsite. My routine backups will be to an older CCC backup and to Time Capsule. In my initial testing however the restored data appears fine. I also have my original drive which is fully readable, I'll wait a month or so before I wipe that drive and stress test it.

Update: Time Capsule is trying to backup 340GB over WiFi; it's doing a full backup. This will take weeks. I've set my TC backup to omit all but Users, tomorrow I'll bring it to the computer and connect by ethernet.