Aperture woes - and iPhoto 6 image export

[PS. If you have trouble reading this, note that Blogger's formatting and editing tools are astonishingly broken. Don't use bullets!]

Apple - Support - Discussions - Aperture performance vs. iPhoto ... sums it up. Aperture 1.1 is just too darned slow save on the very highest performance machines and the very highest end GPUs. It's an application for the next generation of Macs. The only caveat is that there may be a way to make it workable by constraining views to smaller thumbnail subsets.

So now I'm on Plan B. This is using Adobe Elements for initial RAW and JPG image review and edits, then export as JPG 99% and store in iPhoto 6. Alas PNG won't work since it strips metadata and JPEG2 won't work because of unclear metadata standards and an uncertain future. TIFF 6/EXIF with compression or DNG (basically TIFF 6 with better metadata) is the other option, but when iPhoto edits TIFF I think it stores the edited file as uncompressed TIFF -- huge.

In the meantime I did some experiments with iPhoto image formats and export/import. I used GraphicConverter 'save as' to make some of these inferences. I'll update what's below with more testing:

• If I import RAW from my Digital Rebel XT into iPhoto 6.02/10.4.6 the resulting image uses the Adobe RGB color space.
• iPhoto image export does not include an image preview (good!)
• If I edit a RAW file in iPhoto the edited file format is JPG. Oddly enough iPhoto has a "reprocess RAW" option, which can only transform the original, not the JPG. I assume that's related to how RAW processing occurs in Aperture. [Note: There's an advanced iPhoto option to save edited RAW as 16 bit TIFF. This would not only change this behavior, I suspect export original might export the edited TIFF rather than the CR2 file. Since 16bit TIFF files are enormous I didn't do this.]
  
• When iPhoto exports as PNG, JPG or TIFF it exports the edited image. When it exports as "Original" it exports the edited image for PNG or JPG, but the original image for RAW/CR2. The latter makes sense when one realizes that iPhoto always converts CR2 to JPG for editing, there's no "edited CR2" to export. So the app has to choose between exporting a JPG (clearly wrong) and exporting the original (not so clearly wrong, but not right either).
  
• When iPhoto exports TIFF the metadata is well preserved on round-trip but there's no compression at all. A 5-6 MB PNG vs a 22MB TIFF!
• When GraphicConverter converts TIFF to JPG (resource fork turned off, the fork adds about 100K, use QuickTime for compression) there's very little growth in file size from a 90% to 100% image quality setting. When using GC to produce images that will be archived, there's no reason to drop below 100% quality.

Some interesting comparisons on an exported image from iPhoto (image sizes in millions of bytes, not in MBs). Note that Preview is an amazing little application, but it doesn't let one save CR2 or 16bit TIFF as anything but lossless.

TEST ONE

Original CR2: 13.27 converted to:

JPG w/ iPhoto: 4.1
PNG w/ iPhoto: 13.1
TIFF 8 bit w/ iPhoto: 23.9
TIFF 16 bit LZW w/ Preview: 62.23 (Note that CR2 RAW images have 12 bits of color. JPG only has 8. TIFF has either 8 or 16; one sacrifices date, the other wastes disk space.)
PSD w/ Preview(!): 47.7

The 8bit TIFF converted to:

JPG BEST w/ Preview: 8.56
JPG 100 w/ GC: 4.11
JPG 99 w/ GC: 4.03
JPG 95 w/ GC: 4.01
JP2 Lossless: 8.63

TEST TWO

CR2: 11.1
JPG w/ iPhoto: 2.9 (23.83 as TIFF)
JPG in camera: 3.54 (23.89 as TIFF)

Wow. This surprised the heck out of me. A few comments as I figure this out:

Preview has a lot of unexpected capabilities. Most apps, including GraphicConverter, can't fully handle 16 bit TIFF, but it has no problem.

I'd figured from my first test that iPhoto's JPG export was minimally lossy, the only less lossy export coming with Preview -- it seems to save as lossless JPEG (rare format). In my second test the in camera JPEG is significantly larger than the out-of-camera iPhoto CR2 to JPEG result. I wondered then if the camera was losing less data than the iPhoto CR2 to JPG conversion. However, when I normalized both by using GC to save as TIFF, the sizes were 23.89 and 23.83 respectively. In other words it in-camera JPEG is simply a less efficient compression, it holds no more real data than the JPEG iPhoto produces from the CR2 file.

More conclusions later, but for now:

CR2 is a very compact format. In terms of capturing all image data, the next jump is to PSD. Almost 400% growth required.

iPhoto's JPG output is better than I'd expected. It's the equivalent of GraphicConverter's 100% option. Also to my surprise, there's no reason to use anything less than 100% when saving images in GC -- file size savings from greater compression are minimal.

Preview may support lossless JPG, which is almost unheard of. JPEG2000 lossless gives similar results.

iPhoto is a very sophisticated product.

My new workflow is as follows:

Use RAW in my camera.

Import images from camera into an 'iPhoto Intake Library' (I use IPLM to switch libraries).

Edit in the Intake Library. When done export as JPG. Note, however, images will have Adobe RGB color space rather than my preferred sRGB.

Rename the exported images using my personal convention (date/roll info is part of file name, no possible future name collisions).

Import into iPhoto for storage.

My primary concern with this is that I'm working in Adobe RGB rather than, as I prefer, sRGB.

Why PNG sucks: It's the metadata, stupid

I've been experimenting with various file formats to use with iPhoto and RAW images. iPhoto generated TIFF files completely uncompressed and thus quite large, iPhoto doesn't provide JPG fine tuning (lossy compression control), JPEG2000 is still not widely used, etc. PNG seemed a good option -- reasonable sizes, well specified, decent color tables, etc.

But PNG has a lethal flaw, as documented by the Library of Congress digital preservation project:

PNG

The PNG specification allows labeled text (ASCII or UTF-8) elements to be embedded in text chunks and predefines a few standard keywords(element labels): Title, Author, Description, Copyright, Creation Time, Software, Disclaimer, Warning, Source, Comment. The compilers of this resource are not able to assess the degree to which such metadata is found in practice or whether other keywords are in common use. An attempt in 2000 to develop open source tools to convert EXIF images (including EXIF metadata) to PNG seems to have been abandoned. See
http://pmt.sourceforge.net/exif/drafts/d020.html. Without such tools and agreed practices, PNG can not rank highly for self-documentation.

In other words, there are no standards, or even exif-like pseudostandards, for embedding metadata (time image acquired, etc) in PNG images. Obviously there are no standards in common use for associating a portable metadata XML file (for example) in a bundle with a PNG image.

So 'save as PNG' means toss out the metadata. PNG is worthless as an image archival format. Shame.

BTW, this LOC site is an exception reference. In contrast to the PNG tragicomedy, note the discussions of JPF (still in the twilight zone) metadata ("self-documentation"). There's some very sophisticated thinking there, unfortunately no-one supports this yet ...

All JPEG 2000 files are made up of "boxes," as described in the Notes below, including an XML box typically used for metadata. Regarding JPX_FF, Annex N of Part 2 of the specification provides detail about metadata and offers but does not require a specification based on DIG35 elements. This metadata specification includes four broad metadata categories: (1) image creation ("how," e.g., about the camera), (2) content description ("who," "what," "when," and "where"), (3) history ("how the image got to its present state," i.e., provenance metadata in the digital library lexicon), and (4) intellectual property rights (IPR) metadata (which may be used in conjunction with technological protection systems). Additional boxes inherited from JP2_FF include one for a unique identifier for the image or identifier-references to other digital objects, e.g., a UUID, and another for IPR metadata, possibly redundant with that included in the XML box.

JPF wraps JPEG2000 compression formats, which are proprietary and have lots of IP issues. Contrast now to Adobe's open DNG (described as a "subtype of TIFF 6"):

See TIFF_6. Additional metadata may be embedded in a file using tags from (1) TIFF/EP or EXIF_2_2 (see also TIFF_UNC_EXIF), (2) IPTC (TIFF tag 33723), and (3) XMP (TIFF tag 700). Regarding TIFF/EP and EXIF, the specification states that TIFF/EP stores the tags in IFD 0, while TIFF_UNC_EXIF stores them in a separate IFD. Either location is allowed but the EXIF location is preferred. Proprietary metadata that may be used by camera manufacturer's raw convertors is to placed under private tags, in private IFDs (Image File Directories), and/or a private MakerNote. (pp. 12-13)

TIFF 6 appears to be the preferred format for image archiving by the Library of Congress. I can't tell what version of TIFF iPhoto produces. It's not compressed, but that doesn't say much. It does contain EXIF metadata, so maybe it's TIFF_UNC_EXIF.

On review I'd say we need iPhoto to export to DNG with compression and use DNG as a native lossless format.

Between the Devil and the Deep Blue Sea

iPhoto 6 is fast and responsive. But ... I export as JPG and what quality setting can I assign? None. I export as original, and what's exported doesn't have my edits (for the RAW files anyway).

And there's no supported way to combine or merge iPhoto Libraries.

Ok, so then there's Aperture, now at 1.1.. Still buggy by all reports. Still crashy. Slower than before, and it used to be very slow.

Arrgggghhh.

Play music without adding to iTunes Library

Macworld: Mac OS X Hints: Listen without adding to library in iTunes

Make sure you can see the iTunes window, specifically the Source column, along with the song you wish to listen to. Hold down the Option key and drag the song file into an empty spot in the Source column. iTunes will create a new playlist containing the dragged song, but it will not add it to the library. When you’re done listening, just delete the playlist. The original file will be untouched, and once the playlist entry is gone, iTunes will no longer be aware of the song, either.

OS X as OS/2: XP API in 10.5?

Will history repeat? IBM had the right to use the Windows API in the 1990s. That's why OS/2 could, really, "run Windows better than Windows". Alas, Microsoft did things with their source code and API to break OS/2's version of the API.

Cringely claims Apple has the same API rights that IBM once had, and that this time will be different:

PBS | I, Cringely . April 20, 2006 - Native Speaker

... I believe that Apple will offer Windows Vista as an option for those big customers who demand it, but I also believe that Apple will offer in OS X 10.5 the ability to run native Windows XP applications with no copy of XP installed on the machine at all. This will be accomplished not by using compatibility middleware like Wine, but rather by Apple implementing the Windows API directly in OS X 10.5.

Huh?

Wine is great, but it is also a moving target subject to Microsoft meddling. If Wine gets too good, Microsoft can "accidentally" break it at will. But Microsoft can't afford to do that with its own Windows API. The courts will no longer allow checking for a different underlying OS as Redmond did back in the days of DR-DOS. Besides, unless we are strictly talking about Microsoft apps, there isn't even much code involved here that Microsoft CAN meddle in. The wonder is, of course, that Apple could even dare to do such a thing?

Oh they can dare. Not only that, this is one dare Apple can probably get away with.

Remember Steve Jobs' first days back at Apple in 1997 as Interim-CEO-for-Life? Trying to save the company, Steve got Bill Gates to invest $150 million in Apple and promise to keep Mac Office going for a few more years in exchange for a five-year patent cross-licensing agreement? The idea in everyone's mind, of course, was that Microsoft would grab lots of Apple technology, which they probably did, and it quite specifically ended an Apple patent infringement suit against Microsoft. But I'm told that the exchange wasn't totally one-way, that Apple, in turn, got some legal right to the Windows API.

That agreement ran for five years, from August, 1997 to August 2002. Even though it has since expired, the rights it conferred at the time still lie with the respective companies. Whatever Microsoft grabbed from Apple they can still use, they just aren't able to grab anything developed since August 2002. Same for Apple using Microsoft technology like that in Office X. But Windows XP shipped October 25, 2001: 10 months before the agreement expired.

I'm told Apple has long had this running in the Cupertino lab -- Intel Macs running OS X while mixing Apple and XP applications. This is not a guess or a rumor, this something that has been demonstrated and observed by people who have since reported to me.

Think of the implications. A souped-up OS X kernel with native Windows API support and the prospect of mixing and matching Windows and Mac applications would be, for many users, the best of both worlds. There would be no copy of Windows XP to buy, no large overhead of emulation or compatibility middleware, no chance for Microsoft to accidentally screw things up, substantially better security, and no need to even take a chance on Windows Vista.

It would be great to have this functionality. I suspect Microsoft could still break support however, probably with an XP service pack. The DRM features they'll eventually require for Office won't be compatible with XP, and Apple won't have the Vista API. At most it's a transitional solution, though it might be a winner for a year or two.

Cringely also claims that 10.5 won't run on PowerPC CPUs due to kernel swap. Alas, I can believe that. Apple can be very cruel about obsoleting hardware; that's their perverse incentive. (Microsoft's perverse incentive is to break file formats -- of the two I actually prefer Apple's behavior). Maybe they'll give PowerPC users a coupon!

Update 4/22: Gruber politely and convincingly tears up this column. So, virtualization yes, but not likely API integration. Kernel change maybe, but probably not until 10.6.

Keylock and usability in the Palm Tungsten

My Tungsten E2 was dead in the water. It wouldn't respond to any keypresses or gestures. I figured either a disastrous hardware failure or a glitch with some of the old odd software I rely on. I was ready to try a hard reset. To do that one must press and hold the power switch -- a switch I rarely touch because it's known to fail in the T/E2.

That brought up the keylock warning dialog. Keylock had accidentally engaged, that's why nothing worked. I had only to press and hold the powerswitch to get the dialog to disable it. Press and hold turns it on.

Fascinating! Is it really a usability flaw, or more a side-effect of my use of the command buttons rather than the power switch to turn on my Palm? (I do that to avoid wear on the fragile on/off button.)

I've now disabled keylock so it never activates. It was previously set to manual activation.

Update 4/22: On reflection, I think it's a usability bug. With keylock on pressing the power button gives the message that it's on. However, pressing the other buttons that start the PDA do not produce this message. With keylock on, pressing any button that would normally power up the PDA should produce the same keylock notification message.

World's best USB drive enclosure?

I bought this USB enclosure for my work backup drive about a year ago. Not only has it worked perfectly, it's quiet, fan cooled, has a perfect rating from over 280 geeky NewEgg buyers and is still sold:

AMS VENUS DS-2316B2BK Black External Enclosure - Retail at Newegg.com

It may just be the perfect USB drive enclosure. I'm buying two more to support my home backup.

Update 6/1/06: Never fails.

I write something nice about a device and it shows some warts. See the fan update to an earlier post.

Update 9/1/06: No spin down.

This is the killer requirement, but almost all USB enclosures have this problem. Dan's Data tells us that spindown is essential for the survival of consumer hard drives. I have my laptop set to spindown drives after about 15 minutes of non-use, but I don't think the drive ever spins down in this Venus enclosure.

In summary, requirements for an ideal USB/firewire enclosure:

• drive spindown
• extraordinary passive cooling (not sure this is possible)
• high quality large circumference very quiet fan that's easy to replace (if not passive cooling)
• well designed wall wart (I prefer the ones where the transformer is a foot or so from the plug)

I'm still looking, but maybe the Network Storage (NAS) devices will be the real answer.

Update 11/16/08: This is the same drive family as reviewed in 2004 by Dan's Data. I'm looking at this because my backups have exceeded the capacity of the drives in these enclosures. So I'm looking for 1TB drives but, for the sake of convenience, will probably buy then bundled with an enclosure. Now if I could find an enclosure that spins down the drive ...