Someone else has noticed a one year lifespan for a modern XP hard drive

I have two corporate XP machines, and between then I have to replace a drive every six months. My home XP and OS X drives last for years.

So I now have two completely unrelated corporate backup systems that run nightly.

I haven't seen anyone else comment on the lifespans of corporate hard drives, so I liked this post (emphasis mine)...

DadHacker » Blog Archive » Thoughts while rebooting

On the IT-ridden machines I regularly have to swab out twenty megabyte log files, logs from things that I didn’t even know were running on the machine, and when I find something like “ArScnr38″ running I have no idea if it’s spyware or something that an IT monkey stuck on my laptop to scan my Excel spreadsheets.

It’s hilarious when four different scanners are fighting for disk access. No wonder our drives are dying after like a year in service. I don’t work late, so I can only imagine what the buildings sound like at 3AM when Windows Update goes into its happy dance and reboots every single workstation.

“Shhhh… wait for it.”

“What, Dad? I’m sleepy.”

“Any second now…”

clikclickClickClikCLIKCLICKCCLLIICCKK-CLICK-***KA-CHUNGGGKGKGKG!!!!!***

“Wow! Do the lights flicker like that in every time zone?”

“That was nothing. Wait until they all ask the DHCP servers for an address!”

...The IT philosophy of bloat appears to be: “Screw the user, we own the machines, and if they can’t get work done with them then they can’t do any damage. More scanners! And loggers! And Java-based enterprisey things with *****up XML configuration schemas! If there’s CPU or disk space left we’re not doing our jobs; we have to pay for that call center expansion somehow!”

That's what I see. Between my heavy duty database work, the antivirus scanner, the corporate HP monitoring systems, Windows Search indexing, and the two nightly backups the hard drivers are being worked to death.

I really need to switch to an in-office NAS with a hot-swappable RAID array -- so I can rotate out bad drives without the hassle of a restore.

Aperture's SQLite Database

I'm going to fire-up the SQLite Database browser and follow these directions: The Deep Dark Depths of the Database (and some fingerprinting) - Inside Aperture. I wonder what iPhoto is doing these days -- be interesting to look for SQLite files in other Apple applications.

I recall that SQLite was thought to have threading issues with 10.4. I wonder if that's better with 10.5.

A last lesson from our Samsung i500

I like my iPod. I liked the Palm Vx. I even mostly like my MacBook.

So I ain't impossible to please. Just not easy.

So it was hard so say the long good-bye to our Samsung i500s and all their accesories!

They were great tech for their day and today. Palm Classic goodness -- fast, super reliable, elegant. Graffiti One. Compact clamshells with good sound quality and an excellent form factor. Line 'em up against the iPhone and they do pretty well. Sure -- no mobile web. On the other hand, way more reliable and they have tasks too!

Alas, they gradually died. When my wife's failed we switched to AT&T for her BlackBerry and to line-up for my iPhone.

There was one major flaw of the i500 though -- and one related lesson. All the connectors were proprietary - and transient. No USB goodness (BlackBerry, Nokia 6555), and no evil-but-ubiquitous connector like the iPod -- just another one off from Samsung.

So, when a "free for anyone" listing on Craigslist got no hits, I had to toss everything out. Connectors, chargers, accessory cables, -- the whole bit.

That's the last lesson, though most of us have figured it out by now. Don't buy devices that use proprietary connectors -- unless they've made it to evil-but-universal status.

It's a great way to narrow your purchasing options ...

Introduction to the relational database

ATPM 14.04 - FileMaking: Getting Relational turns out to be a fairly comprehensive and quite readable introduction to relational database design. I didn't see anything that's FileMaker specific, it applies equally well to MySQL, Microsoft Access, Oracle, etc. (The promised f/u articles will be FileMaker specific.)

It's a handy reference to keep around for colleagues, students, etc.

Charles Ross covers a range of topics typically taught over the course of several lectures, so don't be surprised if it takes a while to get through the entire article. If you need to understand the topic, this is worth printing out and studying.

More iPhoto humiliation: the vanishing edited video

iMovie '08 is so abominable that it makes iPhoto look good ... by comparison.

Still, it's not hard to humiliate iPhoto, even if we forget that after 8 releases it still can't import a Library.

We know that iPhoto has a nasty video export bug that can result in total data loss. That's bad, but this is at least comparable:

1. Import 3ivx MPEG 4 encoded video with AVI wrapper into iPhoto.
2. Right click on the Edit button and open in QuickTime Pro.
3. Trim the video and save. QT Pro offers to save in .MOV format. Choose that.
4. Return to iPhoto. The video you see is the previous version.

Basically the "Edit" option in iPhoto doesn't work for Video. It should be "grayed out" and unavailable for selection.

I bet QA found this one in early testing of iPhoto '08, but product management decided it wasn't worth fixing.

They should be ashamed.

Firefox 3 beta: they didn't tackle the tab UI problem

I've been using Firefox 3 beta 5 with OS X. I've noticed a few problems -- windows initially appeared almost off screen, one window would abruptly close, but overall it looks like a good incremental improvement. It hasn't pegged my CPU yet, so that's encouraging. (Firefox 2 would do that without any help from Flash.)

Alas, they didn't do anything to tackle the multi-window tab UI problem. I was looking for a view that would render windows and tabs in a browseable outline view with tab names and mouse-over window views, but there's no sign of this.

Lesson from FLIP Video: your video is doomed

Petabyte-equivalents of home video have vanished over the past 40 years. Odd sized films, film degraded and glued together, beta tapes, odd VHS variants, digital tapes, digital disks, abandoned digital file formats ... there are many routes to the graveyard.

Things are not about get get better. I don't have any data, but I'm blogging -- I don't need any friggin' data.I'm guessing most video now is coming from cell phones, digital cameras, and new packagings like FLIP Video. So how well will this video do over the next 40 years.

Digital cameras commonly use "Motion JPEG" [1]. Mobile phones use all kinds of formats, but they're converging on a wrapper called "3GP", behind that wrapper are all kinds of semi-standard data formats. The FLIP Video camera the kids and I are using on our Wisconsin Dells holiday uses 3ivx, of which I recently wrote:

Gordon's Tech: FLIP Video Ultra camcorder: iMovie HD works, iMovie '08 doesn't

...Videos are in "AVI" format -- that's a metadata wrapper around a codec. In this case coded is 3ivx MPEG-4... 3ivx Technologies is hoping you'll buy the full version from them...

I ended up writing an extended post with lots of updates; I learned quite a bit about iMovie '08 (disgraceful, Apple shipped at least one year too soon [3]), QuickTime and QuickTime Pro, MPEG Streamclip, video formats, video codecs, editing software, etc.

I haven't quite figured out the optimal strategy for editing and storing FLIP Video 3ivx encoded video, but I think there's a very good chance the 3ivx files will be completely unreadable within 15 years. It's a completely proprietary format, with no particular reason to expect it to become a lasting "standard".

So what will last? Well, I'm betting reasonably compliant still image JPEG will be readable a hundred years from now [2], so I think Motion-JPEG video might persist. Motion-JPEG's simplicity makes it easy to edit too, assuming one doesn't try to convert from the highly JPEG compressed images to any lossy format. I'm not so sure about the sound formats though. DVD-Video without copy-protection might also be expected to last, but that simple name hides a lot of complexity and variation with sound and video compression standards and metadata wrappers.

So called DV-stream is one name given the streamed version of 'digital video' standard (actually is a standard), but a casual glance at Wikipedia reveals lots of room for incompatible variations on digital media. It does lend itself well to editing (no intra-frame compression) but it makes for huge data files. It's dying off as MPEG-4 and HDV "standards" take over.

Hmm. Sounds like a real mess. I've read lots of discussions about archival image formats, so I'm sure this Google archival video search will yield lots of great advice.

Well, as of April 2008, not so much great advice. Basically, no advice at all.

Those petabytes of lost home 8mm home film recording are about to joined by peta-giga-tera bytes of every conceivable unreadable combination of video, audio and metadata formats.

Don't get to attached to those precious video moments ....

-- Footnotes --

[1] I'm still figuring out what this corresponds to in QuickTime Pro's export menu. I think it's the JPEG export option that shows up in the video export list, but I've reason to suspect things are even more confusing than they appear.

[2] Assuming our non-human inheritors are curious about their precursors. JPEG is what I store my photos in, I assume the original RAW files will be absolutely unreadable within 10 years.

[3] Seriously. The more I play with it the worse it gets. For example: even the one or two video formats it can import aren't recognized when the same files are stored in an iPhoto repository. If anyone ever has the delusion that Apple has some special magical interest in their customers, they need to review the iMovie '08 story. I use a lot of Apple products, but they're not marvelous. They're only better than the alternatives.

HP 15C emulator for the iPhone

Found this via Cosmic Variance: hpcalc-iphone - Google Code. It's a GPL project, includes the beloved HP-15C. I'll want this one, it will be out with the official SDK.

WD Hard drive - what's the warranty?

I go through hard drives ... like clockwork. Every six months the six work and home machines I maintain need a new hard drive.

None of the brands seem better than any other. I try to pay about $150 - $200 since that seems to the price of reasonably current drives with most of the bugs worked out.

The only other thing I look for is the warranty. Not that I ever bother to use them, but I figure a longer warranty means the manufacturer has a bit more confidence in the product.

Today I had to replace a drive quickly, so I didn't look for a warranty. When installed it, I couldn't find any reference to a warranty period.

I found something on Western Digital's web site when I entered the serial number:

Warranty Check for End User

By default, the warranty date is calculated from the manufacture date. However, if you have proof of purchase, we can update the warranty to calculate it from the purchase date. If you feel that the warranty date needs to be updated from the purchase date, please follow the instructions below.

1. Send us an email from this link:
   http://wdc.custhelp.com/cgi-bin/wdc.cfg/php/enduser/ask.php
2. Put in "Update warranty" in the subject line.
3. Attach a picture or a scanned copy of your purchase receipt (jpg, jpeg, or gif formats only, max 500KB)
4. Send the email.

The date I was given was 10/2008, or about seven months from now. So I'm guessing the US warranty is one year, but if you don't have a receipt you can still make a claim for a shortened period.

Odd development.

PS. I am getting tired of restoring from backup. I'm planning to at least put NAS storage at home, with a RAID array so I can just swap dead drives without having to do a restore.

Aperture and Time Machine: a joint bug

Apple suggests disabling Time Machine when Aperture is running  :

Time Machine may back up the entire Aperture Library on each run

As a workaround, turn Time Machine off (in Time Machine preferences, in System Preferences). When you want to manually back up, quit Aperture, then choose Back Up Now from Time Machine's menu bar icon (Mac OS X 10.5.2 or later), or by Control-clicking or right-clicking Time Machine in the Dock and choosing Back Up Now.

Translation: We have a nasty bug.

Aperture 2.1 is out -- and the installer is broken ...

I got hit by this bug:

Apple - Support - Discussions - Aperture 2.0.1 doesn't recognize that ...

Aperture 2.0 or later is required to install this update.

The Aperture 2.1 48MB updater doesn't recognize that Aperture 2.01 is installed. I tried various hacks that have worked for others: removing pref files, removing receipts from all prior Aperture installs, removing the Aperture ID file from Pro Apps, deleting Aperture, emptying trash, reinstalling from DVD using an Admin account, etc.

No joy.

Not coincidentally, when I run Software Update I'm not notified of the 2.1 update. On my G5 iMac running 10.4.11 and all security updates there's something broken in the Aperture registration department.

I was able to install the 184MB Aperture 2.1 trial version. It recognized my registration key and it includes the "dodge and burn plugin"

Even after successfully installing 2.1 trial however, the updater still does not work.

Very annoying.

BTW, the sample projects that are installed from the DVD occupy 3GB of disk space. They're stored in Library\Application Support\Aperture.

What Time Machine can and can't do

via DF, I came across a great review of OS X Time Machine backup software. For example:

X.5 Time Machine (Quarter Life Crisis)

...A point that is rather sore as well for me is the fact that Caches folders are completely excluded from backups. By Spotlight’s poor design the Caches folder contains a Metadata folder which applications have to store their data in if they want it to be found in the index. Those metadata files will not be in a backup. As a consequence restoring your system from a backup will leave you with an incomplete Spotlight index until you run all the applications which stored data in the Caches/Metadata folder and make sure they re-create them. Ultimately this is a Spotlight issue, I think, but with Time Machine being made by the same company, they should have had an eye on it...

Quarter Life has an extensive list of critiques: the UI is truly awful, there's no encryption of the backup (!), there's very limited control of what's backed up, etc.

Firewire, USB and SATA performance on Macs

I have an old 3G Firewire iPod and a several USB iPods. The old Firewire iPod is a joy to sync. Extremely fast, instant dismount. The modern USB devices are a pain. My Mac Firewire connected drive feels as fast as an internal drive, my PC USB drives are sluggish.

Gigabit ethernet connected drives, in my experience with my CPUs in heavy use, are much faster than local USB drives and even comparable to local Firewire 400 connections.

This, of course, contradicts the theoretical performance figures for USB drives. It makes me mourn the lost beauty of Firewire 400.

So I really enjoyed this discussion of Firewire, USB and SATA on Macs, with Windows specific footnotes.

AppleInsider | Exploring Time Capsule: theoretical speed vs practical throughput

USB has a faster theoretical maximum than Firewire 400 (400 Mbits/sec; 50 MB/sec), but Firewire 400 is actually much faster than USB because it uses smarter peer to peer interface hardware rather than pushing low level work onto the PC host's CPU as the simpler master to slave architecture of USB does.

On a Mac, Firewire is typically around twice as fast in real world transfer rates, with USB hitting around 18 MB/sec and Firewire reaching 35 MB/sec throughput. Windows' implementation of USB has historically been faster than Mac OS X's, with Windows' USB reaching throughput closer to 33MB/sec..

Firewire on a Mac is far faster than USB on an XP box, I suspect Firewire on XP is comparable to Firewire in OS X and also much faster than USB in real world use (ie. when the CPU is loaded down with other tasks, and so unable to respond to USB demands).

Given my experience I'd expect a USB drive on an gigabit wired Airport base station to be significantly slower than a SATA drive inside a wired Time Machine.  I hope that will be tested in a future post.

The case of the vanishing Outlook Navigation Pane Shortcuts - and how to backup and restore.

On Friday March 21st my customized Outlook 2003 Navigation Pane Shortcuts were fine. On Monday March 24th I they'd vanished.

This has happened before. It's bad, because I really depend on these things.

I don't know why they vanished, though I suspect an update of some sort triggered an old bug. These things are known to be fragile in Outlook 2003. I resolved to come up with a way to restore and back these up, and with a bit of Usenet help I did manage that.

Here's what worked for me.

1. Find out where Outlook stores this data: Microsoft kb: Outlook file locations [1] mostly answered that question. It says these customization are stored in: "drive:\Documents and Settings\<user>\Application Data\Microsoft\Outlook\Outlook.xml". There's an error in the kb though, the file is not named "outlook.xml". When Exchange Server is in the picture the file is named by the Exchange/Outlook profile. In my case that's "Default Outlook Profile.xml".
   
2. I used our corporate backup software to find the most recent backup prior to when the truncation occurred.
   
3. I saved the last 19KB file to a local folder and used a text editor to extract the portion with the tags: <userdefined> from the backup. Then I exited Outlook and copied it into the current version of this file.
   
4. I then launched Outlook and added ONE new Shortcut to a store in my PST File. If you don't do this you get a "messaging error" when you try to use the old Shortcuts that point into local PST files. My guess is adding a new one forces Outlook to update some binary cache file it won't otherwise update.

I still don't know why my Navigation Pane Shortcuts vanished. They may have been a casualty of something that damaged another part of the "Default Outlook Profile.xml" file. I'll see if it happens again, but from now on I have a backup and an easy way to restore them when they get wiped out.

-- footnotes --

[1] It's astounding how long and complex this list is. Outlook is a very old and rickety piece of software. Slipstick also has a good reference list.

[2] Example:

<userdefined>
        <linkgroup name="PIM">
            <wdLnk>
                <ltype>shortcut</ltype>
                <reckey>2B1967ED1BF9A2489BE8AFD510569E6E</reckey>
                <eid>0000000038......000000000</eid>
                <rootfold>1</rootfold>
                <name>Outlook Today</name>
                <storeid>1</storeid>
                <urlhint></urlhint>
                <clsid>0078060000000000C000000000000046</clsid>
                <icondata></icondata>
            </wdLnk>
            <wdLnk>
                <ltype>shortcut</ltype>
                <reckey>000....E750000</reckey>
                <eid>000000002B1967ED1BF9A2489BE8AFD510569E6E0100BB4E71E8D95434469F02C78FF25B3D85000000981E750000</eid>
                <name>Calendar</name>
                <storeid>1</storeid>
                <clsid>0278060000000000C000000000000046</clsid>
                <icondata></icondata>
            </wdLnk>
....
</userdefined>

FLIP Video Ultra camcorder: iMovie HD works, iMovie '08 might

[Update - Over the course of using and testing this device I became very negative about it for all OS X users and even for XP users. I don't believe the video files will be readable in five years. On the other hand, as per my 7/08 update, on one machine iMovie '08 now edits this video.]

I bought a Flip Video Ultra Series Camcorder for the kids to use. I've jotted down some initial impressions below.

1. It feels flimsier and cheaper than the original model a friend of mine has. I'm surprised by this. I'd recommend buying the older model, which is now sold at a discount.
2. There's no CD in the box, the software is delivered in the camera's memory store. Cute, and it works.
3. Videos are in "AVI" format -- that's a metadata wrapper around a codec. In this case coded is 3ivx MPEG-4. iMovie HD (old version) can import these files after the 3ivx codec is installed, but iMovie '08 (current version) cannot (more below).
4. Amazon had Jan 2007 firmware update. I downloaded and installed it but my camera was up to date anyway.
5. The bundled software for OS X video editing is universally derided. I did not install it.

I installed the 3ivx MPEG-4 decoder from FLIPVIDEO:System:INSTALL:Macintosh 3ivx from a non-admin account. It used a standard Apple installer, no restart required. Install is into a Applications:3ivx (shared applications) and it includes an uninstaller* and a link to 3ivx.com [1]

Videos are found in FLIPVIDEO:DCIM:100VIDEO.

Since iMovie '08 only imports DV, MPEG-4 and .mov files I used Mike Ash's QTAmateur to translate the AVI files to DV files and imported these into iMovie. QuickTime Pro would also work but costs $30 -- and Apple makes users repurchase a license with every significant update.

The need to translate prior to import into iMovie '08 is very irritating. I'm working on something better, or I could just use iMovie HD. I've left feedback for Apple requesting support in iMovie '08 for importing AVI files when codecs (such as 3ivx) are installed. I'll also contact some Mac gurus I know.

I'll have more on the camera later once I figure out a good import workflow. Apple definitely deserves multiple hard blows for the lack of file import support in Movie '08 vs. iMovie HD.
--
[1] 3ivx is hoping you'll buy the full version from them. The current version is a small fix ahead of what Flip Video is shipping:

3ivx Technologies Pty. Ltd., the MPEG-4 Video & Audio specialist, announces the immediate availability of version 5.0.2 of the 3ivx MPEG-4 compression suite for Windows, Mac OS X & Linux....Version 5.0.2 fortifies the 3ivx MPEG-4 Filter suite against software exploitation from maliciously crafted video files.... improved Vista compatibility and additional QuickTime 7.3 compatibility...

The 3ivx site has some interesting configuration information, but the decode page for OS X is blank. The 3ivx site references iMovie HD (they call it simply "iMovie"), but not the problematic iMovie '08.

[2] Apple's iMovie was probably cutting into Final Cut Express sales, so Apple tossed it out and gave us 'iClip' instead.

Update 3/23/2008: I started a thread on Apple's Discussion list . It's gotten at least one helpful response that suggested MPEG Streamclip.

Update 3/23/2008: If you don't have it, you can download iMovie HD from Apple. I don't know if it really checks for iMovie '08, most everyone with a Mac will have one or the other though.

Update 3/30/2008: I found an AppleScript that should do what I need. It tells QuickTime Player to open multiple AVI files, process them to DV stream, then delete the originals. Unfortunately, it's giving me cryptic error messages.

Update 3/31/2008: I'm beginning to think Apple quietly disabled AppleScript driven conversion in QT Player. Nobody seems to know anything about it.

MPEG Streamclip is a free video converter and editor that will work with the 3ivx encoded AVI files FLIP Video produces. QuickTime Pro ($30) won't do batch conversion, but it will directly edit the AVI files -- and it may be much simpler to use than iMovie. Perian doesn't help with conversion, but some feel it's a better solution than the 3ivx supplied codecs. VisualHub does batch conversion and is well regarded, but it costs $23 and the "trial version" is very limited. Mike Ash's QTAmateur also does batch conversion and it's free.

I considered the possibility of a script solution that would automate the flow into iMovie, but neither iMovie '08 or iMovie HD are scriptable [1]. So the best one could do would be to flow into iPhoto and collect from there.

I like the idea of a simple editing solution the kids can use on their video clips, without a translation step. iMovie HD doesn't require the translation step, but it's too much for now. I think I'll try MPEG Streamclip first, then, if that doesn't work, I'll spring for QuickTime Pro. If that fails then I'll fall back to iMovie HD. I'm going to leave iMovie '08 out of the picture for now, it just aggravates me.

Update 3/31/2008b: MPEG Streamclip

I'm back from my MPEG Streamclip. It's very impressive as a free app, but it's pretty rough. I came across a lot of odd behaviors, and it's a very industrial interface. There was no budget for eye candy! I came up with some interesting numbers in testing a small video:

• initial size (AVI file, 3ivx coded): 12 MB
• convert to DV using QT Amateur or iMovie HD import: 84MB file, exquisitely slow.
• MPEG Streamclip: painless quick trimming, but can't save as AVI (no encoder). Oddly enough, none of the 'Save As' options worked. Export worked though ...

After I trimmed the small video to a 10 second clip I tried exporting from MPEG Streamclip to various formats:

• Conversion to mp4 H.264: small clip took minutes (2.5MB)
• Conversion to QuickTime Motion JPEG A: very fast but 17MB file
• Conversion to Apple MPEG 4: very fast and the file is only 2.2 MB (quality probably less than H264)

iMovie '08 was able to import the H.264, Motion JPEG and Apple MPEG 4 files.

If it was just me I'd go with MPEG Streamclip, do initial trimming with it, export as Apple MPEG4 then drop the clip in iMovie. I think Streamclip is too rough for the kids though. So on to QuickTime Pro.

Update 3/31/08c: QuickTime Pro

Well, we have a winner. QuickTime Pro is a very nice tool for trimming video then saving it.

Just as with Streamclip, I experimented with various video formats. QT Pro will open the original AVI file and save it as an MOV file, but that doesn't help. It's simply changed the wrappers.

I tried exporting a QuickTime movie using H.264 compression, but it was unbearably slow on my G5 iMac. MPEG-4 Improved had very tolerable speed and image size and quality, but it was not compatible with iMovie '08. Until I turned "streaming" off, then it worked.

So I think I'll teach the kids to use this workflow:

1. Copy videos to a shared folder.
2. Trim and save using QuickTime Pro.
3. If they want to assemble the clips into a movie we can use iMovie '08 or iMovie HD.

Update 4/30/2008: My Amazon review - not for OS X customers.

Update 6/5/08: Flip's owners, rebranded as Pure Digital, have finally acknowledged there's a problem with iMovie. In the meantime, Apple's QuickTime 7.4.5 broke the 3ivx codec used by the Flip camera on OS X 10.4 (but not 10.5). My costly version of QuickTime Pro is now of no use. Neither Apple nor PureDigital has acknowledged the problem, but the Apple and 3ivx support forums have lots of complaints.

Update 7/3/08: The FLIP 3ivx videos are now directly importable and editable in iMovie '08 7.1.2 with the default FLIP provided 3ivx codec, OS 10.5.4, Intel processor, and QuickTime 7.5.

I've been using Perian on my PPC iMac since QT 7.4.5 broke 3ivx audio

With that codec and QT 7.5 & iMovie 7.1.2 & 10.4.11 FLIP video import attempts don't produce any error message, but they don't work. I need to reinstall the 3ivx codec and try again.

Update 12/12/08: Apple has even provided official documentation on import from new FLIP camcorders.

--

[1] Scriptability is a great way to test when Apple software is polished and ready for use. Apple commonly releases poor quality products then tunes them up over many iterations (ex: iPhoto). Applescript support seems to require a mature product with a healthy code base and good design, so the lack of Applescript support in iMovie HD, iMovie '08 and (really) Aperture 2.1 are all bad signs.