XP: Recent lessons from the dark side

[see update about the two "versions" of WDS and a later update as the saga continued ...]

At home I sail the often calm waters of OS X. At work I fight the fury of the storm, trapped in the XP triangle.

Now, it must be admitted that if my Dell XP laptop were sentient I'd be condemned as a cruel master. I torture the darned thing. I know few who see as much of XP and Office's brittle nastiness as I. Maybe if I treated OS X the way I do XP it would break to. (Vista? You're joking, right?)

That said, a recent flurry of cascading messes taught me a new lesson and reminded me of old ones.

1. Sometime in the past few weeks my 75GB drive suddenly had only 9GB free. It's dangerous to fall below 20% free space on a heavily used XP or OS X system, and I think this was one of the "straws" that pushed my XP system from its usual metastable state to accelerating collapse. In retrospect I'd somehow ended up with a 5GB orphaned pagefile.sys. I couldn't see it, because I somehow had Explorer configured to not show system files [12]. I eliminated the orphaned pagefile.sys by accident [1], but I think if I'd had Explorer showing me system files I'd have seen it, and dealt with it. New lesson: always display the hidden files so you can track pagefile.sys.
2. A combination of Windows Desktop Search [5], an unstable corporate network with intermittent Exchange connectivity failure, a 3.5 GB Outlook PST file [11], severe disk fragmentation [2], bugs with Outlook 2003 [3], my use of Microsoft's Onfolio [4], my insanely persistent use of Palm synchronization [6], Microsoft's Live Meeting Outlook Add-in [7] finally led to a system meltdown with increasingly odd Outlook behavior and, finally, OST corruption.
3. Good news: no covert alerts of drive read/write errors in XP's monitoring tools (XP quietly tracks many disk failures without notifying even admin users) and no chdksk/scandisk/whatever-it's-called-now problems.

So now my OST file was corrupt. Happily, that's usually not a big deal and it wasn't this time either. I turned off WDS (I snoozed indexing but I think I should have disabled the indexing service using XP's service manager - it kept trying to return to life) and found my OS file in "C:\Documents and Settings\[user_id]\Local Settings\Application Data\Microsoft\Outlook". I renamed it and restarted Outlook, which then rebuilt my OST file from the Exchange Server. The new version was about 7MB smaller than the old one (103MB) but I seemed to have everything -- including some old tasks that suddenly reappeared from the twilight zone. I then ran my series of Outlook clean-up switches [9]

/cleandmrecords
Deletes the logging records saved when a manager or a delegate declines a meeting.
/cleanfreebusy
Clears and regenerates free/busy information. This switch can only be used when you are able to connect to your Microsoft Exchange server.
/cleanprofile
Removes invalid profile keys and recreates default registry keys where applicable.
/cleanreminders
Clears and regenerates reminders.
/resetfolders
Restores missing folders for the default delivery location.

DANGER (don't run this one unless you need it):
/resetnavpane
Clears and regenerates the Navigation Pane for the current profile. This will vaporize your Outlook Shortcut pane

When all was done I restarted WDS and had it rebuild its index from scratch, then I set my Palm sync to have Outlook overwrite the Palm.

Another fun lunchtime with Microsoft ...

------------- footnotes --------------------

[1]I removed my cache to free up enough space to run defrag, and when I restored a fixed 2GB cache the system asked if I wanted to delete an old pagefile.sys. Then my free space reappeared. A new 160GB high speed drive is on order.

[2] XP won't defrag when free space is less than 15%. OS X is much less prone to serious fragmentation.

[3] Still, it's much better than its predecessor.

[4] XP has the world's best blog writer, Windows Live Writer, but XP's corporate-friendly blog readers are very weak - and getting worse. Onfolio was the best, but Microsoft has left it to fester post acquisition. I fear it's becoming increasingly unsafe. OS X has the opposite problem -- lousy authoring tools, great readers. Of course OS X can also run WLW in a VM ...

[5] Really, I need all this stuff. But WDS is trying to index 4GB of Outlook and hundreds of thousands of system files.

[6] If you're not a Palm addict, I beg you, don't start. Life with Palm and Outlook/Exchange is like juggling antimatter, and it gets worse all the time.

[7] Ok, so this is another straw on that broken back. I am very suspicious of that plug-in and how it impacts Outlook/Exchange behavior with an unreliable network.

[8] This is typical of whenever I regenerate Outlook's OST file. Something old always reappears, it's never been important. Bugs.

[9] I do them one at a time exiting Outlook after each one.

[10] Outlook 2003's Shorcut pane is a "pain in the ***". So dumb, yet so essential in a complex Outlook configuration.

[11] This is why WDS is not an option.

[12] This is the default setting, but I always change it. Not this time apparently.

Update 9/28/07: Wow. Microsoft is in such bad shape. After all of the above I discovered Windows Desktop Search wasn't working properly -- in ways to diverse and complex to document here.

I found that Microsoft has two somewhat different products they call "Windows Desktop Search". If you Google on WDS you will find the product they aim at the corporate sector. Don't get that version (so called "3.1").

I ended up uninstalling WDS-Corporate-individual user 3.1 I installed the version you get with Windows Live toolbar..

1. Install Windows Live Toolbar
2. This should install the right version of WDS. If not use "toolbar options" and "install buttons" to find and download WDS...

The version number of the "good" WDS looks like an IP address ...

Update 10/23/07: The saga continued with some improvements as noted above, but then the flakiness returned. In particular Outlook would exit with a hung process, and I'd lose network access. My employer has larded up my system with various inventory services, but, mercifully, does let me disable services. I kept working through the list, disabling various XP services but with little impact. I reviewed my list installed software, and uninstalled various apps I don't use. That' when I noticed a very suspicious Yahoo updater (don't you hate that every damned app has its own update infrastructure?) -- removing that did seem to help my startup time!

The error frequency grew, and it became apparent there was a hardware component to the debacle -- one that wasn't showing up in XP's event logs.

I hate hardware failures -- especially those the OS can't detect.

I replaced the hard drive on the general principle that drive failures are common. If that doesn't work it's time to test memory and even the video memory.

Phone won't charge? Palm won't sync? Scrape the connectors

Eons ago, when the legends coded in binary, Jerry Pournelle was forever urging his BYTE column readers to spray something on electrical contacts. It kept some kind of demonic influence at bay.

I thought of him when I recently fixed my wife's Samsung i500 and my battered Palm Tungsten E2*. Her phone wouldn't charge, my Palm wouldn't sync. In both cases scraping the metal contacts with a pin cured 'em. Crud was blocking the connection.

Now that's satisfying ...

* I know people complain about Apple's LiOn battery life, but the Palm battery died when that sucker was about 8 months old!

iPod Diagnostic Mode and failure code 702

This is why Apple is pushing so hard for flash memory. Cars in particular are murder on hard drives, and we all use our iPods in vehicles.

My iPod was starting to skip a ridiculous number of songs. It would play a a minute or two then jump to the next song. It's been years since I ran diagnostics on an iPod, but the methodShop iPod Diagnostic Mode was great. Apple's help page on "song skipping" was worthless. Once you reboot (select/menu hold) and enter diagnostics (select/rewind) with the device plugged in you use the menu and previous buttons (M25 Diagnostics 0.7) to select either auto or manual diagnostics.

I ran the auto diagnostics and got a red screen (M25 Diagnostics 0.7) with Failure Code 702, 25 fails on the automatic diagnostics. I suppose one might call this the ipod "red screen of death".

Now, one would assume Apple might help with this, but the Google Search "ipod failure code 702 site:apple.com" fails. So does a search on Apple's "all documents" support site. it's not hard to find a reference on Apple's Discussions though -- so they're not censoring it completely.

I then did the manual test, HardDrive is under the initial IO menu (menus seem to vary with the newer devices). Options are HDSpecs and HDSMARTData. HDSMARTData showed no errors however! Specs, interestingly, told me my hard drive temperature, but nothing useful. The "NTF" option has more tests, but none mention the hard drive.

I'll try a "restore" tonight and retest, and if that fails I'll check my credit card's extended warrantee policy. The next option will be to look at some of the commercial drives, or see what Apple's built-in Disk Utility offers. I don't see a lot of net discussions on this topic, so it might in fact be a relatively rare problem. Just my luck ...

Update: Looks like there's a semi-radical approach I can try

• mount iPod in disk mode and run disk utility to repair
  
• do a factory reset/restore via iTunes
• test with both OS X Disk Utility and the built-in diagnostics
• if it still fails, reformat as HFS+ via Disk Utility and follow these restore directions.
• if it still fails at that point it's probably toast.
  

Update: No, it's not likely the hard drive. The drive passed every test I threw at it, including OS X Disk Utility. I even zeroed out the entire drive, reinitialized, followed the restore directions, etc. I always get the same red screen and error message:

M25 Diagnostics 0.7
Completed
Failure code: 702 with 26 fails

I suspect it's the SDRAM, not the drive.

Update 11/9/07: It's been a few weeks since my original post, but nothing has turned up anywhere to explain this error code. On the other hand, my iPod hasn't been skipping at all, even though the test results never changed.

My best guess is that this is a count of bad sectors on the disk, but that the OS normally masks out the bad sectors. So they represent some lost capacity, but when I restored my data to the drive it went to the good sectors.

It's weird that this is genuinely undocumented, it's the sort of geeky thing that one expects to find an easy answer too.

I did receive a comment about audio problems with a 'red screen of death' test. Mine is not having those problems, it seems fine now.

Update 5/12/2009: No further problems occurred, so my above fix worked for this particular 702 problem.

Interesting comment on scanning color negatives

Color negatives are hard to scan. This is rarely mentioned ...

Scanner Review: Microtek ScanMaker i900

... Converting color negatives to positive is a black art. We discussed it at length in two Advanced articles, explaining what the orange mask does and the proper way to account for it. Fortunately, Microtek's two software solutions both include advanced negative conversion modules that, if nothing else, are excellent starting points. SilverFast Ai includes NegaFix and ScanWizard Pro has a similar feature, as does VueScan...

In my experience, given limited time, you may get better seeming results from a high quality print than from a negative -- especially if the negative is old. Color is funny.

Epson vs. Canon scanners: who has better OS X support

I'm thinking of getting a flatbed scanner that will do at least 12 35 mm negatives in one batch job. I want to pay under $600. It seems my choices are Epson and Canon.

I could check review sites, but they're almost always worthless. There's another way to make a choice. Which driver software is most compatible with OS X? The first place to check is the relevant support sites.

Epson has some dated material on their site, but they at least have a page outlining scanner support that was current as of 10.4.4. I liked the long list of scanners that work out of the box with OS X Image Capture. The current V700 PHOTO has universal drivers tested through 10.4.9 and is directly supported by VueScan without drivers.

Not bad. Now lets look at Canon. I've previously written about the horror of a CanoScan install, and Canon's printer drivers are notoriously ill mannered. (HP? You're joking, right?)

So they're off to a bad start, but let's try Canon's 8800F page. They have software, but nothing about Intel, Universal, OS versions, etc. Not good. Vuescan won't work with the 8800F unless the Canon drivers are installed. Not good. OS X support page? Minimal.

Gee, that wasn't so hard. I didn't have to look at a single product review.

BTW, if you want a review, this is the best I found for the V700. Scanners aren't changing very much, so I'm comfortable buying at the high end. The Nikon slide scanner I bought 3-4 years ago is still current today.

Update 9/24/07: Product Recommendations from Ed (vuescan) Hamrick (emphases mine):

Best 35mm film scanners: Nikon CoolScan (all models) - good color, good quality, fast

Best low-end flatbed scanners: Canon LiDE 20/25/30 - small, inexpensive, get power from USB [jf: but horrid OS X drivers]

Best high-end flatbed scanners: Epson Perfection 4990/V700/V750 - fast, good quality

Best A3 document scanners: Epson GT-15000 and GT-30000 - reliable, good quality

Best raw file software: Adobe LightRoom - reads VueScan's Raw DNG files (Apple's Aperture doesn't)

Another point for the V700. So Canon is out completely, but Epson isn't completely unchallenged, because now I'm adding the Microtek i900. BTW, I found this bit of the review very useful:

If you scan a 35mm film frame at that resolution, your maximum enlargement for a 300-dpi dye sub printer is 4x6. To get an 8x10, you have to be able to scan 2400 dpi...

That's a nice reference to ahe at hand. To do a 35mm scan comparable to a modern dSLR the resolution would have to be at least 10,000 dpi, which probably exposes the limits of film.

Update 9/24/07: I was very tempted by the Microtek i900, but it turns out that they have a serious customer service problem ... They sound like a pretty small company. So if the device has no problems it might be the best scanner on the market for its price range, but if anything goes wrong you're out of luck. (I do love Amazon's reviews, esp. the 1 star reviews ...)

... Microtek customer "service" was anything but helpful. In fact, they have ONE technical support person on staff--I know because I spoke to him several times.

...I contacted the service department for Microtek who send you to an online repair service that will give you an email response in 48 hours. Turns out that the unit that I paid over $500 for is out of warranty and there is NO repair service for Microtek in the United States!

... I almost bought the i900, but wound up buying the i800 instead. You can read my full review under the i800 page, but basically I've had a pretty terrible experience with this company. My scanner suddenly broke after a couple of months, and the only way to contact their tech support is to wait until they email you back which seems like it takes anywhere from three days to two weeks. My replacement scanner had very dirty glass on the inside surface...

Update 1/6/08: The Epson V700V750 is still their flagship product. This technology doesn't change much any more! An excellent UK MacUser review summarizes strengths and weaknesses. After market negative holders might be indictated.

Clarkvision: Digital Workflow Summarized

I found ClarkVision via Kotke, I've posted about both on Gordon's Notes recently. This ClarkVision description of workflow is noteworth because the author is an uber-Geek (MIT PhD planetary physics), an astronomer, and a super-serious photographer. The combination yields an almost unequalled knowledge of digital photography details.

I also like it because, it's consistent in many ways with my far more modest experience. I do a lot of these things, but I sacrifice quality for speed and portability. Also, use Aperture image editing, not Photoshop.

I don't agree with his approach to metadata (file names, nothing more), but I sympathize with the goal of avoiding lock-in. I'm making a calculated guess that the mass of photo geeks will provide a solution to save my metadata; until now iPhoto and Aperture both support writing applications that could extract most metada (if Aperture were able to export XML/XMP sidecars I'd be almost sanguine ... but, ominously, it doesn't).

Here are excerpts, the article is a great read ... (emphases mine with some inline comments]

Clarkvision: Digital Workflow

... I scan all my film at the full precision of the scanner. Good scanners are at least 12-bits/channel. The output file is 16-bits per channel TIFF images. I do the minimum processing at the time of the scan and save corrections for the photo editor where I have more control. Scan parameters include:

* Straight-line transfer curve.
* Brightness correction only.
* Little to no color correction...

... Digital Cameras. I do both jpeg and raw format output. Jpeg is only 8-bit, while raw on many cameras is 12 bit and a few now have 14-bit output. Raw files are converted to 16-bits/channel TIFF files... [jg. I use RAW in Aperture, no TIFF conversion.]

... In the photo editor

* 1) I only do 16-bit editing. If the starting file is 8-bit (e.g. jpeg), the first step is to convert to 16-bits/channel. [jg. If you work with RAW in Aperture this isn't an issue ...]

o Why? with integer math, there is always round-off error of 1 bit.... If you do multiple editing steps, added errors can result in poor intensity precision with an 8-bit file. This is called posterization. 16-bit editing provides enough precision so that posterization is not a problem....

* 2) Color Space. Check the color space and convert to a wide color space if not already there. I generally use Adobe RGB 1998.

* 3) Adjust Levels. First adjust the brightest portions of the image to your liking using the levels adjustment tool. For example, in a landscape image I might examine the brightest areas of the images, like clouds and adjust the levels adjustment slider so that the brightest parts just reach a value of 254 or 255 on the 8-bit scale typical of the slider tool. ... This levels adjustment is done on the entire image.

* 4) Curves Adjustment. The next step is curves adjustment using the curves tool. Never use Photoshop's contrast and brightness adjustment tools as they are additive. The curves tool, like the levels tool is a multiplicative tool. Multiplicative has the correct math to mimic changes in scene brightness, exposure, or f/stop changes. In the levels tool, the upper slider is used to derive the multiplier, and lower slider is an offset (a subtractive adjustment), and the middle slider changes the multiplier to a 2-part piecewise line multiplier.

* 5) Dodge and Burn selected regions. Select different regions as desired that may be at the limits of dynamic range and dodge and burn to bring them into printable range. For example, bring up shadow detail, or darken clouds. First select the area with one of the selection tools. Next feather it. Feathering makes the selection area a smooth transition to the rest of the image. Once feathered, adjust the level with one of the three following tools:
o Curves tool,
o Levels tool, or
o Shadow highlight tool.

* 6) Increase Image Size. If I'll be making larger prints, I'll interpolate the image to a higher pixel count. I usually use bi-cubic or bicubic smoother interpolation in Photoshop. [jg: this surprises me. It seems a bit magical, but this guy knows his stuff. This one item suggest how sophisticated modern Photoshop is ...]

* 7) "Sharpen." The main tool many use is called unsharp mask, a filter. But unsharp mask does not actually sharpen! The way unsharp mask works is that the image is blurred using a certain radius, and then the original image is differenced with the blurred image, and the result is added back to the original. The effect is to modify contrast around edges. The amount added back is usually controlled by the user (called amount in Photoshop). The effect is actually a change in acutance, not sharpness. But increased acutance gives the appearance of increased sharpness.

However, there are other methods that actually do sharpen. I use Richardson-Lucy image restoration. Photoshop currently does not have this tool.. [jg: Aperture has two sharpening tools, I use Edge Sharpen but I don't know how it works ...]

* 10) Save the file. Depending on my intended use, I save as a 16-bit/channel TIFF files, or convert to 8-bits/channel and save as jpeg. For best quality, 16-tiff files are necessary. I only save as jpeg at highest quality and only when I do not need highest quality. [jg. I never need his quality, so I always save as JPEG, highest quality, for maximal longevity. I work with RAW in Aperture and toss 'em when I'm done. (The Horror!]...

Flash picture with eyes open? Divide by 3

I think this rule applies to flash photography, but the idea is universally applicable. Flash is not long for this world ... fortunately.

Rule of thumb to avoid photographing people with their eyes... (kottke.org)

... to avoid photographing people with their eyes closed: divide the number of people by three (or by two if the light is bad). ... Jeff writes: 'Way back when we only used film I learned you could tell before looking at the photo whether someone blinked by asking them what color was the flash. If it was white or bluish white, then their eyes were open. If it was orange, then their eyes were closed and they had 'seen' the flash through their eyelids.'