Obscure OS X Tiger upgrade bug: runaway process related to automated file defragmentation?

MacInTouch Home Page

This may occur specifically with upgrades from older systems that may date back to 10.1. Some OS crashes may also trigger it. Sounds complex:

Several people followed up on a strange problem with "runaway" processes in Mac OS X:

[MacInTouch Reader] I had the same problem with the unkillable "update" process regularly hijacking the entire processor on my iMac G3 400MHz after upgrading to Tiger over a Panther installation. It would occur every 12 hours or so and could only be stopped with a forced reboot. This discussion at Apple put me on the right track to solving the problem. Specifically, checking the system log in Console revealed repeated errors such as:

"hotfiles_evict: err 28 relocating file 27611"

suggesting that the system was getting "stuck" while relocating a certain file. The discusion also suggested using hfsdebug to identify the file and delete it manually.

Unfortunately on my iMac, hfsdebug was unable to identify the file while the "update" process was running, so I had to record the file number from the error message, then run hfsdebug in the Terminal after force-rebooting the iMac. I was then able to locate and manually trash the offending file in the Finder.

This went on for a few days, with the "update" process reappearing every 12 hours or so, getting stuck on a new file. I noticed that the stuck file identification numbers were increasing (about a dozen files between 14931 and 112976), and that all of the files were quite old, created around 2001 or 2002. (I had not performed a clean install since the Public Beta days.)

Well after a few days of catching the "update" process in the act and manually deleting the problem files, I can happily report that the problem has disappeared and my ageing iMac is good as new!

["Xratchy"] The "update" problem has nothing to do with "System Update". The update program will have added lines to your system.log. Check them out for

hotfiles_evict: err 28 relocating file 20

The meaning for the error number can be found in

/System/Library/Frameworks/Kernel.framework/Versions/A/Headers/sys/errno.h
#define ENOSPC 28
/* No space left on device */

It's a bug in the kernel in the disk optimizing (defragment-hotfiles) code, where the opitomizing process doesn't find a free contiguous-block of disk space.

It's triggered in this case when there is no more continuous free block available, where the process needs/expect one, then it will just retry the search and brings your system to a halt.
The optimizing process is also triggered by opening a file, not only on reading/writing.

Try to make sure to have 10% disk space free at all times (if it's error 28). (Virtual memory can eat a lot of disk space.) Check your disk for errors after the crash (with your Tiger disk - Panther doesn't find/fix them all).

OS X esoterica: roots in NeXTStep and the Installer

The following as a comment in a discussion thread on the location of OS X applications. In general OS X applications, once installed, should not be relocated. This is similar to Windows, but in addition many OS X applications have very rigid requirements for installation paths. This is more restrictive than Windows.

It's also very strange for MacOS veterans. The original MacOS maintained a superb level of indirection between an applications physical storage and its unique indentifier. This level of indirection was broken with OS X, a result of the UNIX roots of NeXTStep and thus of OS X.

Alex Blewitt describes this in some detail. For OS X geeks his discussion is quite revealing. Emphases mine. Note the nasty bug -- in general I avoid symlinks.

# Alex Blewitt wrote on May 25th, 2005 at 1:09 am:

The applications can’t easily be moved and then referenced with a symbolic link. It might stand a better chance with an HFS Alias, but that’s no good if it’s on a different partition, since HFS Aliases only work on the same HFS partition (IIRC).

The problem is that the Mac OS X Installer program (which dates back to NeXT days, by the way, not Mac OS proper) packages its applications up in a compressed format called PAX; which is rather like the Unix TAR but just different. Unfortunately, the un-PAXer engine (pax -r) will overwrite symlink’d directories and replace them with real directories instead. This has been a known bug since time began; it may have been fixed in 10.4 but I’m not expecting it to have been.

Although most apps can be moved, there are a few ‘gotchas’. Unlike comment 3 indicates, most apps don’t write into /bin or /usr/bin (the only ones really being stuff like X.app and XCode.app which include command-line-compilery things). However, they can install private frameworks into /System/Library/PrivateFrameworks/ (which is a kind of Mac OS X dynamic link library, if you’re interested). The iLife apps are an example of this, as are the Apple Pro tools (which use Pro Application Support, or some arrangement of those words). [jf: these framworks have resources which remind me of old MacOS apps]

Since the app in a few cases is a shell to the underlying framework (e.g. Safari is a set of widgets around the WebCore framework, QuicktimePlayer is a set of widgets around the Quicktime framework etc.) it’s not safe to just move the app elsewhere, as it will rely on the framework versions too. This is one reason why they are all bodged to install only on / [jf: / is root] in case you’re interested; they have to write the app, and the framework in the OS partition there as well [jf: the path to the framework is a relative path]. For example, you can’t (easily) set up iPhoto.app on a networked share, because you also need the framework set up as well. (However, you can set the framework up on a networked share too …)

It’s also worth noting that ‘Applications’ is the well-known name for putting apps in, which is why if you create a user account and put a directory called ‘Applications’ in, it turns into the specific icon for an Applications directory. Ditto for Library, and Mac OS X will go through a search path for finding apps:
~/Applications /Applications /Network/Applications (/Developer/Applications)
as well as libraries:
~/Library /Library /System/Library /Network/Library (/Developer/Library ?)

From this, it’s relatively easy to create a share and mount a directory into the /Network/Applications and /Network/Library which are part of the known search path. Whilst it doesn’t matter too much about the applications, it’s really handy having the frameworks available in /Network/Library as well. To achieve this, you’ll need a server capable of hosting shares (I use NFS; I’ve found AFP to be problematic at best of times, but it’s getting better) and then mount them into /Network/Applications and /Network/Library.

For example, if you have a FireWire drive that you want to put stuff in, stick them in /Volumes/MyBigDrive/Applications (and ../Libraries as well). Then, set up an NFS export (see Mac OS X hints for how :-) for /Volumes/MyBigDrive/Applications and then re-mount it as /Network/Applications; ditto for Libraries. You should then see them in the /Network tab in your drawer.

When updates come out, you can install them onto a different ‘target’, although (to my knowledge) not via the Installer app itself. Download the package (Download and Keep from Software Update), then run it with ‘installer -target /Volumes/MyBigDrive -pkg SecUpd_1023.pkg’. (If you want to get rid of excess languages, you can also add -lang en to only install English)

A few of the apps (mostly the iLife ones) will attempt to complain if the drive isn’t / — but that can be fixed. There’s a file called something like preinstall in the SecUpd_1023.pkg/Contents/Resources directory; there’s usually something like ‘if [ $drive = ‘/’ ]; echo “You must install this on /”; exit 1′ in there. Removing this line, or getting rid of the exit will allow you to install it anywhere you want.

The other approach (and the one that Apple wants people to do) is to image a virgin Mac with the disk image utility, and then make the image available via NetBoot. That way, all macs have exactly the same image at all times; when an update comes in, you can test it on your ‘virgin’ mac, and then push the image everywhere. I’m not sure if these imaging tools are only part of Mac OS X Server, but given that they are just DiskUtility type ‘make an image of this hard drive’ it’s not entirely impossible to do it outside of a Server environment.

Of course, I’ve not had time to go into the /Library/Receipts folder for what you’ve installed where, but you might want to move the receipts into the shared /Network/Library/Receipts for the packages that you have installed on the shared drive in the past…

The Robservatory � My favorite browser

The Robservatory � My favorite browser

iPod esoterica: manually putting iPod into Disk Mode

Putting iPod into Disk Mode

The page has directions for newer and older iPods. These are for the older non-click wheel iPods:

1. Before manually placing the iPod into Disk Mode you should verify that it has a charge, if not you need to either charge iPod before continuing or connect it to power. You can either connect it to a high-powered USB port, or plug the iPod Power Adapter into an electrical outlet and connect iPod to the power adapter.
2. Toggle the Hold switch on and off. (Set it to Hold, then turn it off again.)
3. Press and hold the Play/Pause and Menu buttons until the Apple/iPod logo appears, then release them. This resets iPod. When you reset iPod all your music and data files are saved, but some customized settings may be lost.
4. When the Apple logo appears, immediately press and hold the Previous and Next buttons until the Disk Mode screen appears. (For iPod Software 1.0 through 1.1, a FireWire logo appears on the screen.)
5. Disconnect iPod from the power adapter and connect it to your computer.

Definition of a blogger post: date and subjet

Note the structure of the 'persistent url' used by Blogger:

http://jfaughnan.blogspot.com/2005/05/derivatives-for-small-investor.html

It contains the year/month/date and text taken from the subject. In database terms, it is the date (YMD) and subject text that uniquely defines a posting. Change either and any links to the prior posting will break.

I would have much preferred a level of indirection with a unique ID for a blog posting. This design, if deliberate, says quite a bit about how the Blogger engineers think about blogs and blog postings. Heaven forbid one should have a typo in subject line! It can never be fixed, or the links to the posting will fail. Likewise, if one wishes to update a posting and draw attention to it, one must edit locally and not revise the date, then create a new posting pointing to it.

These are fundamental design decisions, but they will be invisible to most users. Much software is this way; the mechanics of identity (in this case blog identity -- foreign keys in other words) are the invisible 'deep themes' of the software.

All IDE ATA drives are not the same

[Update 6/5: The Maxtor has been fine after being relocated and reformatted. Then I received a note from Vantec, who've been serious about looking into this, that they'd bought a Seagate 200 and it worked quite well. That was the final clue. What could be affect two different drives in slightly different ways, get worse over time, cause drive corruption and seek errors, not affect the older lower performance drive, and be better in some settings than others?

Heat.

So I checked out the cooling fan on the Vantec case. It is immobile. With some compressed air I can 'kickstart' it, but it has basically seized up. The higher performance drivers either produced more heat or were less tolerate of overheating; without the fan they were cooked.

I need to do some more testing, including asking Vantec for a replacement fan, but my guess is that heat was indeed the culprit.

[Update 5/21: ... the Maxtor in the Vantec cartridge has been malfunctioning and now will not format. I wonder if the drive has been damaged! I've updated my table.]

If you think of standard ATA IDE drivers at all, you probably think they're pretty much an interchangeable commodity with some modest performance differences, some variations in rarely used diagnostic and configuration software, and unpredictable quality problems.

That's what I thought. I was wrong.

I tested 3 drives from three vendors (Seagate, Maxtor and Western Digital) in 4 different settings. All were fine in a routine IDE setup, but there were problems with a firewire enclosure and a removeable drive bay catridge.

	Maxtor Diamondmax	Seagate Barracuda	Western Digital WD 800
	200 GB	200 GB	80 GB
PC MB IDE controller	Yes	Yes	Yes
PC Paradise IDE controller	Yes [2]	Yes [2]	Yes [2]
OWC Firewire Drive w/ iBook	Partial [1]	Yes	not tested
Vantec EZ-Swap MRK 103F in PC	No	No	Yes

[1] Works as long as the iBook doesn't go to sleep.
[2] This Paradise IDE controller emulates a SCSI drive. It supports 200 GB drives and works well with everything - except Retrospect 6.0. Retrospect, a notoriously difficult pile of software, has to be forced to use "NT Passthrough" when writing to a drive attached to this controller. This secret preference is enabled with the magic keystroke: Ctrl-Alt-PP. No, I'm not making this up. With the WD 80 GB drive Retrospect would stop after at some magic limit (I forget what it was), with the Maxtor it wouldn't even start -- but it would lock up my XP machine with an unkillable process that demanded a hard reset. With "NT Passthrough" enabled this problem goes away. This bug may be fixed in Retrospect 6.5, though I suspect others take its place.

So, not all drives are the same -- by any means!

The Maxtor worked initially in the Vantec PC cartridge, but then failed and may be damaged. The Seagate never worked. The Seagate works much better than the Maxtor in the iBook attacked OWC external firewire case. The Western Digital works in the Vantec PC cartridge, I haven't tried it in the firewire drive (too much trouble for now).

Most surprising. The lessons I draw for now are:

1. Before purchasing an even slightly non-standard container for an ATA IDE drive check with the vendor what drives are known to work. If they say "all drives" then don't trust them. (SATA may be better).

2. Of the 3 vendors tested, only Wetern Digital MIGHT work in all configurations, but more testing is needed.

3. Not all bad behaviors come from one problem. In the course of managing this very aggravating situation I ran into three: the Seagate incompatiblity, a bent pin, and the Paradise/Retrospect incompatibility. And some people have to resort to television to explain increasing IQs!

Google accelerator bug: Webmail incompatible

Faughnan's Notes: Google Web Accelerator for XP and Firefox (ok, for IE too)

I was working on webmail access to a user account in my domain and the Horde and Squirrel web clients seemed both to be failing. Perhaps this is coincidence, but after disabling Google Accelerator they worked again.

Macintouch spotlight review

Spotlight (Tiger Review)

The best overview I've seen of Spotlight thus far. The bottom line -- it works well enough to be useful, but it has at least two more revisions to go before it will be a solid tool. I'd guess 10.4.4. (10.4.1 is out, 10.4.2 will focus on bug fixes, 10.4.3 will start to address functional issues).

Using Spotlight with application controlled data stores is the interesting question (Entourage, FileMaker, etc). This is much more ambitious than merely indexing documents. I dimly recall old issues of BYTE referring to a 1970s era OS that had a true database as its file store -- it could manage this trick.

O'Reilly: Build a Simple MP3 Player for Your Site

O'Reilly: Build a Simple MP3 Player for Your Site

Embed tag and JavaScript. I've seen bits of this in many places, but never in one article.

TAGBAG Dashboard Widget: spotlight metadata management

TAGBAG Dashboard Widget

...One of the proposals for getting things done via Dashboard and Spotlight is to tag your files via the built-in Spotlight-Comment option. This allows to set several tags for every important file you have, and later on you can search for these tags, or simply created searchable folders which consist out of all those files containing special tags...

Tagbag helps you to keep track of which tags you used. It always shows a list of all tags you have been using so far. This makes it easier for you to just use a small but sorted set of tags, without forgetting about which tags you used where. In addition to that it allows you to see a list of all files having a certain tag by simply clicking on the tags name...

Similar to the way labels work in Gmail.

Moving about images in iPhoto 5 and prior

ExtraBITS

"In versions of iPhoto before 5, when you zoomed in on a photo in edit mode, you could hold down the Command key to drag the image around; it was a lot easier than using the scrollbars to move around within the image. In iPhoto 5, though, Apple made Command-clicking while in edit mode set a white point for the image (it works only if you click on an appropriately white part of the image), and as far as I knew, eliminated entirely the Command-dragging approach to moving around while zoomed in. Kudos to Markus Silpala, then for discovering that you can in fact still drag a photo in edit mode to move around - the trick in iPhoto 5 is that you must hold down the Spacebar while doing so."

Disabling the obnoxious Symantec Norton Antivirus security alerts when auto-protect is off

MCSE World Forums - Symantec Security Alert - How do I stop them?

I don't like the 50% performance hit induced by NAV scanning. I'm inclined to take a chance on leaving it off much of the time (I have pretty excessive backup tendencies); I'll still run my nightly full system scan and the mail scans should still work.

Problem is, Norton produces an obnoxious alert every five minutes if one disables the auto-protect feature. Thank heavens, this works:

Take the path C:\Program Files\Common Files\Symantec Shared\Security Center\ to locate two files; symwsc.exe and symwscno.exe. Create a new folder in the same directory and place the two files in it rather than delete them (just in case...). You shouldn't get the alerts after this.

Thanks shingirai kanyemba.

Remove DRM from iTunes music: JHymm

JHymn Info and Help

If I've paid for music and I will be using it in a way consistent with fair use, I have no personal issues removing DRM protection. Here's how.

Apple Remote Desktop (ARD) client service is now installed by default in 10.4

macosxhints - Some Apple Remote Desktop client features

Nice tips for ARD use. So now we hear ARD is installed by default. We also hear about slates. Any chance Apple will do video iPod peripherals that also allow remote access to a Mac machine?

macosxhints - 10.4: How to rotate OS X's display

macosxhints - 10.4: How to rotate OS X's display

Includes fixes for when rotation disables a display! Read before trying.