Thursday, September 05, 2013

OS X Mountain Lion cannot delete sparsebundles containing over 262,144 bands (2TB+)

Working on my primary Mac reminds me of working on cheap old plumbing. I'd start out fixing one piece, then another would break, then I'd fix that, then I'd start swearing and banging and I'd break something else...

I haven't done plumbing for a long time, but every year or two I have to fix something big on my main machine and, in the process, other things break. 

This year's breakdown began when my 2009 iMac's 2nd 1TB drive started to fail (The SSD replacement is in the queue.). Next, after cloning the primary drive to an external firewire drive, I noticed an old Permissions bug I've probably ignored for years. Sophos Antivirus didn't seem to like the clone, and I decided it wasn't worth the bother anyway, which led me to Sophos uninstall heck (mostly missing documentation). Next building a Mountain Lion Install SD card for my new SSD exposed weird behaviors of the ML Installer.

I was ready for a break when my SSD came and Mountain Lion 10.8.4 installed nicely (alas, it brought me iTunes 11 as well. I was trying to stay on 10). I decided it was a good time to move my iTunes Library to an external firewire drive. That meant I needed to make some room. No problem, I could just delete a 2.4TB Carbon Copy Cloner sparsebundle...

Cue the shark music. I'll put the details below the fold, but OS X cannot delete sparsebundles with large numbers of bands. The best explanation, and the only plausible solution, were this Mac OX Hints article and its comments ...

Script to delete huge sparsebundle images - Mac OS X Hints

... for i in {0..1000000}; do rm -rv /Volumes/Video1/deleteme.sparseimage/bands$(printf "%x" $i); done ...

... This is a pretty common issue, and a very frustrating one because the error isn't clear. Wildcards in the command line expand, which if you have 250,000 files in a directory means you are sending:

rm file1 file2 file3 ... file250000

That's where you get too many arguments from. (It's also technically a kernel-level error, not a bug with rm; see link below)...

and see ...

osx - OS X - argument list too long for rm -rf *, 4000 files - Super User

I tried getconf ARG_MAX and the limit for Mountain Lion is 262,144. Unfortunately an Sparsebundle band is about 8MB, and 2.4TB means 314,572 bands -- over the Mountain Lion ARG_MAX limit. When I tried to delete Mountain Lion sat calmly, without much change to Activity Monitor or Console, then a beachball began to spin, then the machine quietly and completely died (actually, in one instance a background app continued to work). The bug [2] kills OS X deep in its BSD Unix heart.

So what did I do to delete this undeletable monster?

Well, I think if I'd understood the problem early on [1], I might have had some luck with " for i in {0..1000000}; do rm -rv...". However, there were two additional contributing factors. One was that this was an encrypted sparsebundle; but the bigger one was that by the time I got to the best fix I'd tortured the file a dozen different ways. I had to move data off the drive to other storage, reformat the drive, then move it back.

I can't wait for my next misadventure. In the interests of anyone searching on this general topic, here are some of the things I found along the way:

- fn -

[1] If I had started by searching my own blog post archives I'd have found the solution immediately: OS X Limitations: working around deletion of large numbers of files (time machine image) 7/2012. Stupidly I used Google, which doesn't favor my posts.

[2] Apple fails this test so many ways. The Unix limitation, the ability to create sparse bundle images that can't be deleted, the lack of error handling, the lack of diagnostic messages, etc. That's on top of the many other problems with deleting files and emptying the trash (permissions, locks, etc).

Update 9/6/2013: This is so weird I can only report it for what it's worth. OS X wouldn't 'mount' the disk in order to format it. I think I had to first format it as FAT before I could format it as HPFS.

No comments: