Micro-Frameworks for web app development

A developer colleague (M.A) sent me a brief list of micro-frameworks organized by language. His list is in the same vein as Microjs: Fantastic Micro-Frameworks and Micro-Libraries for Fun and Profit but quite a bit shorter.

For my own future reference, here's his list organized by server-side language

• Java – Spark or perhaps something old fashion like Tomcat or Spring MVC in Tomcat
• Groovy – Grails or Ratpack
• Javascript – node.js or Meteor
• Ruby – Sinatra
• PHP – PHP
• Python – Django or Bottle

For my own amusement (and perhaps my 14yo) I'd be inclined towards either Django (Python and packaged on DreamHost, my longtime hosting service) or Meteor (he likes).

PS. Clearly the world needs an AppleScript micro-framework. (ok, sick joke)

The personal (corporate?) search tool I want

The modern publicly traded corporation is to data as water is to iron. Corrosive. There are reasons why this is getting worse - legal, regulatory, economic, political, technological. It's a long story, but trust me on the corrosion part.

Now if I kept all my data  on my personal dry dock workstation (we still have them) I could resist this, but there's power in sharing. So over time pieces of my extended (work) memory have fallen into abandoned repositories. Recently the number and complexity of these abandoned repositories ran past my cognitive limits.

In the long run part of the the solution is a corrosion resistant knowledge repository, but in the short run I need a way to track and search my archives and working repositories. I need an integrated personal search tool for pulling in data from a variety of server based search APIs. [4].

I haven't seen a tool like this and I can't be the only one who needs one; there's probably a (very) wee market here [1]. At least 10 people. Worldwide.

It's not too hard to imagine how it might work as a web app:

There's a search box, a drop down with "All" or single target searches. Send (?Customized) strings to APIs of various repositories like SharePoint_1, SharePoint_2, Yammer, XWiki, Confluence, Rally, JIRA and so on. Get results back, convert to a normal form, display in a grid.

On the other hand, a web page of links to the various search engines would be better than nothing, and an embedded set of search forms would be quite good esp. with a little javascript to copy a string from one field to every search string.

So what I need is an environment that lets me start with a simple web page of links, then add embedded forms, and gradually build more capability over time. A kind of hobby project I can work on when I'm stalled on my real work and need something to that's plausibly work related.

Maybe Meteor ... 

[1] Before Google there were tools like this for the public net, but post-Google those have been relegated to (mostly) failed meta-search engine projects like dogpile, search.com, and, arguably, duck duck go. I haven't found tools that work inside corporate firewalls.

[4] My personal custom search engine fills a similar role for the Google-accessible net.

See also

• Gordon's Notes: The corporate wiki I want 9/2013
• Gordon's Notes: Project Memfail: Tackling my search space problem 9/2013
• Search Engines: Google, AltaVista, and Profusion @1999. The Profusion domain name is currently owned by a data science consulting group.

App.net: Supporting account substreams with PourOver

[This one's for @duerig.]

The earliest mention of "channels" in my web archives dates to 1996 [1]. There's not much more than a word about them, but I remember what I was thinking. There were a lot of things I wanted to share [2], but I didn't enjoy harming unwilling bystanders. I wanted broadcast channels (now we call them streams) that could be carved from my global shares [3].

The problem, of course, is that my interests are probably not your interests. Emily is my most faithful reader, but she skips my tech shares. On app.net some like my diverse shares, but others favor dialog and social chat. Political opinions? Religion? Right. Limited scope.

So, in the interests of minimizing collateral damage, like a political post appearing in a stream of iOS comments, I'd like an easy way to do streams off my shares.

Happily Pinboard, which I use as a micro-blogging platform publishing to @johngordon  (PourOver) and kateva.org/sh (IFTTT), supports those kinds of streams. Every tag has a feed, and when posting to Pinboard I can enter single character tags corresponding to streams. It's not the most elegant UI, but it works.

At the moment though all of my shares stream into one app.net channel (mixing metaphors there, but it kind of works). If my app.net account supported sub-channels/streams (I know that work is in progress, might be done) it seems like either PourOver and/or Pinboard stream-feeds would be a good step towards reducing drive-by share damage.

Update: app.net thread. Hope to see these pieces come together over the next few months.

- fn-

[1] My web "posts" from the early 90s are now embarrassing. The web was new then, even Alta Vista was years away. There was so much I couldn't imagine. More subtly, we live in the Randall Munroe web now. I know there are minds at play far beyond my own meager insights.
[2] Sharebot I am.
[3] In those days Global Shares were static web pages. I tried to generate things that were a cross between blog posts and Simplenote entries via FileMaker web page generation.

See also

• Filtering my Google Reader Share with Yahoo Pipes 9/2008 - That was my most serious streaming effort, not coincidentally around election time.
• Gordon's Tech: App.net - using Duerig's custom RSS feed to see only root posts from selected people
• Tweeting Google Reader Shares and Notes via feedburner 2010 - Google Reader shares had channel-like options
• Google reader shared items to Facebook 2009 - another example of trying to stream

Aperture's multi-project display and why you've never heard of it.

Since at least 2010 Aperture has been able to display multiple Projects side-by-side in a tabbed UI by option-click on Project name. I was amazed to learn about this a few weeks ago. Why isn't this prominently discussed in Aperture's manual or help file? Why, even after I showed it worked, is it so hard to find documentation?

I mean - this is big. I've been looking for it for years, missing iPhoto 9's easy ability to split and merge Projects and move images between them. Aperture's single project focus is my biggest complaint. At last I can move images between Projects ...

No. You can't. 

Which is probably why Apple has never documented this feature -- it's obviously only half-built. Rather than pull tabbed albums projects from the release Apple left them in, but removed documentation. Sad that in some fine Aperture updates since 2010 this feature was never completed.

Maybe in Aperture 4.

Apple still has an express replacement service for iPhone: $187 for iPhone 4

Years ago I think I used Apple's 'express replacement service', probably for a 3G. Apple sent me a refurb, I returned my phone and used the refurb.

Turns out, the express replacement service is still around. Request the express replacement option for an out of warrantee service and you can swap your defective old iPhone for a refurb (with a fresh battery). In our case we have an iPhone 4 with a balky home button and another with a balky power button -- both with bad batteries. Good devices, but not much resale value. An Apple Store service swap would be great if one could be arranged, but an express replacement service would be a lot more convenient.

According to Apple's web site there's a $29 service fee, a $7 shipping fee, and device specific fees:

• iPhone 4 or earlier: $150
• iPhone 4s: $199
• iPhone 5, 5c, 5s: $269

I don't know if the replacement phones are carrier locked, so there's a risk of sending in an unlocked phone and getting a carrier locked phone back. 

An eBay iPhone 5 seems to cost $200-$300 and is probably much lower quality than an Apple refurb, so $187 is pretty competitive.

Analyzing mobile phone plans: Our old AT&T plan vs. H2O wireless

We're enrolled in a no longer available AT&T plan that included a secondary phone option of 

• shared family minutes, long distance, etc.
• $10 monthly per device
• $15 for 200MB data (with alerts when limit nears)
• No SMS

At the moment my daughter is on this plan, the boys are on H2O wireless. Thanks to typical carrier hidden fees and some SMS usage billed at extortionary rates** the average monthly cost is about $33.

I'd been planning to switch her to H2O wireless too, but after doing a bit of analysis I can see why AT&T discontinued the plan we have -- it's oddly cost-effective for someone with the discipline to control data use. (The latter requires a level of self control that excludes her brothers.)

Here's a rough cut at what a valuation looks like for staying on this plan over 2 years (with a new contract iPhone) vs. switching to H2O wireless:

Item	Estimated value
Phone subsidy	450
AT&T sneaky phone fee	-40
iPhone 5S 16GB initial fee	-200
Two year payments	-840
H2O plan cost	120*
NET	-510

So compared to H2O wireless staying on this plan would mean:

• Pay an extra $510 over two years
• Get an iPhone 5S (list $650)
• Get enough data for location services, light email, iMessage, Facetime-audio

If we wanted an extra iPhone 5S her current plan would make sense. We don't, so I'm likely to switch her to H2O wireless when her contract expires - like her sibs. On the other hand, if we lose a phone it makes sense to add her back on this plan and get a new contract.

Again, I can see why AT&T discontinued this plan. It is oddly competitive with paygo options.

* As of today H2O wireless for our kids use pattern costs about $80 for two years - voice and SMS only. It used to cost $200 for two years, this $80 price is weird and seems unlikely to last. In fact I'm not sure I can even get this plan for her any more, I think it was an artifact of some pre-smartphone pricing.

** Most of her texting is via iMessage.

PS. Walmart online has an interesting list of prepaid options for AT&T compatible MVNOs.

iOS 7 has completely removed ability to play Podcasts through Music.app and Video.app

Message received when connecting my daughter's iOS 7 4S to iTunes:

Until iOS 7 one could get reliable iTunes Podcast sync by deleting Podcast.app and viewing Podcasts in Music.app or Video.app. Those abilities are now gone. Not a surprise, too bad Apple brought a lot of bugs with the transition including iTunes 11.1 Media Kind bugs causing podcast invisibility, and claims of massive data loss when synchronizing archived podcasts with iOS 7

It's well past time for me to switch to either Downcast or Instacast. My app.net colleagues are reviewing the tradeoffs with me.

(I'm still on iOS, I like the kids to find these bugs first.)

iOS 7 fixes iOS parental controls webkit hole. Finally. (EXCEPT for Siri)

It's been exactly three years since I wrote Apple's iPhone parental controls are completely broken.

Sure, you could turn off Safari -- but there was no way to disable use of webkit embedded browsers. A lot of apps and games kids like, including encyclopedias, use links that bring up an embedded browser. From there it's often a few hops to Google and beyond.

I ranted about this in various places, but mine was a lonely voice. (One can imagine many reasons why most parents don't seem to be concerned about full web access with iOS devices, but, whatever the reason, there's clearly no clamor for a fix.)

Today, years after I gave up, Apple fixed parental controls in iOS 7. You can use Restrictions:websites:specific websites only to restrict both Safari and webkit access to urls. I believe the changes were made pretty deep in the iOS network stack, they seem to affect all browser use.

iOS comes preconfigured with a set of approved sites. The list is not simple to edit but they are all fine with me. You can add others.

There are bugs. Even preconfigured sites seem to sometimes require second authentication on attempted access. Still, it's a big, albeit very late, improvement.

IOS 7 is quite slow on the iPhone 4s two of the kids use, but this one feature is worth the sluggishness.

Update: In early testing #1 says he can't hack the current restrictions. It also seems to be far more useable than superficially similar site restrictions in OS X Mountain Lion; Mountain Lion's current mechanism has been completely broken. I wonder if some serious attention went into making this work.

Update 11/29/2013.

Siri: "Show me pictures of dogs". Shows dogs.

Siri: "Show me pictures of xxxx"....

You have to disable Siri, there are no parental controls there.

iTunes 11.1 is unable to browse some older podcast files - with a partial workaround (Fix)

The first hints of a problem with iTunes 11.1 came via an app.net referral to Kirkville: Apple Has Broken Podcasts. A large numbers of older podcasts were no longer seen in iTunes. They weren't deleted, but iTunes didn't show them.

My first thought was gratitude for my multiple onsite and offsite backups. Unsurprisingly, I'm impacted too. I have 367 episodes of In Our Time in my iTunes Podcast folder, but only 311 are browsable in iTunes -- either via Podcasts or Music. On the other hand, a Smart Playlist searching on the album "In Our Time" finds them all even if I specify Media Kind = "Podcast" in the search criteria. 

Search won't find the lost podcasts however -- only Smart Playlists.

My guess on this bug is that Apple changed the rules on what shows up in Podcasts or Music so that certain older files with a Media Type of 'podcast' don't display in either category. They're still in the iTunes database, and so discoverable via a smart playlists, but iTunes can't browse them. If you remove these files from iTunes, then add them back in, they may be reclassified so they'll be browsable again.

I wonder if there's a way to do that via AppleScript.

I'm hoping this bug gets enough attention that Apple fixes it in the next month or two.

Update 2: The bug is related to Media Kind

Media Kind has long been an Achilles Heel of iTunes. It's an attribute of media that shows up in Smart Playlists and should be changeable via the Information (Get Info) window, but there's no 'column' option for showing Media Kind in lists. It looks like this is a Media Kind option.

I created two Playlists to identify my affected IOT podcasts.

I made one Playlist by dragging all the files that showed up in the Podcast view into a static Playlist.

I made another by Smart criteria: Album = In Our Time.

Then I made a 3rd to identify what was in the Smart Playlist, but not visible elsewhere:

That showed my hidden podcasts.

Of the 50 or so hidden podcasts, I tried changing media type to Music. That worked for two of them. They were now visible for search and browsing. The other 48 appeared to let me change Media Type, but when I checked again they still showed as Podcast. It seems iTunes 11.1 is ignoring the Media Type attribute and using a different source of metadata to decide what is a "Podcast". That's bad, but what's worse is that podcasts of mine that used to show as Music no longer show there, but they are also omitted from Podcast.

Update 3: A workaround: Media Kind Podcast -> Audiobook -> Podcast

I couldn't change the media kind for the '48' to Music (seemed to change, but didn't work), but I could change to Audiobook! Problem is Audiobook UI can't scale this way.

Once I'd changed the Media Kind to Audiobook though, I COULD change it back to Podcast. After that all my IOT files were Podcasts and once again visible to browse and search. Note to do all this I had to use Smart Playlists -- these were the only parts of ITunes 11.1 that could display my 'invisible' podcasts.

Heavens, but iTunes 11.1 is a hot mess.

See also:

• Podcasts playlist in iTunes is empty March 2012, so not 11.1
• Why did iTunes 11.1 (126) delete all: lots of people affected by this, though some seem to have actually lost data
• Kirkville: Apple Has Broken Podcasts
• iOS 6.01 Podcast app: Die, Apple, Die. 11/2012. Apple hates podcasts.
• Apple breaks Smart Playlists on iPhone and iTunes alike (yet again) 10/2009. Apple had Media Kind bugs four years ago.

Curse of the refurb iPhone: Apple refurbished 4S has audio compiler defect, drops microphone on calls

When Emily broke the screen on her 2yo 4S I paid for a refurb replacement so I'd get a fresh battery. I'd heard good things about Apple refurbs.

Wrong.

It now sometimes happens sometimes happens that her microphone doesn't work during calls. A power cycle or simply waiting a while can resolve it. Happens about once a week. I think this is the cause:

During a call, the other person can't hear me - iPhone 4 - iFixit

I'm sorry to tell but it is 99% sure that your 'audio compiler ic' is the issue. This is the IC which performs the noise cancellation. It is only active during calls and not during memo recording... Reballing this IC is 99% gonna solve your problem. However this requires special tools and good knowledge to be performed..."

There's a possible workaround using the hearing aid option, but I'll take the phone back to the Apple Store. I wonder if they'll believe me -- I'll try to catch it on video.

It looks like this is a known manufacturing flaw:

ian's iPhone Repair: Microphone Issues with iPhone 4

If this audio compiler doesn't work, then the person on the other end will not hear you - or they will hear a very muffled voice, or a lot of static. Due to what is largely believed to be a manufacturing error in Apple's Chinese factories, a number of iPhone 4 models were assembled with this tiny chip soldered to the mainboard (motherboard) improperly. Either too much solder or glue was used, and the connection is tenuous at best.

The problem is that the phone could work perfectly for a very long time, and then after a drop (even one that doesn't break the glass), a hard jar, or even for no reason at all, this chip loses its connection to the mainboard and causes the problem you may have experienced.

There's a 178 page Apple thread on this problem: iPhone 4S - Outgoing call no audio. Given the length of the thread it's disturbing Apple missed the problem on our refurb. They should offer a recall.

Update: I found if I gently tapped the 4S on a surface I could trigger the microphone loss, so I was able to get a recording of the drop out.

Update 9/7/2013: Apple service-swapped the service-swap (so now I'm two phones removed from the phone AT&T would have unlocked in 2 months, but I'm told AT&T can manage this if I have the paper trail).

It wasn't easy though. I'd power-cycled the phone to switch SIMs, and last time I did that it took days before I could replicate the bug. So the tech couldn't replicate the problem -- and nobody at the Rosedale Minnesota Apple store had ever heard of the (alleged) audio compiler microphone cut-off defect. It took my poorly done video of the defect in action, a Google search with precise hits, and showing this blog post to get the exchange. 

I sympathize with Apple here -- a hardware exchange for a non-replicable defect is a lot of money to lose. I wonder if they're not allowed to use Google though.

Mountain Lion and the encrypted boot drive: Implications for migration assistant and what happens when you delete the only account that had FileVault 2 access (bug)

For several reasons I decided to enable encryption on my new SSD boot drive. I used the admin account on the drive. I then migrated data from my old drive, restarted, switched to my regular admin account, and deleted the admin account I'd created for drive setup.

That's the problem. Even though that account has been deleted, when I restart the Mac the startup partition (Apple_Boot Recovery HD?) I'm asked for the password for that account.

That sounds like a bug, but it could be worse. That's because when you setup a boot drive as FileVault, then use Migration Assistant, you have to enable FileVault unlocking for each of the migrated users. [1]. I'd unwittingly deleted the only account that was authorized to decrypt my boot drive.

Once I enabled my other accounts for unlocking they appeared on the startup menu -- along with my deleted admin account. So the deleted account is still used by the hidden boot partition, and it probably can't be deleted nor can the password be changed. So, yeah, it's a bug.

FileVault 2 makes me nervous.

See also:

• [1] OS X: About FileVault 2 - Apple support. This is mandatory reading. "f you want to make the Mac available to a user that does not have unlock capabilities, log in, then when you see your own desktop, choose "Log Out (user name)" from the Apple () menu. Also, you can unlock the disk, then choose the other user's name from the Fast User Switch (appears as the currently-logged in user's name) menubar item in the upper-right part of the screen ... When FileVault 2 is enabled, Recovery HD does not appear in the Startup Manager (which is accessed by holding Option during startup).  However, you can select the Recovery HD by holding Command-R as Lion starts up."
• OS X: How to create and deploy a recovery key for FileVault 2 - This might be the most advanced support article I've read. The recovery key for a FileVault 2 encrypted disk is shown ONCE on startup and cannot be later displayed, but using this method one can save a key that can be used when a password is forgotten. (Maybe this is what Apple does when you elect to save credentials with them.)
• osx - Disable a user's ability to unlock a FileVault 2 volume at startup/login time - Ask Different: This is the best overview of the bug with FileVault 2 and inability to "remove, from the EFI loginwindow, a user who should no longer be able to unlock the startup volume."
• Using fdesetup with Mountain Lion’s FileVault 2 | Der Flounder 7/2012 - Remove users from the list of FileVault enabled accounts.
• training.apple.com/pdf/WP_FileVault2.pdf: Apple Technical White Paper. Best Practices for Deploying FileVault 2 - Deploying OS X Full Disk Encryption Technology

Update 9/11/2013

I tried sudo fdesetup list and the list did not include the unwanted user account. So I restarted and this time it didn't appear. So perhaps 1-2 restarts after enabling users took care of my orphaned EFI LoginWindow account.

I've seen some other odd behaviors, but I may get to those another time.

Google Checkout ends Nov 20, 2013 - review all of your Google Apps eNom registered domains now

Google is ending their Google Checkout service, it closes 11/20/2013. The support document on this discusses retiring Google Checkout Merchant.

That's now how I used Google Checkout though, I used it as a way to manage net purchases [5]. This is where things get confusing; because while merchant use of Google Checkout is dead, my transaction history survives with Google Wallet. As best I can tell Google Checkout non-Merchant became Google Wallet [2]. It lists, for example, all of my Google App related transactions, Google Storage [1] and domain registrations.

Ok, the last may need some explanation. Years ago, before Google wed Verizon under the red moon and drank the blood of innocent children to become the Dark Lord of the Net, Google Apps was launched as a freemium service. Anyone could sign up for a domain (most often managed by eNom) and get the full suite of Google Services for up to 20 people (or maybe more). All the payments were managed through Google Checkout, so there was one place to manage charges.

Those were the glory days. I know it's hard to remember a time when Google wasn't a synonym for Evil, but once upon a time we thought Google was different [3]. We snarfed up eNom registered Google Apps domains - I have 7 of and use 3 every day including kateva.org. Despite the blood and spells and yada yada Google lifted a claw and let us keep the free domains even after they ended the program.

It was all quite easy -- but now Google Checkout is gone, and we can't use Google Wallet to pay for those eNOM registrations [4]. So we have to go to every ... single... Google Apps account and enter our credit card information.

I just finished the 7. If I used Clark Goble's beloved Keyboard Maestro it would have been faster.

It wasn't entirely a wasted exercise. I'd set some of those domains up with what is today a laughably hackable password. They've probably been doing unspeakable things while I was away (though the pw still worked). They now all have 20+ character random string passwords that are almost as hard to type as they are to hack (don't even think of 'tap'). They also have new recovery and security alert features.

I'll gradually move the eNOM domains over to my Dreamhost account; I like Dreamhost as a registrar and that way I'll only have to update my credit card once.

 - fn -

[1] I'm grandfathered into the very cheap $20/year rate. Google hiked prices after that.

[2] I'm not sure what Google Wallet is, but it's not intended for payments for online goods and services -- where it once acted as an intermediary between my credit card and merchants. It seems to be for mobile and in-person payments. I don't know where Google is going with this, but I don't care either.

[3] Even in 2009 they were changing ...

[4] The Google Apps charges are $0.00 monthly.

[5] Seems I was the only one. It worked well for me, but I guess now there's only Amazon.

OS X Mountain Lion cannot delete sparsebundles containing over 262,144 bands (2TB+)

Working on my primary Mac reminds me of working on cheap old plumbing. I'd start out fixing one piece, then another would break, then I'd fix that, then I'd start swearing and banging and I'd break something else...

I haven't done plumbing for a long time, but every year or two I have to fix something big on my main machine and, in the process, other things break. 

This year's breakdown began when my 2009 iMac's 2nd 1TB drive started to fail (The SSD replacement is in the queue.). Next, after cloning the primary drive to an external firewire drive, I noticed an old Permissions bug I've probably ignored for years. Sophos Antivirus didn't seem to like the clone, and I decided it wasn't worth the bother anyway, which led me to Sophos uninstall heck (mostly missing documentation). Next building a Mountain Lion Install SD card for my new SSD exposed weird behaviors of the ML Installer.

I was ready for a break when my SSD came and Mountain Lion 10.8.4 installed nicely (alas, it brought me iTunes 11 as well. I was trying to stay on 10). I decided it was a good time to move my iTunes Library to an external firewire drive. That meant I needed to make some room. No problem, I could just delete a 2.4TB Carbon Copy Cloner sparsebundle...

Cue the shark music. I'll put the details below the fold, but OS X cannot delete sparsebundles with large numbers of bands. The best explanation, and the only plausible solution, were this Mac OX Hints article and its comments ...

Script to delete huge sparsebundle images - Mac OS X Hints

... for i in {0..1000000}; do rm -rv /Volumes/Video1/deleteme.sparseimage/bands$(printf "%x" $i); done ...

... This is a pretty common issue, and a very frustrating one because the error isn't clear. Wildcards in the command line expand, which if you have 250,000 files in a directory means you are sending:

rm file1 file2 file3 ... file250000

That's where you get too many arguments from. (It's also technically a kernel-level error, not a bug with rm; see link below)...

and see ...

osx - OS X - argument list too long for rm -rf *, 4000 files - Super User

I tried getconf ARG_MAX and the limit for Mountain Lion is 262,144. Unfortunately an Sparsebundle band is about 8MB, and 2.4TB means 314,572 bands -- over the Mountain Lion ARG_MAX limit. When I tried to delete Mountain Lion sat calmly, without much change to Activity Monitor or Console, then a beachball began to spin, then the machine quietly and completely died (actually, in one instance a background app continued to work). The bug [2] kills OS X deep in its BSD Unix heart.

So what did I do to delete this undeletable monster?

Well, I think if I'd understood the problem early on [1], I might have had some luck with " for i in {0..1000000}; do rm -rv...". However, there were two additional contributing factors. One was that this was an encrypted sparsebundle; but the bigger one was that by the time I got to the best fix I'd tortured the file a dozen different ways. I had to move data off the drive to other storage, reformat the drive, then move it back.

I can't wait for my next misadventure. In the interests of anyone searching on this general topic, here are some of the things I found along the way:

• OS X Limitations: working around deletion of large numbers of files (time machine image Variants of this bug typically showed up under Snow Leopard with Time Machine sparsebundles, but those were easily managed with rm -r or even putting the file into a folder and deleting the folder. This version was much nastier.
• You can't empty the Trash or move a file to the Trash: Apple's official documentation on the many ways that Trash emptying can fail. They don't mention this one though.
• If you have a file in the Trash you can't delete, pull it out of the Trash into the user account that owns it. Then move it over to an Admin account for surgery.
• Onyx and secure delete tricks don't work on this kind of monster.
• TidBITS Talk - Deleting old sparse.bundle?: quite good discussion but these are mere 100MB sparsebundles -- well under ARG_MAX limits.
• Trash won’t empty? The ultimate guide to deleting stubborn files | MacYourself 12/2009 - lots of classic manuevers for more common and simpler problems.
• I tried opening the image in Disk Utility and removing the partition, erasing, reformatting, etc. Might have worked if I hadn't tortured it so much and possibly worth a try in the future.

- fn -

[1] If I had started by searching my own blog post archives I'd have found the solution immediately: OS X Limitations: working around deletion of large numbers of files (time machine image) 7/2012. Stupidly I used Google, which doesn't favor my posts.

[2] Apple fails this test so many ways. The Unix limitation, the ability to create sparse bundle images that can't be deleted, the lack of error handling, the lack of diagnostic messages, etc. That's on top of the many other problems with deleting files and emptying the trash (permissions, locks, etc).

Update 9/6/2013: This is so weird I can only report it for what it's worth. OS X wouldn't 'mount' the disk in order to format it. I think I had to first format it as FAT before I could format it as HPFS.

Downloading a fresh copy of Mountain Lion Installer and using Lion DiskMaker to create an SD card boot drive

Today, in preparation for a 1TB SSD install following the demise of my 2nd 1TB Samsung 2009 iMac boot drive, I created a Mountain Lion 10.8.4 installer disk:

 

The process was more difficult than I'd expected, with some of the same Mountain Lion install problems I'd experienced during my original migration from Snow Leopard. The process of downloading and retaining the Mountain Lion Installer was particularly obscure, and none of the web references I read were entirely correct.

At this point I'd normally be railing about Apple, but it's clear they're as bewildered by the end of physical media distribution (DVD in this case) as the rest of us. (Though they certainly could have made SD distribution more palatable.) As I traced through the process I could see how Apple has been iterating on ML delivery over the past 8 months. Maybe things will go better with Mavericks.

After I was done with this process I did find Apple's official description, hidden in an obscure and mis-titled technical document ...

 OS X: About OS X Recovery

... If you completed your installation of OS X, your installer may have been removed after your successful first login to OS X Lion or OS X Mountain Lion. Mac App Store's Purchases page should show Install OS X as being "Installed", and disallow its download, when viewed from a computer running OS X.

To redownload the installer on a computer running OS X Lion or OS X Mountain Lion, press and hold the Option key while you click the Purchases tab. If the button to the right of the Install OS X Lion or OS X Mountain Lion item doesn't change to "Install" and allow you to download OS X, use Spotlight to search for "Install OS X Lion or OS X Mountain Lion" on your computer.

Launch the Install OS X Lion or OS X Mountain Lion installer you downloaded from Mac App Store. The installer should be in the /Applications folder. Follow the onscreen instructions to complete the OS X installation. Be sure you install Lion on the external storage device you've connected for this purpose, not your computer's boot drive. When installation to your external device is complete, you can re-run Install Mac OS X Lion or OS X Mountain Lion installer and upgrade the boot drive of your computer. A Recovery System will likely not be created, but if you need to reinstall or repair your boot drive at a later date, you can connect the external drive you just prepared and hold Command-R while restarting computer in order to boot from the external Recovery System.

Did you get that? Yes, the damned option key again - though as of 9/2013 the installer showed me a download button. They're wrong about this though: "The installer should be in the /Applications folder". Things are ... more complex. Even a bit astonishing.

I'll describe what I did today, though at the rate things are changing this advice is probably only good for a week or two. I was guided by bits and pieces of these additional references:

• OS X: About Recovery Disk Assistant - A shorter and more understandable guide to Apple's current recommendation.
• mac - Installing Mountain Lion on a fresh SSD - Super User
• Install Mountain Lion on As Many Macs As You Want Without Purchasing Additional Copies - Lifehacker
• upgrade - Can I download Mountain Lion and install it later? - Ask Different (I updated this with a corrected answer)
• install - How can I download the Lion installation file? - Ask Different - Lion description, somewhat older
• How do I install Mac OS 10.8 Mountain...: Apple Support Communities

The last of these, when combined with Apple's OS X: About Recovery Disk Assistant is a pretty good guide to what the Apple recommended (normal) route would be for installing Mountain Lion from the App Store on new boot media (HDD, SDD, etc). Here's via1083's explanation in the case of a fresh SSD install.

1. Before installing run Apple Software Update to make sure that you have the latest software.
2. Download the OS X Recovery Disk Assistant. This will allow you to create a bootable Recovery partition on a USB thumb drive that will allow you to reinstall OS X Mountain Lion. You need to have at least 1GB of space on your USB drive. For more information, including how to properly set up a USB thumb drive, visit About OS X Recovery Disk Assistant.
3. Shut Down your Mac and connect the USB stick that you prepared with the Recovery assistant to your Mac.
4. Turn on your Mac and hold the option key. Select the Recovery USB thumbdrive as the boot up disk. Make sure that the computer successfully boots up to it.
5. Exit the Recovery USB by going to the Apple icon and selecting shutdown. Disconnect the USB thumbdrive.
6. Proceed to upgrade your MacBook Pro with the SSD drive.
7. Before turning on your Mac, re-connect your Recovery USB thumbdrive.
8. Turn On your Mac, hold the "option" key and boot from the USB thumbdrive.
9. When the OS X recovery tools menu appears, select the Disk Utility Options.
10. Follow Kappy's tips on SSD drive preparation.
11. Quit Disk Utility.
12. In the OS X recovery tools menu select the "Install OS X" option.
13. Follow the instructions of the installer and agree with the terms and conditions, OS X will then proceed to the downloaded from Apple's servers. (This may take a while depending on your Internet connection's speed. Also, the computer will send its serial number to the Apple servers to validate the copy of OS X Mountain Lion.)
14. OS X Mountain Lion will proceed to install. This process will take some time, your Mac will reboot once installation is finished.
15. Follow the OS X set up assistant instructions after your Mac boots from installation.
16. Done! Proceed to install the rest of your Apps and restoring your documents if you use a back up solution

That's not the route I took though. I created a Mac OS X Installer SD using Lion DiskMaker.

 

First though I wanted a fresh copy of Mountain Lion. That turned out to be ... weird. I won't go through all the twists and turns but there's one key thing to know that I've not seen documented anywhere else:

• If you have a version of Install OS X Mountain Lion.app in range of Spotlight the App Store install process with update that version rather than create a new version in /Applications.

Yeah, that is hard to believe. But look at the log file I dug up from the installer (edited here, emphases mine, note the use of the tmpdir during install, I was using a purpose-created admin account I called 'temp'):

 

Sep  2 08:04:51 SaintPaul-2.local Install OS X Mountain Lion[5079]: @(#)PROGRAM:Install  PROJECT:Install-735

Sep  2 08:04:51 SaintPaul-2.local Install OS X Mountain Lion[5079]: @(#)PROGRAM:IA  PROJECT:InstallAssistant-360.1

Sep  2 08:04:51 SaintPaul-2.local Install OS X Mountain Lion[5079]: Hardware: iMac11,1 @ 2.66 GHz (x 4), 8192 MB RAM

Sep  2 08:04:51 SaintPaul-2.local Install OS X Mountain Lion[5079]: Running OS Build: Mac OS X 10.8.4 (12E55)

Sep  2 08:04:51 SaintPaul-2.local Install OS X Mountain Lion[5079]: Env: PATH=/usr/bin:/bin:/usr/sbin:/sbin

Sep  2 08:04:51 SaintPaul-2.local Install OS X Mountain Lion[5079]: Env: TMPDIR=/var/folders/js/329jxxks1gx77q7n_62bg6fh0000gq/T/

Sep  2 08:04:51 SaintPaul-2.local Install OS X Mountain Lion[5079]: Env: SHELL=/bin/bash

Sep  2 08:04:51 SaintPaul-2.local Install OS X Mountain Lion[5079]: Env: HOME=/Users/temp

Sep  2 08:04:51 SaintPaul-2.local Install OS X Mountain Lion[5079]: Env: USER=temp

Sep  2 08:04:51 SaintPaul-2.local Install OS X Mountain Lion[5079]: Env: LOGNAME=temp

Sep  2 08:04:51 SaintPaul-2.local Install OS X Mountain Lion[5079]: Env: SSH_AUTH_SOCK=/tmp/launch-aetmfB/Listeners

Sep  2 08:04:51 SaintPaul-2.local Install OS X Mountain Lion[5079]: Env: Apple_Ubiquity_Message=/tmp/launch-suN6EE/Apple_Ubiquity_Message

Sep  2 08:04:51 SaintPaul-2.local Install OS X Mountain Lion[5079]: Env: Apple_PubSub_Socket_Render=/tmp/launch-NTelRv/Render

Sep  2 08:04:51 SaintPaul-2.local Install OS X Mountain Lion[5079]: Env: COMMAND_MODE=unix2003

Sep  2 08:04:51 SaintPaul-2.local Install OS X Mountain Lion[5079]: Env: __CF_USER_TEXT_ENCODING=0x1F7:0:0

Sep  2 08:04:58 SaintPaul-2.local Install OS X Mountain Lion[5079]: Using distribution archive from /Users/jfaughnan 1/Public/AppInst/Mac App install/OSUpdates/Install OS X Mountain Lion.app/Contents/SharedSupport/OSInstall.mpkg

Sep  2 08:04:58 SaintPaul-2.local Install OS X Mountain Lion[5079]: Configuring products from SharedSupport folder

Sep  2 08:04:58 SaintPaul-2.local Install OS X Mountain Lion[5079]: Using product <OSInstallDVDProduct> based on media at /Users/jfaughnan 1/Public/AppInst/Mac App install/OSUpdates/Install OS X Mountain Lion.app/Contents/SharedSupport at distance 5...

 

Yes, the installer found an older version of Install OS X Mountain Lion and updated it rather than create a new app in /Applications.

 

Ok, now that you've got your head around this behavior,  here's an expanded version of what I posted to Ask Different.

1. As of 9/2013 the app store lets us download multiple instances of Mountain Lion without necessarily clicking on the option key to get the Download button to appear.
2. Lifehacker recommends downloading from a fresh purpose-created app store account. I suspect that's because they ran into the update behavior using an account with access to a prior ML download. I did this, but not sure it's necessary.
3. After the 4.4 GB download completed I quit the installer. Since the installer had located /Users/jfaughnan 1/Public/AppInst/Mac App install/OSUpdates/Install OS X Mountain Lion.app/Contents/SharedSupport/OSInstall.mpkg it updated that package [1]. I think if it hadn't found that prior installer it would have generated an install app and stored it in /Applications.

Once I figured out what had happened I used Lion DiskMaker to create the ML installer on an 8GB SD card. It works by mounting InstallESD.dmg and BaseSystem.dmg then creating  the boot and ESD components on the SD card.

 

After completion I tried holding down the option key on restart -- with a wireless Bluetooth keyboard. That didn't work, in fact I ended up with a familiar no mouse kb startup that resolved when I disconnected my firewire connected Epson scanner. I then used Startup Disk.app to choose the SD boot disk and I confirmed it worked.

So now I'm ready for the replacement SSD drive to arrive.

- fn -

[1] Spotlight found that installer even though it was in a different user account because it was in a shared folder.

Update 10/30/2013: @gaelicwizard explained the Spotlight-updating-Installer behavior in an app.net thread. This is simply how the app store updater works -- the app being updated doesn't have to be in the Applications folder. Nothing special about the OS X Installer.

Uninstalling Sophos Antivirus version 9 for Mac - the web site documentation is incorrect

[It's undocumented, but after I wrote this I discovered the Antivirus 9 install puts an app called 'Remove Sophos Anti-Virus' in the Applications folder.]

---------

In a moment of madness, I installed Sophos Antivirus.

Forgive me, I have sinned. It's a mess.

Unfortunately, the online documentation for version 9 removal is incorrect: Using the Terminal to install or remove Sophos Anti-Virus for Mac OS X. 'Remove Sophos Anti-Virus.pkg' no longer exists and there's no longer a GUI uninstaller.

Instead you have to run remove_v9.sh. I tried to do that from my Admin account -- but that failed. You have to run it from the account it was installed from, even though this app is a root install and the original user account might not be an admin account. This worked from my non-admin account in OS X Mountain Lion where Admin is my Administrator account:

jfaughnan$ su Admin Password: bash-3.2$ cd /Library/Sophos\ Anti-Virus bash-3.2$ sudo /Library/Sophos\ Anti-Virus/remove_v9.sh Password:

I got this ...

WARNING: this script permenantly removes Sophos Anti-Virus for Mac version 9 it should NOT be used on a version 8 installation

if there are errors reported by this script, run it again (it fixes itself)

press control-C now to abort (automatically continues in 5 seconds) run with -f (next time) to skip the 5 second delay ..... removing launchagent com.sophos.uiserver for user 502 removing /Library/LaunchAgents/com.sophos.uiserver.plist stopping Sophos Anti-Virus.app removing /Applications/Sophos Anti-Virus.app removing /Library/Sophos Anti-Virus removing /Library/Frameworks/SAVI.framework removing /Library/Frameworks/SUMScanKit.framework removing /Library/Application Support/Sophos removing /Library/Caches/com.sophos.sav removing /Library/Caches/com.sophos.sau removing /Library/Preferences/com.sophos.sau.plist removing /usr/bin/SophosUpdate removing /usr/bin/sweep done error: leftover path /Library/Caches/com.sophos.installer/ needs to be removed error: leftover path /Library/Caches/com.sophos.sxld/ needs to be removed bash-3.2$

 It didn't "fix itself", but I removed the residual paths manually. From an earlier failed effort I suggest also checking (Library is not user Library, it's root):

error: leftover path /Applications/Sophos Anti-Virus.app/ needs to be removed error: leftover path /Library/Frameworks/SAVI.framework/ needs to be removed error: leftover path /Library/Frameworks/SUMScanKit.framework/ needs to be removed error: leftover path /Library/Sophos Anti-Virus/ needs to be removed error: leftover path /Library/Caches/com.sophos.installer/ needs to be removed error: leftover path /Library/Caches/com.sophos.sau/ needs to be removed error: leftover path /Library/Caches/com.sophos.sav/ needs to be removed error: leftover path /Library/Caches/com.sophos.sxld/ needs to be removed error: leftover path /Library/LaunchAgents/com.sophos.uiserver.plist needs to be removed

What a mess.

Don't like your OS X user name? Set an alias.

On one of our machines the admin account is Admin/Administrator, on another it's admin/Administrator, and on a third it's Kateva. Very annoying, wish I'd been consistent. Today I learned I could just set an alias that's easy to type when needed.

Works great. Nice tip.

Update 9/2/2013: I was thinking about this in the context of OS X Migration Assistant. I wonder if there are issues when an Alias matches an account name on the target Mac. The same problems might occur with Time Machine restores. So I'll use this feature cautiously, and consider removing aliases prior to doing a migration.

Permissions bug in OS X: Fetching forever - SOLVED with Workgroup Manager (Bonus: Reset user account permissions with passwordreset.app)

See the update for how I finally solved this bug.

-

This is apparently a longstanding OS X bug that particularly affects veterans. I noticed it while debugging an odd behavior I noticed after a Carbon Coper Cloner operation.

When I look at permissions on some of my Home folder files I see this:

The Fetching is forever -- OS X is trying to resolve a reference that no longer exists. In a new user account permissions look like this:

or this ...

 

One cause of the bug is described by jsd2 in a 2011 Apple Discussion thread on the problem:

Newly created users in Leopard, Snow Leopard, and Lion have "staff" as their primary group, but users left over from Tiger have a primary group that has the same name as the username.  In my system, files and folders created by Tiger-originated accounts show the proper ownerships in Terminal, but if you Get Info from Lion's Finder, the Sharing & Permissions section apparently can't find the proper group name, and continues to display "Fetching…"  If I create a new account, it correctly shows "staff" as the group.

I tried creating a Group that matched my short username (jfaughnan) to see if that would fool Mountain Lion into eliminating the "Fetching" message but it had no effect. So, by now deep into this post, I continued on.

Next I used terminal (ls -l) to inspect permissions on one of those Fetching examples (edited a bit here)

drwxr-xr-x@ jfaughnan  502 

The sequence drwxr-xr-x should be read d rwx r-x r-x and it means this is a directory and it can User can read/write/execute, Group can read/execute and World can read/execute.

502 is my system assign UID (not to be confused with more modern UUID, both can be seen in User and Groups Advanced), but it's showing up here where one would expect to see Group ID (GID). 

Here's the result for a different account, created, I think, in Snow Leopard

drwxrwxrwx  Admin  staff 

In the 2nd case the GID is able to resolve to one of the Mountain Lion group names, which one can display with $ less /etc/group (or skip to the bottom of this post and check out workgroup manager):

nobody:*:-2: nogroup:*:-1: wheel:*:0:root daemon:*:1:root kmem:*:2:root sys:*:3:root owner:*:10: everyone:*:12: staff:*:20:root _appstore:*:33:utmp:*:45: authedusers:*:50:

In the first case the GID can't resolve to the Group Name, because in Tiger there was a GID set equal to my UID, in this case both were 502 - as jsd2 described above.

Now, at this point you might ask a dangerous question. You might point out that ls -l is only showing ONE GID/GroupName, but a file in OS X can belong to multiple Groups (these do). Well, cough, I think OS X permissions are broken. Indeed, while SharePoint 2007 was proof that Microsoft was broken, Mountain Lion permissions are proof Apple is broken.

It gets worse. I used the Finder to remove the 'Fetching' line from "Sharing & Permissions" on some of my old files. Doing this did not, as I expected, remove the '502' reference that shows up with ls -l. It made no difference to the ls -l results at all.

Which suggests to me that the real problem can't be exposed with these Unix permission displays. The real issue may well be buried in ACLs, or wherever OS X now keeps track of File-Group relationships. 

Which is where I end, because for me I think this "Fetching" thing isn't causing any real problems. I'll leave with only two additional notes:

• There's a way to allegedly reset permissions on the Home Directory from the Recovery partition by running 'resetpassword' in Terminal and then NOT resetting password but instead Reset Home Directory Permissions and ACLs. Darwin help you if you try this - it was documented by Apple for 10.5 but NOT Mountain Lion. Have several good backup first.
• If you want to explore further, try Workgroup Manager 10.8 (See OS X Server: Admin tools compatibility information). Fire it up, accept default of local machine, and give it your Admin account un/pw. Lots of interesting information there. When I looked into it I found a NEW Group with GID 503 that had same name as my UID. I wonder where that came from ...

- fn -

[1] I believe since Carbon Copy Cloner cloned my original drive this is the same UID I had then; migration assistant would have changed it. See: OSX Tips: Setting-up a new Mac from an old one, its backups, or a PC Problems after using Migration Assistant.

[2] useful Python command: python -c 'import grp; print grp.getgrnam("staff").gr_gid'

See also

• Gordon's Notes: Is there a club for people who hate OS X permissions? 1/2010

Update 9/1/2013: I've solved the Fetching forever bug. It was easy in the end.

This is what I did:

1. As above, I used ls -l to show me that the unmatched GID associated with the Fetching Forever bug was 502. In my case it was still the same as my UID. (I think ls -l shows one the PRIMARY group associated with a file, whereas Get Info shows all groups.
2. I used Workgroup Manager 10.8 to create a new Group I called Old502. I could have called it anything. The trick is that Workgroup Manager lets us specify the GID. I gave this group the GID of 502.
3. Now, instead of Fetching Forever, OS X Get Info shows Old502. Everything is as it should be, at the cost of an extraneous Group that does no harm.

Update 9/2/2013: In the course of another project (future post) I took a look at Apple's occult user account password reset tool.

I ran it from a booted Mountain Lion Install "disk" (SD card) but it might look the same from a Recovery Partition or Recovery Disk restart.  I started terminal and entered resetpassword, and, after a few minutes, I got this (I entered some values):

It's a weird and awkward UI and the app is obviously mis-titled and completely undocumented, but it appears that after years of confusing users with Disk Utility's near worthless Repair Disk Permissions there is indeed an official way to repair a user's Home Folder Permissions. I have a Disk Migration operation pending, I'm going to give this a try once the data is in place on  fresh Mountain Lion instance.

Update 9/9/2013: After I used migration assistant to move my accounts to a new system I think the permissions were reset back to the standard. The 502 bug persisted though, after I removed the Group I'd created for GID 502 I got 'Fetching' back again.