Carbon Copy Cloner won't create sparse bundle disk image (grayed out) - Mojave

On Mojave as of May 2022 CCC v 5.whatever wouldn't create a sparse bundle disk image for me when I selected new disk image as destination. The 'action button' was grayed out. 

I switched from the non-admin account I've long used with CCC to an admin account and I was able to do it.

I was able to create disk images using Disk Utility from the non-admin account.

I don't have time or energy to debug further, but if you run into this issue try an admin account.

PS. Once upon a time CCC would just create a disk image if the task referenced one but none was found on the target drive. That's no longer an option, if you are setting up a new drive modern tasks will require the disk image to exist. It's hard to get the 'right' image manually, so you really want CCC to create it for you.

Also, by the way, and unrelated to above, you need to use AFP if you're doing network CCC backup to a sparse bundle disk image.

Updating macOS - the paranoid approach (updated from 2008)

I wrote the first version of this on 9/13/2008. I was looking for it 11/21/2020 and decided to see if I could update it a bit in advance of going from Mojave to macOS Big Sur sometime in 2021 (I skipped Catalina entirely). I'm also experimenting with a revised publication date in Blogger.

For my own amusement I kept the original below. Here's what I do now for macOS updates:

1. Make a fresh Aperture backup (still using it!) from within Aperture.
2. Test both my Carbon Copy Cloner and Time Machine backups including a test file restore. I create two CCC clones and take one off-site. I don't usually make bootable clones but I do this time.
3. Remove my backup drives.
4. Disconnect everything.
5. Reset SMC, reset NVRAM.
6. Run hardware diagnostics, Onyx cleanup, and Same Mode boot.
7. Turn off Time Machine backup.
8. Update OS.
9. Login to each user account on the machine and get iCloud working, check that Google services are connected (Mail, etc), run Notes, Contacts, Mail, etc.
10. Do backup to fresh carbon copy cloner drive. Note Time Machine is still off.

After a day or so I have to decide what to do with Time Machine. I usually start over with a fresh TM backup rather than try to continue with the old one. My primary backup is CCC.

The original post from 9/13/2008

I prefer an OS update to petting a rabid wolf or getting a kidney transplant. Even so, they're not my favorite things. It was clear early on that Apple had botched 10.5 - a prelude to the MobileMe fiasco and the iPhone OS 2.0 fender bender. So I waited to 10.5.3 before updating our non-critical MacBook. That didn't go all that well; I'm still having problems.

On the other hand there's a lot to like about 10.5, and I imagine I know what to watch for.

After lots of experience with 10.5.4, I was read to risk an update to very important machine -- our trouble free PPC iMac G5 running10.4.11.

It's the procedure I follow with all major OS updates. I do all the work through my admin account.

1. Check the backups are working so I know I have a current backup of data. I like to do a test restore of randomly selected file.
2. Have another machine available in case the update runs into problems -- you may need Google.
3. Don't do the OS update on a desktop machine during bad weather. This is a bad time to have a power failure. Make sure you can't accidentally pull a plug or turn off the power. (I once bricked a peripheral by hitting a power switch with my foot.)
4. Do a safe boot to clean up the system and verify the drive.
5. Disconnect all USB hubs and all firewire devices. Attach only an Apple keyboard and an Apple mouse.
6. Pull the network cable (see below). You can plug it in when you need to get software updates. Nowadays there are all sorts of things a partly updated machine can destroy if it can get a the net.
7. Restart then remove Preference Panes from admin account (ctrl-click then delete in preference view). Review and remove suspicious login items. Use Spotlight to find all apps or utilities with a date prior to 2004 - remove any that aren't needed.
8. Uninstall known bad actors. I know, for example, that my copy of Missing Sync for Palm OS won't work with 10.5. I don't need it any more, so time to use the uninstaller. Remove Retrospect's client if present, that will need to be reinstalled.
9. Turn off sync services, such as Spanning Sync. Don't turn them on again until you've run iCal, Address Book, mail, etc for the first time. I recommend turning off everything related to synchronization, including .Mac/MobileMe, anything in iTunes, any add-on services. To be extra sure, pull the network cable durign the update. Don't allow the machine to access the net without your control.
10. I've already removed the evil Adobe Acrobat Reader and RealAudio.
11. Copy the 10.5.4 Combo Updater to the desktop. I don't want to run 10.5.2 a moment longer than necessary. Confirm I have plenty of free drive space left.
12. Review Mac OS X 10.4, 10.5: About installation options so I don't miss the 'Archive and Install' option [1] . (Made that mistake before!)
13. Insert DVD and click the install button.
14. Go walk the dogs, do the dishes, etc. Just the DVD verification takes an age and a half. (Yes, you can skip the verification. I prefer to let it run.) The update should proceed without any questions, so you can let it go.
15. After the upgrade and reboot it can take a long time for the admin account to come up. Be patient.
16. Restart again (to let caches be build properly) then apply the 10.5.4 compo updater. The machine will restart.
17. Check all login items for all users. There's a bug in the 10.5.2 Archive and Install procedure that can cause login items to be applied across user accounts.
18. Check for other updates. I was surprised I had to install iTunes 8 again -- it had been installed earlier. I imagine if I hadn't done this, and I'd tried to sync to my iPhone, the heavens would have fallen. You have to keep checking until no new updates are found.
19. Run iCal and Address Book. Anyone else notice that 10.5 Address Book backup is under the export/archive menu now? Back 'em both up before any iPhones sync.
20. Enable Spanning Sync and do an iCal sync with gCal.
21. Run Keychain Access and Keychain First Aid.
22. Run any app that iTunes works with or that intersects with the iPhone.
23. Cycle through all accounts, looking for obvious trouble.
24. Hook up the peripherals, download drivers for the MacAlly keyboard, etc etc.
25. Expect Spotlight to suck CPU and drive the fan until the search indices are rebuilt. Let it run overnight.
26. The long recovery begins.

There were a few curious things about this update:

1. MobileMe didn't appear in software update, so it was only when I went to the old .Mac preference panel that I was asked to update to MobileMe. This might have caused some problems if I'd installed MobileMe.
2. iTunes regressed to an earlier version. I had to update to iTunes 8 again. This would have caused serious problems if I'd missed this.
3. Spanning Sync keeps telling me its deleting appointments from Google Calendar, but it doesn't say what it's deleting. I don't know why this is happening.
4. The update resurrected a number of old apps and login items that I thought were long gone. They're reaking havoc on my syncs.

[1] Select this option if you want to install a "fresh" system on your computer. This type of installation moves existing System files to a folder named Previous System, then installs a new copy of Mac OS X. You cannot start up your computer using the Previous System folder. Archive and Install installations require the largest amount of available disk space because you need to have room to preserve your existing System and the new one you are installing. This is a good choice if you've already backed up your important files and are trying to resolve an existing issue. Mac OS X-installed applications, such as Address Book and Safari, are archived, and new versions are installed in the Applications folder. Some applications, plug-ins, and other software may have to be reinstalled after an “Archive and Install.” Fonts that were installed in the Fonts folder in the top-level Library folder can be installed in your new system by copying them from the Previous System folder.

iCloud backup and my lost authenticator codes

When my local Apple store tech was unable to remove the battery from my iPhone 8 they gave me a new device -- which was SIM locked to AT&T.

Well, everyone has to start somewhere, including Apple techs. Hope they improve soon.

Anyway, between the initial restore and the factory reset to clear the SIM lock I've been through two iCloud restores in the past week.

iCloud restores kind of suck now. I think they worked better a few years ago. The good news is that my photos were restored (I don't use Apple Photos/iCloud so I needed that backup). The bad news is that so many apps needed credentials reentered or new certificates generated -- especially when doing a restore after a hardware change.

The worst news is that Google Authenticator lost my authenticator codes. As near as I can tell they are restored from iCloud if the hardware is unchanged, but not if the hardware changes. Or maybe it's a bug. Whatever the reason, I lost 'em. 

It was suspiciously easy to regenerate Authenticator codes for my Microsoft account. Not too hard for Google either, because  they've moved to preferring an Apple-like proprietary two factor authentication mechanism. It is a bummer for Dreamhost though -- so now I'm going through support to try to recover access to my domains and web content.

It's hard to reconcile security and backup/restore. For example, Google Wallet and your biometrics (finger/face) aren't backed up either. On the other hand your Keychain credentials are in iCloud, and anyone who can get into your iPhone can read all of your passwords (try: "Hey Siri, Show me my passwords" or see Apple's hidden password manager). So your 4 digit Apple device passcode is not a great idea.

PS. I'm storing Authenticator codes in 1Password now. Which, like most small company software, has its own security concerns, not least that it would be relatively easy for China, say, to acquire the company or insert a backdoor into the source code.

Can you do a Time Machine backup to a USB flash drive (thumb drive)?

 I'd wondered if it was possible to do a Time Machine backup to a cheap Flash Drive. My daughter is going to college and probably doesn't have a great need for backup (iCloud Document/Desktop, iCloud Photo, Google Docs, etc) but I'd still like to do something.

So I wondered about a compact Flash Drive. In the twilight of the web Google couldn't find me an answer, so I ran my own test. I used an old San Disk Ultra Fit 128GB USB 3.0 Flash Drive in an old USB 2 MacBook Air running High Sierra. I formatted the Flash Drive as encrypted HPFS and let Time Machine run the backup.

It took about 4-5 hours to do the initial 80GB backup but it seemed to work fine.

I doubt these Flash Drives are super reliable, but I think this is an option. I can see taking advantage of it while traveling for example. Unfortunately her 2020 Air doesn't have an open USB slot where an Ultra Fit could live, but it could be a part of her Anker 7-1 USB-C docking station. A modern San Disk Ultra Fit USB 3.1 is $33.

Carbon Copy Cloner was quietly excluding 1Password stores from backup (Corrected: app, not data stores)

This morning's heart attack:

"CCC no longer excludes 1Password by default."

What the fork were they thinking?!

-----------

Update: OK, looks like they did this briefly in 5.1.18 and, even though their language is sadly unclear, it was the the 1Password app rather than the 1Password credentials that were not being backed up. That link lists all the files not backed up, though as of today it's not been corrected for 5.1.19.

So I still have chest pain, but not a heart attack.

Things old persons don't understand -- what happens to all those school Google Docs?

Two of our kids are ending their St Paul Public School careers. Both have a collection of Google Docs.

The school does not seem to provide any mechanism for mass reassignment of document ownership to a personal Google account. From what I can tell the school actually blocks ownership reassignment. (Ownership management is one of the several significant issues with Google’s document sharing infrastructure [1].)

So what do students do with all those documents [2]? Olds like me have no idea. They don’t just let them all evaporate … do they?

(I use CloudPull, one of my favorite macOS apps, to create a local repository. The download process converts Google “docs” to Office files. Of course there’s nothing like this for iOS.)

- fn -

[1] Only owners can truly delete an owned document, and ownership cannot be transferred for non-Google “docs”. I think all own/share privileges are at the document level, but documents may inherit some properties from their folder “container” — but not ownership. Yeah, I don’t understand this. Not sure anyone does :-).
[2] Due to some cognitive disabilities and temperaments my guys can’t answer this question… I guess I should ask my daughter …

Update: of course I just write this and today I get for all my CloudPull accounts: “CloudPull was unable to export your backups”. It turns out CloudPull had lost track of my backup directory. I don’t know why. I relinked in Preferences:Advanced and it worked again. It didn’t write anything to console when that happened. Support was great at helping me fix this.

Update 8/26/2018: I again ran into “CloudPull was unable to export your backups”, this time on my personal (36GB export) gDrive. I cleaned up some other non-active accounts, used Help force reindex, and booted into Recovery mode and ran 1st Aid (it fixed things). Then it worked.

Photos.app flailing during sync with iCloud Photo Library? Maybe it's a permissions problem.

My daughter uses Photos.app and a 50 GB iCloud Photo Library to manage her videos and images [1]. She edits on an older Air with a small SSD, in that environment Photos.app caches scaled res images and only downloads full res when editing.

In addition I run an instance of Photos.app for her that stores full res images. The Photos.app Library is stored on an external SSD that hangs off an Elgato T2 Hub attached to my beloved Air. The hub has been very reliable under El Cap and Sierra.

I have a user account for her on my drive, and in that account the external library is the Photos.app System Library. My Time Machine [3] and Carbon Copy [4] backups include that Library.

All was well under El Capitan. A few months ago I upgraded to Sierra [2]. Yesterday I decided to update her Photos.app library — only to discover I was a few months behind [5]. Her user account hadn’t been updated to Sierra; when I opened Photos.app her Library had to be updated.

Things did not go well. Photos.app said it was uploading @8,000 images (really it shouldn’t have uploaded anything, but Photos.app sucks), then @2,000, then @11, then … You get the idea. It did that when I went to bed, and it was doing it in the morning.

After a bit of playing around I discovered that a Sierra bug meant that she no longer had write permissions to the external SSD, even though macOS said she did. I switched to an admin account and there she had no permissions, so I added her. After that she could write to the SSD. Photos.app “stuck upload” was because it had no write permissions at all.

I decided to create a fresh Photos.app Library for her. To do that I turned off WiFi and did option-Photos.app startup to create a new Library. I copied the old Library to an external drive and deleted it. I then opened the new Library, made it the System Library (interestingly it showed images from a cache!), turned on WiFi and enabled iCloud Photo Library. The images then downloaded from iCloud (source of truth) and restored my local backup copy.

- fn -

[1] She is chronically running against the limit — which isn’t all bad. It enforces some editing. I might switch to sharing a 200GB plan, but I’m not sure how that will work with our current family use of a single iTunes password. Future experiment needed.

[2] I like to wait at least 8 months before accepting Apple’s dangerously buggy macOS updates.

[3] Our two Airs do Time Machine backups up to a Synology NAS. After some initial issues that has been utterly trouble-free. The NAS has two RAID 1 drives, if one fails the other survives. This is another reason I wait for macOS bugs to get fixed; I also need things like VMs and NAS to be updated.

[4] CCC backups to a 4TB low heat drive in a Voyager cradle with Firewire 800 connection to the Elgato hub. I rotate 4 drives. Rotation is every 2-3 weeks, 1 drive is across town, the other in my Van. A Yellowstone eruption would take them all out unless the van outran the pyroclastic flow. It is a shame that offsite internet backup has failed.

[5] Only automated backup ever works — and no form of backup is reliable.

Annals of iOS inconsistency: Contacts vs Notes vs Reminders - backup and sharing

https://www.icloud.com/#settings currently shows an “Advanced” subsection for restoring Contacts. It provides options to restore an iCloud data set “archive” from iCloud (not to be confused with restoring an entire iOS device backup):

Notes aren’t on the list though. They have their own note-specific backup restore option, but it’s at the level of an individual note and there’s no version restore, only the ability to undo a delete for 30 days by restoring a Note from “recently deleted”. (BTW, if you Share a Note only the Owner can “delete” — but anyone with Edit privileges can remove all content — and since there’s no version undo that means anyone who can edit a Note can delete it without a recovery option.)

Sharing is another area of odd inconsistency. Notes must be shared one at time, but multiple Reminders can belong to a set of People.

I’d like to see Notes add Google-style Note-specific version save/restore and share by container (folder) as well as Note, but there’s no rumor of that in iOS 11. I’d pay for a third party solution for iCloud, similar to what CloudPull does for Google App docs, but I fear the demand is too small (for example). An Apple iCloud Drive folder view of Notes [1] would be a big help; I’d then be able to restore an individual Note from a Time Machine or Carbon Copy Cloner backup …

Anyone have an AppleScript to create a local daily snapshot of Notes? (There is this, but in Sierra Apple omitted AppleScript dictionary support for PDF creation).

The world moves in unexpected ways. We seem to be converging on a form of backup that’s a regression for people like me, but a big improvement for most. There’s probably some kind of futurist principle there — the good-enough mass solution will drive out the elite ideal …

- fn -

[1] The main reason I’m still on Simplenote is that nvAlt on my Mac maintains a synchronized file store that works just like this. Perfect data freedom — but almost nobody appreciates this …

[2] As of Sierra at least some parts of Notes are in /Users/[username]/Library/Group Containers/group.com.apple.notes. This location has changed a few times. Note content is distributed between media files (PDF, etc) and text in a sqlite database, so recreating an individual Note document as, say, an RTF file, is a non-trivial task. For example (sqlite browser):

I suppose Time Machine backups of this folder might be a kind of ‘restore all notes’ option, but restoring a version of an individual Note would be tricky…. (There’s something deep here about the ways in which we assemble bits to create something our brains perceive and our tools manipulate, but it’s beyond my ken. Once upon a time a BYTE article would have traced the roots of the Notes sqlite store back to database file systems of the 1980s…)

Apple drives me into Google's arms - using Google Photos with iPhone and Aperture

Apple has broken me. I’ve left iCloud Photo Stream shares for Google Photos.

First I lost the ability to share from Aperture to Facebook. I think that was probably a Facebook change, but of course Aperture isn’t getting updates any more.

That was annoying.

Losing Apple Photo Stream was much worse. Photo stream wasn’t great, but it was simple for my daughter, sister, and other users to subscribe to. For a time I could use iCloud Photo Library on Photos.app alongside iCloud Photo Streams on Aperture [1].

Then Aperture retched and I lost my shared photo streams (but not, happily, the originals). I played around with restoring iLifeAssetManagement from backup but, despite early promise, I couldn’t defeat Apple’s black box sync infrastructure [2].

That’s it. I’m toast. I surrender. Google’s inexplicable aversion to album creation on upload is the lesser evil now.

I’ve installed Google Photos on my iPhone and enabled backup and sync. I’ll use that to cull and play with photos before I transfer them to Aperture.

I’ve freed up 14GB from my Air’s SSD by deleting iLifeAssetManagement and I’ve installed Google Photos Uploader.app. I pointed that to a folder on an external drive, when I want to share from Aperture I export there for upload. I do my post-upload organization and sharing through the web UI.

Since Google nicely migrated images when it closed Picasa Web Albums my new shares are reunited with my old Picasa web albums. I’ve come home again. Though I’m still puzzled by Google’s weird album aversion.

It’s far from ideal, but Apple has burned me yet again. They seem to despise my data.

[1] Though I gave up on iCloud Photo Library when I realized it was more or less incompatible with importing images from iPhone photo roll to Aperture.

[2] Apple is famous for sync that disallows any kind of troubleshooting.

iLifeAssetManagement

Partial restoration of lost Apple iCloud photo stream shared albums (updated: didn't work)

Something went wrong. It always does.

I had thousands of images distributed across over 60 shared photo streams. One day I rebuilt Aperture’s database and all the iCloud images were in one recovery folder. I deleted them and then most of my iCloud shared albums vanished.

This is a quick summary of how I recovered most of them from backups. I don’t know how this truly works, but it seems that this folder in my user account was a source of truth for iCloud photo streams:

/Users/[my user name]/Library/Application Support/iLifeAssetManagement

I copied what was there to an external drive then deleted it, logged out (necessary to close open databases) then logged in. With Wifi on when I launched Aperture it showed no images at first then downloaded what was in iCloud. So there was some kind of sync.

Next I did the same thing (closed Aperture, deleted, etc) but this time copied a backup of iLifeAssetManagement from prior to the bad event. I then turned off wifi.

On relaunch Aperture showed about 6100 images in “Shared:iCloud”. It rebuilt thumbnails for them. Then I turned on Wifi. Next I saw the count rise briefly as albums I’d shared previously came down from iCloud. Alas, the count started falling again, stabilizing at 5600.

I had most of my streams back — though one stream was much smaller than it used to be. Still, about 80% recovery and I didn’t lose a few I’d done post-disaster.

Better than nothing.

Sync without controls is truly hell (and Apple never provides enough control).

Update: Aperture shows 56 single owner photo streams (one is empty) and 5 shared. iOS Photos.app shows 20. At least one of the iCloud albums not seen in iOS photos.app cannot be found at its public link. The iCloud library and the Aperture iCloud library are not in sync. So I’d call this a failure.

There may be a fatal flaw in my backups. (actually, no)

I’m leaving this one up as a reminder of how scary the world of secure backups is, and how important it is to actually do a dry run of a disaster recovery scenario.

This is the original post. It’s wrong:

Don’t every tell me backup is a solved problem.

I have offsite backups of my data. Two offsite and two onsite Carbon Copy clones that I rotate. In addition to my onsite Time Machine backups.

All encrypted of course, because otherwise that would be terrible.

Great. All set. If the house burns down we’ll have our data (assuming we still need it).

Except those drives are whole drive encrypted with FileValue 2. So each has a unique recovery key. A recovery key that is different form each backup drive and can only be known at the time of encryption. A recovery key that is stored in a keychain on my MacBook. A device that can be lost.

I’d be better off if that recovery key were in iCloud, but I don’t think it is. Or I could follow Apple’s complex directions for managed recovery keys. Or I could have created encrypted sparse image folders for CCC, I’d know the image password then. Or maybe created bootable encrypted disk backups.

I have a bad feeling I don’t really have backups at all.

There’s a fine line between security that makes data inaccessible to bad actors and security that makes it inaccessible to everyone.

I hope I am wrong about this.

It’s wrong because FileValue 2 whole drive encryption actually behaves like the disk image encryption I’m familiar with. I was confused by the Recovery Key complexity. Doing a dry run of disaster recovery shows what happens.

I mounted one of my encrypted backups using my Voyager cradle and a USB 3 to UBS 2 cable with an old Air. I was asked for the password I’d used to encrypt the drive, not for the recovery key. I was able to mount my backups just as I would on any foreign Mac.

That password is the same for all my backup images and it’s stored in 1Password as well as printed. I’m going to add it to the Dead Man / post-mortem document I keep in Google Drive that’s shared with several trusted people.

False alarm. Need more coffee.

See also

• The curse of iTunes encrypted iPhone / iOS backups: the password is NOT your device password (and how to recover it from OS X Keychain) 8/2013. Another surprise.

How to backup your iCloud Notes.app data on a Mac

Apple has a support doc on backing up iCloud data with a less than ideal recommendation for macOS Notes.app backup …

Archive or make copies of your iCloud data - Apple Support

To copy notes, open the Notes app at iCloud.com. Copy the text of each note and paste it into a document on your computer, such as a Pages or TextEdit document. Save the document to your computer. To export your notes as PDF, open the Notes app in OS X Mountain Lion or later. Select the note, then click File > Export as PDF and choose a location.

Maybe Automator could make this scale, but I’ve never had much luck with Automator.

There’s a better approach that, oddly, Apple used to recommend …

How do you backup Notes? | Official Apple Support Communities

Open Notes.
Select View > Show Folders.
Create a new folder called Notes Backup in the On My Mac section of your folders list.
Select one or more notes from your All iCloud folder. Holding the Option key down, drag the notes into the Notes Backup folder. A green plus icon should appear as you're dragging the Notes to the new folder. This creates a copy of your iCloud Notes on your computer.
If you have an iCloud section in your folders list, but not an On My Mac section, you will need to create one.
Quit Notes.
In System Preferences, select iCloud. Deselect Notes.
Open Notes. Select File > New Folder. Name your folder Notes Backup. Create a new note in that folder as a placeholder. Quit Notes.
In System Preferences, select iCloud. Select Notes.
Open Notes. You should now see a section for iCloud and a section for On My Mac. Follow the instructions above for making a local backup of your iCloud Notes.

For this to work you need to enable the “On my Mac Account”. That may be disabled on most Macs and I suspect Apple would like to get rid of it. Which is perhaps why it’s no longer part of the backup support doc. In any case this does work on El Capitan, it does scale, and it enables restore.

If you ask Google how to backup Notes.app, the AI presents this as a “pre-answer” above all web page results. Google’s AI does better than Apple’s support documentation team.

Apple needs a better backup/restore solutions for all of its iCloud data, especially Notes.app, but for now this helps.

Beware: iPhones with mixed FairPlay app ownership may no longer backup/restore as expected

Ran into this with a friend’s devices. He was syncing his and his wife’s iPhones to iTunes under Lion. Not sure how old iTunes library was.

When his wife’s iPhone 6s had to be replaced he found he couldn’t restore from iCloud backup or from his old iTunes backup (Lion OS, I didn’t check his iTunes version). He tried to create a new iCloud backup but that removed  the old one and created a 0kb backup. I couldn’t get iCloud backup working either.

Probably relevant: they had mixed app ownership, some purchased under his Apple store ID, some under her Apple store ID.

I created a backup to a fresh user account iTunes instance on my El Cap machine (but, significantly, my machine is authorized on my App Store ID). Restore seemed to proceed — but app restore didn’t complete from the iTunes backup. Instead it slowly proceeded via iCloud then left apps in dim icon status.

I believe I got things working by signing them up to Family Sharing. Then apps downloaded from iCloud, but not from iTunes.

At the time I write this I’m still investigating media rights and have yet to attempt another iCloud backup.

I wonder if Apple quietly updated its backend DRM rules.

DRM is nasty stuff.

Update:

1. The iCloud backup problem was an unrelated bug. “The Last Backup Could Not Be Completed” bug has many causes, but sometimes it’s just the wrong error message. Instead of saying “buy more storage” (which is what I usually see) iCloud Backup gave a useless error message and quit. I excluded her 5.1GB of images and it resumed.
2. Her new phone has no media. I think I know what happened. She gets her media from a non-DRMd old school iTunes music library. If you backup, as we did, from her phone to a Library on my computer there’s a DRM conflict. Apple’s iTunes DRM is device based, not (AFAIK) user account based. So my Mac wasn’t authenticated to hold her music (maybe I could authenticate, but that’s scary). I think the music got backed up, but iTunes could’t display/handle it, so it wasn’t part of the restore. The bug here is absence of an error message, simply media free restore.

Simplenote is not dead -- and the joy of nvAlt backup

Simplenote.app, an Automattic product I use a zillion times a day, is less dead than I thought. They just released a version for Android, I installed on my ultra-cheap Moto e and in the blink of an eye my notes are there.

Before I did that experiment though, I made a backup.

I launched nvAlt and my local Mac folder of Simplenote RTF files was instantly updated. I then zipped up that folder — maybe 2MB. Stuck the zip in a folder of things like that. A record of the state of my extended memory on this day.

Only a geek can understand the warm glow I get from that special level of backup. The age old problem of Cloud backup (how do you recover a single mis-edited note from a month ago?) solved. (But will nvAlt work on Sierra? Brett Terpstra’s long delayed nvAlt replacement drops Simplenote support.)

Now if only Automatic would fix the #$!%%! broken search on (only) the Mac version. I confirmed search works on the new Android version.

See also

• Beyond Simplenote: I still want a graph layer (concept map) overlay for my memory augmentation notes collection. 5/2016. Why don’t billionaires fund projects like this? More useful than most ego projects.
• Cloud data: Should I trust (Simplenote) Imperium? 8/2010. I don’t trust Automatic either, but with local file stores I don’t need to.

Life with Satan's Own Backup: Synology Time Machine Backup goes bad

A few days after my carefully delayed update from Yosemite to El Capitan, Emily’s MacBook Air complained it couldn’t run its Time Machine backups to our Synology DS215j NAS. I can’t say if this was related to El Capitan or not, but I do wonder about El Cap’s relative deprecation of AFP (which Synology/Time Capsule needs).

The usual approach to this problem is to use the Synology web interface to run “File Station” and delete the .sparsebundle from there (don’t even think of trying to do this through the Mac Finder) then start over.

Instead I ran through a checkup of the NAS. I didn’t find anything — but did update my notes on the confusing configuration (every machine has a dedicated Synology username and quota). I tried various things to mount the old sparse bundle including disabling SMB, etc. Nothing worked, I couldn’t make use of the existing backup.

So then I installed a (long!) physical ethernet cable and tried Finder (AF) copying the .sparsebundle to a local drive so I could browse it efficiently. Every time I tried it quit at about 54GB with a message like “The operation can’t be completed because you don’t have permission to access some of the items.” A different 60GB file copied normally.

I figured that was a bogus error message from a corrupt file, so I used Synology to copy the 200GB directory internally. It copied well, no sign of a file system problem. I deleted the original and I was able to Finder copy the Synology copy to my local machine. BackupLoupe could even browse it, because, you know. you can’t use Time Machine to access a backup unless you’re on the original machine (which is why it’s Satan’s own backup).

All very occult.  It’s probably something to do with extended attributes, AFP, and some kind of El Capitan bug.

Oh. And I started over. Of course.

Update 8/29/2020: Backup also failed when I upgraded the old Air from Sierra to High Sierra. Good thing I have alternatives to this TM backup.

See also:

• How to update Synology Cloud Station Server Clients (Cloud Station Drive) after Synology NAS 6.0 update
• Synology Time Machine backups: How to increase a user quota
• Replacing a Time Capsule with a Synology NAS - Time Machine for multiple machines
• Time Capsule & Time Machine: "Browse Other Backup Disks" doesn't let you access backups from a different device

Synology Time Machine backups: How to increase a user quota

After my Synology NAS updated itself to version 6 one of my Time Machine backups stopped working. It might have been coincidental. Time Machine claimed I only had 350GB free and it needed 1TB, but Synology claimed I had enough free space.

Whatever.

The fix was to increase the quota size for the user who owned the Time Machine disk image belonging to my MacBook Air.

Except I couldn’t do it. I could edit the user easily, but the quota information couldn’t be edited. Clicking on the row did nothing. 

Click-click.

Google helped. It’s a UI issue. There’s nothing in Synology’s UI to tell you to click specifically on the quota number. If you do that you can edit it.

It’s probably a good idea to turn Time Machine backup off while you’re doing this. In any case it’s fixed my problem.

Recovering selected OS X Mail.app messages from a carbon copy cloner backup

Somehow I’d enabled my iCloud Mail account in Yosemite Mail.app. This is irksome, because I don’t use it and, sooner or later, I’d somehow send outgoing emails using that sending service.

Which I did. Somehow. 

Which meant I had a set of Sent emails in the wrong place. Impulsively, in Mail.app, I tried dragging them from iCloud Sent folder to Gmail Sent folder. Something was happening … the dialog “Outgoin Messages”. Yikes! I don’t want them to be resent! So I canceled. (I was wrong, this is what Mail.app does when one is copying from a local store to an IMAP folder.)

All the emails vanished everywhere (I think this is a bug). There was nothing in iCloud Mail on web or OS X and nothing in Gmail — or anywhere else. So annoying — especially because I think I made the same mistake years ago.

I could live without those 49 errant emails, but it’s vexing to lose data. So I turned off internet connectivity and dug into last night’s Carbon Copy Cloner backup (Time Machine restores are so painfully slow I use them as a last resort, and I’m not sure how they behave for attempts to restore a specific set of emails.) The User/Library folder is hidden, but Go To Folder works if you know the path. On my system it’s User\Library\Mail. The actual folder varies by the age of one’s system, my messages are in a V2 folder, not a V3 folder. (I saw lots of weird stuff in there, like a [Mail Lost+Found] folder with old .mbox files. I need to dig through those. OS X Mail is just weird.)

The folder names were odd. The one called IMAP-…. visi.com has my current Gmail data, but I haven’t used visi.com for 10-15 years. Apparently OS X kept the original account name from my 10.3 days. Have I mentioned OS X Mail.app Preferences are kinda horked?

 

The missing iCloud message were in AosIMAP (wonder what Aos stands for?) in a Sent Messages.mbox folder. I copied that to the desktop and used Mail.app Import to bring them in. Despite the .mbox extension I had to use the Mail.app importer, not the .mbox importer.

All the messages were recovered. I found I could copy them safely into the Yosemite Mail.app “All Mail” folder that corresponds to Google’s All Mail, so they were back on Google for search purposes while also available on Mail.app (yes, Mail.app/Gmail integration is scary, mostly Google’s issue).

Sometime I’m going to dig around Mail Lost+Found to see if it really contains old lost messages. I like to keep my data; I’ve got Eudora and Netscape emails in OS X Mail.app from the early 90s.

Update: Most of the .mbox folders in “Mail Lost+Found” were very small. The few that were larger just duplicated messages I already had. So I cleaned out that folder.

Time Capsule & Time Machine: "Browse Other Backup Disks" doesn't let you access backups from a different device

One day your iMac dies. It’s old, but not old-old. Sucks. Good thing you are paranoid about backups. You have onsite backups. You have offsite backups. You have Time Capsule backups. You have Synology NAS backups. You have Carbon Copy Cloner “Backups” (clones). You have …

Ok. I’ve made my point. Anyone this paranoid ought to feel good. Problem is, they’re paranoid for a reason. Data just wants to die.

The “you” is “me” and I’m here to tell you that one small bit of my data almost didn’t make it. One folder full of almost-deleted images got lost, I had to pick it up from a last minute copy of the iMac’s user folder. 

I had to do that because when I tried Time Machine’s “Browse Other Backup Disks…” feature (option key)  …

… it didn’t actually work. That is, I got the right list of disks ...

but when I selected one of them Time Machine showed me only data from my current Device’s current state — and no past data.

I did this first using a Synology NAS backup replacement for my died-young Time Capsule. I thought I’d run into a Synology limit, but I got the same results from older Time Capsule backups. It turns out that “Browse Other Backup Disks” really means “Browse Other Backup Disks … for the current device”…

 Yeah, I hate Time Machine too. OS X Help has some entries on Time Machine, but there’s no real documentation. There’s nothing on “browse other backup disks”.

So, if you don’t have access to your original mac, you are sort of doomed. That’s what happened to me.

I say “sort of” because there are weak options. You can open the disk image and navigate Time Machine’s base storage. You don’t have access to the File System Event Store or hard links though, so things are hard to locate. EasyFind.app might help. Or you can use Migration Assistant, the official solution, and move large pieces of the backup to a local store (only most current versions of course). Maybe OS X Server has some special options …

You can also try Backup Loupe ($10). It doesn’t replace Time Machine’s time-slice views of data, but it does let you browse snapshots and search for file instances. I’m not sure it’s a big improvement on EasyFind, but I bought a copy for emergency use.

The bottom line? Time Machine is a sucky backup solution — just good enough to eliminate strong alternatives. But you knew that. If you don’t have a machine (Device) that “owns” a backup you can use Migration Assistant to copy the latest state of a large amount of data, or if you know a file name you can use EasyFind or Backup Loupe to browse.

Sure, Apple should fix this. They should fix a lot of things.

"Unable to contact iMessage server": try restoring from iCloud instead of iTunes

I picked up Emily’s SIM-Free [1] 64GB silver 6s from the Mall of America Apple store Friday night. I’d used Apple’s reservation system so that, in theory, I’d be in and out. Alas, Friday night at the Apple Store is a zoo — it still took 30 minutes. The staff were so stressed they didn’t try to up-sell AppleCare or setup a contract — just dropped the box in my hand and ran.

There’s an AT&T store in the MOA and it’s not incredibly busy, so we did our SIM swaps there [2]. My son was going from a 4s to Emily’s 5s, so he needed a new SIM.

I restored both phones from iTunes backups. Emily’s worked, though it was a bit choppy. I had to unlock the phone 1-2 times as it went from 9.0.x to 9.1. 

My son’s restore didn’t work. I completely erased the 5s before starting, but there was still an odd feeling about the way the restore proceeded, perhaps because the 5s was still on 8.x (I didn’t realize it had never been updated).  Yes “odd feeling” isn’t very helpful, but I wasn’t paying that much attention. I’ve been down this road a few times.

Prior to the backup I’d removed iCloud, iMessage and FaceTime from his account, planning to put them on post-restore. I had some trouble restoring iCloud — the phone hung on credential entry. I restarted and it seemed to work — but then iMessage and FaceTime weren’t activated. When I enabled them I got a very cramped non-iOS 9 dialog for entering username and password.

I’ve seen that dialog before. It’s something very old — I suspect it’s hard coded for non-retina screens and dates back to the dawn of the iPhone, pre-iCloud. It’s a bad sign, it exposes Apple’s still broken iOS credential management problems [3]. When I did enter my son’s credentials the dialog hung, waiting for a response. I could kill settings; iOS wasn’t frozen. I let it sit for 15 minutes and it eventually responded with something like “Unable to contact iMessage server”. I don’t think there’s a problem with the iMessage server, I think that’s a misleading error message meaning “something went wrong”.

I called AT&T phone support to confirm the IMEI/ICCID relationship was correct at their end. I’ve had my issues with AT&T, but they must give their support staff very good coffee. They are remarkably pleasant and helpful. AT&T’s configuration looked good.

So either the phone was having hardware issues or something had gone wrong with updating one or more of Apple’s configuration systems. There’s lots of evidence that Apple wants iTunes to “die in a hole”, so I decided to try it Apple’s way. I did an iCloud backup, wiped the phone, and restarted with an iCloud restore.

That went smoothly. During the restore I had my son’s Mac account open for Keychain share confirmation, and I got the usual “FaceTime is using..” dialogs. I didn’t have to enter any extra credentials. iMessage and FaceTime activated immediately.

I suspect the combination of iTunes and iOS 8 to 9 and my removing FaceTime/iMessage/iCloud prior to backup exposed a nasty bug in Apple’s frail authentication systems. The real lesson though is that iTunes backup is seriously deprecated. I’d been moving to all iCloud backup and just doing a manual backup to iTunes every few weeks; that’s obviously the way to go.

- fn -

[1] We are currently AT&T customers, and there’s a case to be made that an unlocked AT&T 6s has the best set of antennae and band coverage for AT&T and even international use. You can’t, however, buy an unlocked AT&T iPhone directly, you have to buy it on plan then pay the plan cost to unlock it. Our AMEX purchase protection and extended warranty only work when the full purchase price is on the card. Hence SIM-Free.

[2] In theory you can move a compatible AT&T SIM from phone to phone yourself, but in practice I’ve seen some odd things. AT&T reps tell me their systems don’t update the ICCID (SIM)/IMEI relationships automatically, or at least not immediately. I think this causes some iMessage/Facetime activation delays.

[3] There are separate credential stores for iMessage, FaceTime, iCloud and the App Store — and perhaps for 1-2 other items. If you migrated from me.com to iCloud.com some of these systems require two sets of credentials. Apple tries to hide this from users, but any number of bugs will expose it.

[4] To fit into the iCloud 5GB limit I routinely delete obsolete backups of old phones and I move Photos.app data to our local machines. I see that with 9.1 there are now more controls on what’s part of an iCloud backup, though they are a bit hard to find.

File sharing for the all-MacBook home

Lifehacker’s guide to home file sharing was written in 2010 for Windows users. Excluding a traditional server/file share the options back then were Dropbox, a NAS, and, peer-to-peer sync solutions. Things haven’t changed much since then.

Now that I’ve retired our iMac and gone all-MacBook, I need one of those solutions for a small number of files (MBs, not even 1 GB). Our home’s options are Dropbox, Google Drive, Microsoft’s OneDrive, a Synology NAS with or without Synology Cloud Station, Mac LAN based sync solutions (ex: ChronoSync, note MSFT bundles this with Windows), and an Airport Extreme external flash drive.

There are lots of options, but nothing is quite perfect. Dropbox, Google Drive and OneDrive all move our family data into the Cloud — and I’d like to not worry about that. Sync solutions mean new software, but perhaps only on one machine.

I’m going to stick our unused $20 SanDisk Ultra Fit 64GB flash drive in back of the Airport Extreme. I already use Carbon Copy Cloner as part of our nightly backup, I’ll just back the AE Flash Drive up to disk image on one of the my OWC Thunderbolt 2 dock drives. They in turn are backed up by both CCC (to removable drives) and Time Machine (to the Synology NAS).

That should be good enough. Keep it as simple as possible…

Update: oops. "When you use Airport Utility to change AirPort Extreme Shared Disk(s) security it *seems* to wipe out everything on the disk. Except free space shows data is still there.”  The AE has an operating system with some kind of file system support and access controls, but we have very limited access to it.

This Apple article partly explains what is supposed to happen. From Airport Utility we can create username/password “accounts”. Say “Parent” and “Kids”. When a client connects you are asked username/password, that gives access to the Folder of the same name as well as a “Shared” folder. So Emily and I connect as “Parents” and see the “Parents”  and “Shared” folder, but we don’t see a “Kids” folder unless we connect with that username password.

There’s no way for me to connect with to the AE shared disk (partitions?) and see everything.

When I insert the flash drive into my MacBook I can see how it’s organized, including the folders that were on the flash drive when it was “password” access rather than “account” access.

When I switched “Secure Shared Disks” from “With a disk password” to “With accounts” it didn’t wipe my data, it created a Users folder containing the “Parents” folder and hid the existing folders. I thought I also created a Kids user, but I don’t see that Folder. Bug?

Hmm. This is a bit weird. I could experiment with partitioning the thumb drive on my Mac, but I think I need to look more at the Synology.  The AE’s file sharing security model seems to make backup impossible.

Update 2: I’ll rewrite this when I finalize things, but it looks like the Synology NAS gives me the permission controls I need. I’ll put the shared files there, then use CCC to put them back on an image on my laptop. That image will in turn go back to the Synology NAS Time Machine backup as well as to my local CCC backups.

Update 11/21/2015: I ended up enabling Synology Cloud Station, including installing the Mac client for both Emily and I. So our relatively small (1.5GB) of shared data exists on the Synology NAS (not baked up) and on both of our machines (so multiple backups). It is a strange outcome for the old file sharing/NFS/WebDav model and it doesn’t seem the most elegant solution, but sync seems to be the current technology direction. (Dropbox would be simpler, but we wanted to keep the data local and, of course, Dropbox costs money. The Synology NAS also supports a BitTorrent sync package but the Cloud Station seemed to have more users.

Configuration was a bit odd — you do need to read the documentation. The default setup is within one’s “Homes” folder, so if you want to share with two users (workgroup) you need to create a folder outside that NAS hierarchy and choose to that for sync.

Update 8/23/2016: Synology Cloud Station / Cloud Drive (it has many names) has stopped working reliably with El Capitan. I’ve given up on it. Emily’s MacBook is largely home so I’m moving these files to her machine and making them a file share. Sometimes I won’t have access, but I’ll move some things to a Google Drive we share.