Big Sur bug: Mail Search (corespotlightd) fails on multi-user machine after a user logs out

We are indebted to GanawaGangunawa for figuring out why Mail search was failing on Emily's M1 MacBook Air (known to our family as "Crashy" [1]). It's a Big Sur bug (though I think it happened in some Catalina environments) that hits multi-user machines.

In our case Ben and Emily both have non-admin accounts on her M1 Air running Big Sur 11.2.1 with fast user switching enabled. When Ben logs out Emily's Mail search stops working. There's no error message, but search does nothing and Smart Folders are inactive.

The fix is to kill corespotlightd.

I created an AppleScript with the contents: 

do shell script "killall -9 corespotlightd"

I saved it as an application and put it on Emily's dock. Two clicks fixes her Mail search until Apple fixes the bug.

[1] When we first got the M1 Air it crashed (spontaneous restart) every few hours. Reinstalling Big Sur meant it crashed every few days, with Big Sur 11.2 it didn't crash, with 11.2.1 it restarts every week or so. I suspect a firmware/OS mismatch in the factory was the initial problem and that for the rest that Big Sur/M1 are not quite stable yet. I almost returned Crashy in the 2 week return window but it seems just stable enough. Good chance future OS updates will fix. It does pass hardware test.

Fixing the Mojave Mail split view in full screen bug

For many users Mojave email will periodically open in Split View mode even then Mail Preferences: General split view is unchecked.

I'm trying this fix:

1. Check Mail Preferences:General "Prefer opening messages in split view when in full screen". Confirm Mail opens in split view. Maybe quit and restart.
2. Now open mail preferences and uncheck that option and force quit Mail. (Somewhere in the Mojave era or earlier macOS preferences got wonky such that an app on exiting could do weird and occult things to preferences.)
3. Restart mail with Safari open in full screen and confirm you don't get the Split View -- get Mail as full screen.

Works so far but I wouldn't be surprised if fix doesn't last.

Getting photos from iPhone to Lightroom, Aperture and other non-Photos products

Apple's approach to photography is to keep everything in iCloud and to view or edit the images from a macOS or iOS device running Photos.app (there's also some limited web browser access). There's some limited ability to share albums between family members and other groups, but this has been a mess for years and I'm not sure what parts of it are available in which versions of macOS.

This approach doesn't work very well if you want to mange your photo Library outside of Photos.app. How do you keep track of what you've have reviewed and exported within Photos.app?

I can think of two ways to do this. One is to apply a Keyword to every image that's exported, then use a Smart Album to track photos that are not keyword tagged. The other, which I do for now, is to periodically review everything in iCloud Photos.app, then export everything from there (so it's empty), then import into my photo repository of choice (still Aperture, but Lightroom and others would work the same way. I think one could even use a non-syncing Photos.app library. 

In other words, I use Photos just as a queue for export. If I want to view images on my phone I export from Aperture to Google Photos (for now).

I don't have a great way to review and track photos from family member iCloud libraries. I will probably apply a keyword tag to those I export to reduce duplicate exports.

PS. Photos.app on iOS used to have a special album that identified images that were on-device. That worked well with Image Capture or Aperture Import; I'd delete those images after importing them. These days everything is in iCloud, so I no longer import from the iPhone, I view and export from Photos.app on my Mac. Then I import into Aperture.

Put display to sleep on Mac with Touch ID: ctrl-shift-power replaced by hot corner

I use Ctrl-Shift-Power to put my MacBook Air display to sleep each night. That doesn't work on a Mac with Touch ID. I believe the only option is to use a hot corner and choose display sleep.

Don't blame your hub when your USB Flash (thumb) drive disconnects from your Mac

I tried using a SanDisk USB Flash ("thumb") drive as an alternative Time Machine backup device on my daughter's 2020 Intel MacBook Air (Catalina) and my 2020 Apple Silicon MacBook Air (Big Sur). It worked when directly connected to a 2011 MacBook Air (High Sierra), but when connected to an Anker USB-C hub it kept disconnecting:

I was pretty annoyed with the Anker hub and decided to return it, but then I tried it on my 2016 MacBook Air (Mojave) with a rock solid old Elgato Thunderbolt 2 Hub. The same thing happened there!

So I can't blame the Anker hub too much. I canceled my Amazon return. The bug is probably some mixture of faults in macOS, the processor in Flash drives, and some global Hub/USB flaw. 

It would be interesting to test Apple's hub-equivalent dongle -- the USB-C Digital AV multiport adapter.

Migrating from 9yo 11" High Sierra MacBook Air to 2021 Big Sur M1 Air

I replaced Emily's 8-9 yo MacBook Air 11" running High Sierra with Apple's latest (M1) Air (Big Sur). A few notes for others who might be facing migrations....

1. I used Migration Assistant over WiFi but I didn't migrate any applications. Apps have changed to much, better to install from source. Pay CLOSE ATTENTION when they tell you to write down the user passwords! (I took a photo). Migration Assistant brings over a lot of old junk but it also saves a lot of time; it's a pain to migrate mail archives without it.
2. Only 1Password needed Rosetta so far. As I write Office 365 is installing.
3. Citrix Workspace for Apple Silicon worked! That was biggest risk.
4. I couldn't get Carbon Copy Cloner email notifications working. I contacted vendor. The app works for back up though.
5. I didn't want to use the iCloud Document sync feature and Migration Assistant did preserve my High Sierra settings.
6. You need to open Photos and let it update before reenabling iCloud photo sync. There's no error message -- it just won't work.
7. For multiple users I couldn't update the User Profile Login picture from the user account, I had to do it from my Admin account in Users and Groups preferences. (Needed update for Retina images)
8. I did better skipping iCloud setup initially then doing it from each User account separately.
9. I had to reenable Fast User Switching on 1-2 accounts.

Overall I ran into a few bugs and glitches but High Sierra is 3 releases from Big Sur so that's a hard jump. Really wasn't terrible so far. 

The new Air is rather faster than the 9yo 11" Air, but not amazingly faster than my 5yo MacBook Air (Best Computer Ever Made). Most delays are waiting for servers, so local speedups don't make a ton of difference.

Google's mysterious new blogging platform

Google Blogger has been largely forgotten, but over the past 1-2 years it's been receiving regular updates.

Mostly these have been improvements with a few odd regressions. Some of the regressions have been fixed.

It's kind of curious. Google still uses Blogger for some of their blogs on googleblog.com (ex: Scholar), but they also have a new platform - https://blog.google (Keynote, Data Centers). On the Keynote blog page the RSS feed is hidden (but exists), on Data Centers and Photos blog there's a familiar feed icon top right. Data Centers articles date to 2012, but the .google domain was only registered in 2014. So they've migrated some old content, probably from Blogger.

I looked a the source from a Data Center post and it's surprisingly old school readable. There are commented out tags for handling IE 7 (!) and metadata for Open Graph and Twitter Card. Style sheets refer to "/static/blogv2/css/blog.min.css?version=4.4" />. 

I wasn't able to find any articles on "Google's new blog platform". That doesn't surprise me, Google search is fairly useless these days. Clearly they are up to something internally.

If they do make this a public blogging platform I'm sure it still won't handle paragraph spacing correctly.

Blogger will republish old posts with new dates but keep old URL

So I learned something today about Google's ancient blogger platform.

You can republish old content with new dates without breaking the URL. Today I revised a post I'd written in 2008, but I set the publication date to today's date.

The post republished with today's date and is ordered correctly on my tech blog page, but it kept the old URL embedded date: tech.kateva.org/2008/09/os-x-major-version-updates-my-approach.html.

It's probably always been that way, I just never tried republishing before.

It's something I'll do more often now.

Updating macOS - the paranoid approach (updated from 2008)

I wrote the first version of this on 9/13/2008. I was looking for it 11/21/2020 and decided to see if I could update it a bit in advance of going from Mojave to macOS Big Sur sometime in 2021 (I skipped Catalina entirely). I'm also experimenting with a revised publication date in Blogger.

For my own amusement I kept the original below. Here's what I do now for macOS updates:

1. Make a fresh Aperture backup (still using it!) from within Aperture.
2. Test both my Carbon Copy Cloner and Time Machine backups including a test file restore. I create two CCC clones and take one off-site. I don't usually make bootable clones but I do this time.
3. Remove my backup drives.
4. Disconnect everything.
5. Reset SMC, reset NVRAM.
6. Run hardware diagnostics, Onyx cleanup, and Same Mode boot.
7. Turn off Time Machine backup.
8. Update OS.
9. Login to each user account on the machine and get iCloud working, check that Google services are connected (Mail, etc), run Notes, Contacts, Mail, etc.
10. Do backup to fresh carbon copy cloner drive. Note Time Machine is still off.

After a day or so I have to decide what to do with Time Machine. I usually start over with a fresh TM backup rather than try to continue with the old one. My primary backup is CCC.

The original post from 9/13/2008

I prefer an OS update to petting a rabid wolf or getting a kidney transplant. Even so, they're not my favorite things. It was clear early on that Apple had botched 10.5 - a prelude to the MobileMe fiasco and the iPhone OS 2.0 fender bender. So I waited to 10.5.3 before updating our non-critical MacBook. That didn't go all that well; I'm still having problems.

On the other hand there's a lot to like about 10.5, and I imagine I know what to watch for.

After lots of experience with 10.5.4, I was read to risk an update to very important machine -- our trouble free PPC iMac G5 running10.4.11.

It's the procedure I follow with all major OS updates. I do all the work through my admin account.

1. Check the backups are working so I know I have a current backup of data. I like to do a test restore of randomly selected file.
2. Have another machine available in case the update runs into problems -- you may need Google.
3. Don't do the OS update on a desktop machine during bad weather. This is a bad time to have a power failure. Make sure you can't accidentally pull a plug or turn off the power. (I once bricked a peripheral by hitting a power switch with my foot.)
4. Do a safe boot to clean up the system and verify the drive.
5. Disconnect all USB hubs and all firewire devices. Attach only an Apple keyboard and an Apple mouse.
6. Pull the network cable (see below). You can plug it in when you need to get software updates. Nowadays there are all sorts of things a partly updated machine can destroy if it can get a the net.
7. Restart then remove Preference Panes from admin account (ctrl-click then delete in preference view). Review and remove suspicious login items. Use Spotlight to find all apps or utilities with a date prior to 2004 - remove any that aren't needed.
8. Uninstall known bad actors. I know, for example, that my copy of Missing Sync for Palm OS won't work with 10.5. I don't need it any more, so time to use the uninstaller. Remove Retrospect's client if present, that will need to be reinstalled.
9. Turn off sync services, such as Spanning Sync. Don't turn them on again until you've run iCal, Address Book, mail, etc for the first time. I recommend turning off everything related to synchronization, including .Mac/MobileMe, anything in iTunes, any add-on services. To be extra sure, pull the network cable durign the update. Don't allow the machine to access the net without your control.
10. I've already removed the evil Adobe Acrobat Reader and RealAudio.
11. Copy the 10.5.4 Combo Updater to the desktop. I don't want to run 10.5.2 a moment longer than necessary. Confirm I have plenty of free drive space left.
12. Review Mac OS X 10.4, 10.5: About installation options so I don't miss the 'Archive and Install' option [1] . (Made that mistake before!)
13. Insert DVD and click the install button.
14. Go walk the dogs, do the dishes, etc. Just the DVD verification takes an age and a half. (Yes, you can skip the verification. I prefer to let it run.) The update should proceed without any questions, so you can let it go.
15. After the upgrade and reboot it can take a long time for the admin account to come up. Be patient.
16. Restart again (to let caches be build properly) then apply the 10.5.4 compo updater. The machine will restart.
17. Check all login items for all users. There's a bug in the 10.5.2 Archive and Install procedure that can cause login items to be applied across user accounts.
18. Check for other updates. I was surprised I had to install iTunes 8 again -- it had been installed earlier. I imagine if I hadn't done this, and I'd tried to sync to my iPhone, the heavens would have fallen. You have to keep checking until no new updates are found.
19. Run iCal and Address Book. Anyone else notice that 10.5 Address Book backup is under the export/archive menu now? Back 'em both up before any iPhones sync.
20. Enable Spanning Sync and do an iCal sync with gCal.
21. Run Keychain Access and Keychain First Aid.
22. Run any app that iTunes works with or that intersects with the iPhone.
23. Cycle through all accounts, looking for obvious trouble.
24. Hook up the peripherals, download drivers for the MacAlly keyboard, etc etc.
25. Expect Spotlight to suck CPU and drive the fan until the search indices are rebuilt. Let it run overnight.
26. The long recovery begins.

There were a few curious things about this update:

1. MobileMe didn't appear in software update, so it was only when I went to the old .Mac preference panel that I was asked to update to MobileMe. This might have caused some problems if I'd installed MobileMe.
2. iTunes regressed to an earlier version. I had to update to iTunes 8 again. This would have caused serious problems if I'd missed this.
3. Spanning Sync keeps telling me its deleting appointments from Google Calendar, but it doesn't say what it's deleting. I don't know why this is happening.
4. The update resurrected a number of old apps and login items that I thought were long gone. They're reaking havoc on my syncs.

[1] Select this option if you want to install a "fresh" system on your computer. This type of installation moves existing System files to a folder named Previous System, then installs a new copy of Mac OS X. You cannot start up your computer using the Previous System folder. Archive and Install installations require the largest amount of available disk space because you need to have room to preserve your existing System and the new one you are installing. This is a good choice if you've already backed up your important files and are trying to resolve an existing issue. Mac OS X-installed applications, such as Address Book and Safari, are archived, and new versions are installed in the Applications folder. Some applications, plug-ins, and other software may have to be reinstalled after an “Archive and Install.” Fonts that were installed in the Fonts folder in the top-level Library folder can be installed in your new system by copying them from the Previous System folder.

macOS Mojave Safe Mode can take hours, the progress bar is no help. Also - Disk Utility and APFS

 TL;DR

• When you startup in Safe Mode On MacOS Mojave the progress bar goes to 100% in a few minutes. It then sites there for a long time -- hours on my 3yo MacBook Air (APFS 256GB, half used).
• If you startup in ⌘R recovery mode it's not all obvious how to run Mojave Disk Utility First Aid on your encrypted data partition rather than the startup partition. (Different from running when logged in.)
• PS. Disk Utility changed in Catalina.

What happened

When I'm doing a major macOS update I typically backup two different ways, unplug everything, do a Safe Mode start to clean caches and verify directory structures, run a hardware diagnostic startup, then I update. This time around I was just applying the latest Mojave update but I decided to do a safe mode start for the heck of it.

Safe mode's progress bar smoothly progressed to 100%. And then it sat there. For an hour or two. Finally I gave up and powered down. Clearly something was wrong.

I ran a hardware diagnostic (startup D). No problems.

I booted in Recover Mode and ran Disk Utility. It quickly finished. No problems. (But this was a mistake.)

I googled and found basically nothing (crank note below).

Finally ... something clued me to look more closely at Disk Utility. The partition that First Aid ran against was only a few GB. That made me look around a bit. I can't recall the exact sequence but I had to do something in the menu bar to show all possible partitions. Then I had to select a drive/partition thing to mount it. That required a password for at least one user (I gave it my admin user password). Then I could see the true 250GB data drive. Then I ran Disk Utility First Aid on that ... and it took about 15-30 min for each of 19 Time Machine snapshots. In all it took about 4 hours to run.

That night I boosted into Safe Mode before leaving the computer. When I got up it had completed without a problem.

Basically that progress bar is a bug. (macOS is infamous for missing and misleading progress bars.) In Mojave Disk Utility Safe Mode seems to run Disk Utility First Aid on all partitions/volumes at the very end of its execution when the progress bar is complete -- and that can take many hours. Even if there are no problems. If Safe Mode runs overnight and still hangs then you need to run Disk Utility First Aid to figure out where the problem is -- including running it on all partitions/volumes.

A crank footnote

Fifteen years ago when I ran into similar problems a quick Google search would have returned many blog posts and forum comments explaining the problem. In 2020 my searches turned up a single cryptic Apple Discussion post saying safe mode took a "long time" to complete. I figured an hour was a "long time", but it looks like 8 hours is conceivable. The modern web is decrepit. I doubt, for example, that Google will actually index this particular post.

Things I miss: drag and drop link creation

I've mentioned here a few times that progress is not linear. Howard Oakley has a piece on a related topic today. For example, no application has done text style management as well as Symantec's MORE 3.1 -- which died decades ago. I don't think we'll ever see the like of Apple's Aperture again -- an insanely ambitious app for professional image editing and especially image management. The iPhone is a bit of an improvement over the Palm III, but it took years to equal Palm's task, calendar and note management (yes, really).

Today I mourn one small example of lost progress. It used to be easy to create a link to a web page. You'd click on the something in your browser URL display and drag it onto your web page editor (MarsEdit, FrontPage, Word, some web client editors) and *bingo*, instant link. The page title was the link text, page URL was, well, the URL. I can't do that any more, at least for Blogger (which seems to be in some kind of resurrection lately).

One day ...

PS. Been a while since I thought of FrontPage/Vermeer - Microsoft's 1990s web site manager. It was the Aperture of its day. Very ambitious, buggy, often flawed, but nothing like it now. Parts of it survived into SharePoint Designer, but now that I've mentioned SharePoint I'm spiraling into PTSD ...

iCloud backup and my lost authenticator codes

When my local Apple store tech was unable to remove the battery from my iPhone 8 they gave me a new device -- which was SIM locked to AT&T.

Well, everyone has to start somewhere, including Apple techs. Hope they improve soon.

Anyway, between the initial restore and the factory reset to clear the SIM lock I've been through two iCloud restores in the past week.

iCloud restores kind of suck now. I think they worked better a few years ago. The good news is that my photos were restored (I don't use Apple Photos/iCloud so I needed that backup). The bad news is that so many apps needed credentials reentered or new certificates generated -- especially when doing a restore after a hardware change.

The worst news is that Google Authenticator lost my authenticator codes. As near as I can tell they are restored from iCloud if the hardware is unchanged, but not if the hardware changes. Or maybe it's a bug. Whatever the reason, I lost 'em. 

It was suspiciously easy to regenerate Authenticator codes for my Microsoft account. Not too hard for Google either, because  they've moved to preferring an Apple-like proprietary two factor authentication mechanism. It is a bummer for Dreamhost though -- so now I'm going through support to try to recover access to my domains and web content.

It's hard to reconcile security and backup/restore. For example, Google Wallet and your biometrics (finger/face) aren't backed up either. On the other hand your Keychain credentials are in iCloud, and anyone who can get into your iPhone can read all of your passwords (try: "Hey Siri, Show me my passwords" or see Apple's hidden password manager). So your 4 digit Apple device passcode is not a great idea.

PS. I'm storing Authenticator codes in 1Password now. Which, like most small company software, has its own security concerns, not least that it would be relatively easy for China, say, to acquire the company or insert a backdoor into the source code.

ToDo apps: Microsoft's solution

I've used Appigo's ToDo app for about 12 years (with Toodledo at first). It's had problems over the years, but in general it's been a good subscription choice. There's a fairly hard data lock (maybe SQLite?) but manual reentry is feasible albeit annoying.

Lately, however, ToDo has been more ragged. A recent server side change induced a date bug (time zone?) that in turn showed me I was using a macOS app last updated in 2016. It appears to have been abandoned on the Mac App Store. When I went to Twitter I found Appigo's account was closed years ago for violating TOS. Eventually I found I could download a current version of their other App Store app from their web site.

At the moment the app is more or less working again, though parts of the macOS app UI are kind of weird. I figure there was some violent ownership transition with lost dev passwords in Appigo's history (maybe they got ransomwared?).

I decided to go shopping again. I'm looking at:

• Apple Reminders: hard data lock and I have to upgrade from Mojave to get to latest version (not happening).
• Google Todo: this is one hell of a weird product. WTF is their web strategy? Tied to Gmail? Tied to Calendar? At least there's data export.
• Things
• OmniFocus: poor Omni is in some disarray ...
• Microsoft To Do

Today I dug into Microsoft To Do. Of course it's a mess, but this is 2020 so we expect that. The mess starts with Microsoft reusing product names. To simplify a bit:

• There are classic Outlook Tasks. I'll call these TasksClassic. TasksClassic was excellent in many ways, including, once upon a time, great import/export options and lots of view flexibility (I like to sort by last modified!). Unfortunately it's dead, just barely hanging on in the current desktop app with some degree of synchronization with the new product.
• There's the new Wanderlist-based product variable called Microsoft To Do and ... Outlook Tasks (name reuse!). I'll call these TasksW for Wanderlist.

If you open the Help screen page for macOS TasksW (To Do) it takes one to a page on Outlook synchronization that's obsolete -- because the Outlook.com version of Tasks has switched from TasksClassic to TasksW. On the other hand the version of Office 365 on my Mac still has TasksClassic, and it does synchronize with TasksW as displayed on macOS and iOS Microsoft To Do.app.

Are you still with me?

This gave me a brief moment of hope that there was some data freedom here. I remember the import/export options of old Windows Outlook. Alas, the only import/export from macOS Outlook is Microsoft's PST format. There might be some way to do things with Outlook Windows or with 3rd party tools but I don't have the energy for that.

At this time I think TasksW is probably a decent enough product, but this has reminded me how screwed up Microsoft is. So I'm setting this one aside for the moment.

See also:

• There are no great task managers for the iPhone - but there's hope for 2011: My current task list started out on the Palm III and has migrated many times. I probably have uncompleted tasks from the 1990s. I think we're coming to the end of that data migration though. Note from 2010: "Google might finally provide an API for Google Tasks, allowing iOS client development."
• My ancient spreadsheet on migrating from Palm to iPhone - I think I used Outlook Tasks as part of the migration process, or maybe even Microsoft Access. Alas, too long ago. Painful though.

Can you do a Time Machine backup to a USB flash drive (thumb drive)?

 I'd wondered if it was possible to do a Time Machine backup to a cheap Flash Drive. My daughter is going to college and probably doesn't have a great need for backup (iCloud Document/Desktop, iCloud Photo, Google Docs, etc) but I'd still like to do something.

So I wondered about a compact Flash Drive. In the twilight of the web Google couldn't find me an answer, so I ran my own test. I used an old San Disk Ultra Fit 128GB USB 3.0 Flash Drive in an old USB 2 MacBook Air running High Sierra. I formatted the Flash Drive as encrypted HPFS and let Time Machine run the backup.

It took about 4-5 hours to do the initial 80GB backup but it seemed to work fine.

I doubt these Flash Drives are super reliable, but I think this is an option. I can see taking advantage of it while traveling for example. Unfortunately her 2020 Air doesn't have an open USB slot where an Ultra Fit could live, but it could be a part of her Anker 7-1 USB-C docking station. A modern San Disk Ultra Fit USB 3.1 is $33.

Chrome malware: Managed by your organization

I think my son installed a "managed by your organization" chrome malware extension when he was trying to find Flash. This one showed in Chrome as "searches.network"

The obsolete Federal government website required for his US census enumerator job probably directed him to get Flash. I wouldn't be shocked if he got the malware directly from the Federal site. US government web sites are notoriously insecure. [1]

This class of malware now works by installing an unsigned profile on the user's Mac that activates Chrome's "managed by your organization" mode. It locks the home page and search page so traffic is routed through the malware's server and it prevents a Chrome reset. (It may do other things as well of course.)  For some reason it locked him into Bing, which was a dead giveaway. Smarter malware wouldn't have changed the default search engine.

Once upon a time a quick Google search would have explained how to remove the malware. This is 2020 though, so Google's search results on this topic are mostly garbage. I found one result on a garbage site, however, that must have been partly based on a real site. That clued me to the profile. Once I deleted it then I could do a full Chrome reset. Once I knew the fix I found this guide, which covered the territory. (I can't tell who manages the site, I hope they make money by malware app referrals rather than anything more ominous.)

Before I did this I followed advice from a trusted source and installed the free (but suspiciously marketed) Malwarebyte antiviral. It found nothing. I'll try running one or two more antivirals (AVG, Sophos). Malwarebyte is an easy uninstall, so points for them.

[1] I am the solo family geek, my digital-age children seem to prefer the 18th century. My theory is the latest generation has the same take on computers that, at the same age, I had on automobile engines. It should just work, and if it doesn't work an old person might understand it.