Retiring domain associated with a DreamHost WordPress site - migrating site content to a new domain

I recently decided not to renew a domain managed and registered with DreamHost. That domain had an associated WordPress site. This is the sequence I followed to preserve the content of the site.

1. Turned off renewal for the domain (domain_old).
2. In WordPress for domain_old I exported the site as a XML file which I downloaded and archived.
3. In DreamHost control panel Manage Websites I clicked "new website". This let me add a subdomain to one of my existing domains.
4. In DreamHost control panel Manage Websites I clicked on domain_old and scrolled down to entry for Domain. That says "change your WordPress site to run under a different domain name". I chose the subdomain I created earlier. This took about 5-10 minutes to run.
5. When I then visited the subdomain I created the WordPress site came up.

The process only took a few minutes the first time I tried.

Downloading Facebook posts to document a trip

During a recent trip to Seoul South Korea I did a travelogue through Facebook posts about the odd sorts of things that interest me. I wanted to get them into a folder where I'm storying notes and information about the trip.

I found this worked reasonably well:

1. Follow Facebook's directions for downloading a copy of Facebook data.
2. Specify posts and the date range you want. If time zones are a factor you may want to go a day beyond what you'd expect. (The first download range ended Dec 15 when we left Korea, but that last day was not in the download. I suspect a time zone bug. I did it again specifying Dec 16.)
3. Specify HTML as a format and highest resolution images

I was notified the download was ready after about 30 minutes or so. The downloaded archive pages rendered in my browser. It's not a pretty layout but it has my text and images and comments on the images. It didn't include comments on the posts.

Canon DSLR doesn't show icon for remote control - you probably tried using the smartphone bluetooth connection

I have an old Canon DSLR, the EOS Rebel SL2. I think this post may also apply to other Canon DLSRs. 

If your remote used to work but now it doesn't, and the camera doesn't show the icon to enable remote control, chances are that you tried using the Bluetooth feature with a smartphone. You need to find the wireless setup part of your phone settings, then find the Bluetooth setting and turn it back to Remote.

The older cameras are kind of dumb, if you enable Bluetooth for your smartphone you disable the remote functionality.

Once you're fixed the settings issue you'll see the missing remote icon.

macOS OneDrive, ScanSnap PDFs and the "could not be opened" error in Monterey

When I migrated from Mojave to Monterey I ran into the typical array of macOS upgrade issues -- including having to reinstall Monterey. There's a reason I dread updating macOS.

One of the issues was that OneDrive didn't seem to work with my ScanSnap PDF uploads. In this case there were two suspects - the Monterey update and a OneDrive update. (One of the reasons I upgraded after migrating off Aperture was that OneDrive was no longer supported.)

The iOS ScanSnap client seemed to work as before, and the PDF appeared in the OneDrive folder I used -- but the file could not be displayed by Preview. I got a "could not be opened ... It may be damaged or use a file format that Preview doesn’t recognize."

It took a few searches to find the answer ...

If you were previously navigating to useraccount/OneDrive/DocumentFolder and opening your files from there, that seems to have stopped working now, and you have to go to Locations/OneDrive/DocumentFolder instead.

I'd had a Favorites link to the OneDrive folder that held my scans prior to the update. When I study where that link goes now it's not to the old file system folder, it's to:

/Users/jgordon/Library/Group Containers/UBF8T346G9.OneDriveSyncClientSuite/OneDrive.noindex/OneDrive/ScanSnap

So the folder that used to be in the file system was now buried in Library but the Favorite somehow resolved to it still.

I created a shortcut to a folder of the same name as displayed in Locations/OneDrive and the path there is

/Users/jgordon/Library/CloudStorage/OneDrive-Personal/ScanSnap

Digging into OneDrive preferences it claims my location is "/Users/jgordon/Documents/One..." (yep, truncated path). This is the path OneDrive used to use, but now there's just a Favorite there.  The true path is ... yep ...

/Users/jgordon/Library/CloudStorage/OneDrive-Personal/ScanSnap 

Even though none of the files are in a location that I expect Spotlight to index it does appear to index the files stored there once I revised settings so all files were downloaded. Once I did that however the file was now readable even in the location my old Favorite resolved to: 

/Users/jgordon/Library/Group Containers/UBF8T346G9.OneDriveSyncClientSuite/OneDrive.noindex/OneDrive/ScanSnap

 So this is kind of what I think was going on to cause this particular time wasting problem

1. Apple made everyone switch to their preferred approach to managing Cloud files.
2. The folders that were once in the local file system were gone, but an old Favorite somehow resolved to a similar folder buried in a virtual file system. The file, however, was no longer resident locally, it only seemed to be available if one inspected the virtual folder with Finder. Preview could not access it because it wasn't there, and in Monterey Preview gives a misleading error message.
3. When I used the Locations OneDrive "folder" to navigate I went to a different Library CloudStorage folder where OneDrive will auto-download folders on demand. If, however, full download is active (as it was previously so I can backup but that's not the default) then even the internal system OneDrive uses has a full copy and Preview will open it.

Apple wants all Document folders to be stored in the Cloud and may eventually want all user folders in the Cloud, so part of this is probably to prevent different Cloud Providers from cross-synching folders.

I think the bug hits those very few people who had a Favorite to an old style OneDrive folder prior to upgrading OneDrive. Although these kinds of complex emergent bugs don't hit many people, there are thousands of bugs like this so sooner or later we run into them. Which is why it's now very hard for non-geeks to use a personal computer.

Scripting Photos.app in Ventura (and maybe Sonoma) with AppleScript, Automator, and Shortcuts

[This post will be gradually developed. At the moment search engines return almost nothing on the current mess topic so even in its initial state it should be helpful.]

I recently moved my Aperture Photo Library from a machine running Mojave (last version OS with Aperture support) to one running Ventura and Photos.app. The migration was unspeakable.

Under Monterey there were a number of AppleScripts that mitigated key missing features in Photos (like batch rename). Some seem to ship from Apple with the OS. Others could be found online. Several appear in a compendium of user tips and scripts from Apple Discussions. But things have changed since Mojave and Monterey...

This blog post is going to be about scripting in Photos.app for Ventura and Monterey. In my own experimentation I was able to convert a 2019 Automator.app script containing AppleScript (basically it's a wrapper around a traditional AppleScript) into a Shortcuts.app script that runs as a service within Photos.app. To do this I had to enable AppleScript for use with Shortcuts and in Shortcuts details I could add it to the services menu. (Howard Oakley has an overview of the AppleScript to Shortcuts transition and an earlier article on Shortcuts in particular. The Shortcuts user guide article on this seems to have come from Automator but does actually work.)

I wasn't able to get the directions for adding Automator Quick Action Workflows in the Ventura Automator User Guide to work.

So I can batch rename images now from an entry in the Services menu. Yay.

If I find a good repository of Shortcuts I'll link to that. Otherwise I will try hosting them on my personal site or GitHub. I have an Apple Discussions question on the topic.

xt.local spam: Where it comes from, how to get rid of it

Recently more of the email that bypasses Gmail's spam filters has a return address with the suffix "*.xt.local". I've been able to find low quality posts about this issue from at least 2017, but in the past few months the volume has increased significantly.

One source of the emails has been spammer customers of Salesforce.

I'd previously marked one of these as spam, when I view it from inbox Google shows me: 

"You unsubscribed from <100018015.xt.local>"

If I try to reply I see:

"reply-fec215727d600275-157_HTML-100984021-100022017-13039@e.sixt.com"

If I choose to filter I get this as a filter criteria

Has the words: list:(<100018015.xt.local>)

An expert on spam filtering tells me xt.local is the name of an email list. 

I've added xt.local to Gmail's (increasingly obscure) filters as an automatic delete. I currently have it in there as a return address but if that doesn't work I'll try it as a label. 

Google's 'show original' extracts the following header information:

.... 

From: SIXT <info@e.sixt.com>

To: ********

Subject: Start your week in style! Up to 30% discount on midweek rentals

SPF: PASS with IP 13.111.115.170 Learn more

DKIM: 'PASS' with domain e.sixt.com Learn more

DMARC: 'PASS' Learn more

I'll update this post if filtering on xt.local as the sender address doesn't work.

PS. When I searched on this, even using Kagi (which is looking to be a very interesting search engine if they can stay in business), I found very few useful posts. Even five years ago there would have been dozens. Something rather important has been broken.

Fixing Eero performance issues - try removing a device

We bought a 3 unit Eero 6 mesh WiFi router. From early days we ran into unreliable video problems. Remote users would say our bandwidth was flagged as poor -- despite having a GB fiber connection and, you know, the Eero routers.

I never found any useful diagnostic or testing information, but the fix was to unplug one of the 3 Eero stations. With only 2 devices the video quality was much better. Presumably we have too many devices for the size of our home -- and the software for managing handoffs might not work as well as one would expect.

Converting from Aperture to Apple Photos: Personal results Jan 2022 through March 2024.

Preface

This blog post was written during conversion of my Aperture local and Photos.app iCloud (System) Libraries to a single integrated Photos.app Library. The first post is from Jan 2022 and I finished the process on 3/7/2024.

One of the very last things I discovered was that Aperture will run on Apple Silicon without Retroactive. I've tested under Ventura but not Sonoma. That is so weird. I have my old Aperture Library and a copy of Aperture.app on an external SSD and if I have post-conversion concerns it's a great help to browse the original Aperture Library.

The post is more-or-less made up of sequential updates. Only a desperate person or an AI would have the patience to read and use the material here. If you are reading this I presume. you are desperate. Some of the things described here for Photos.app migration are also necessary for migration to Adobe Lightroom.

Many Apple have mercy on your soul.

--------------------

I've begun early experiments in converting from Aperture to Apple photos. I've not found any good user descriptions of the process. This post will be updated as I do experiments or use the results of experiments to refine my searches.

Before beginning Jason Snell's freely available book chapter on Aperture conversion is a quick read. I also suggest reviewing a fine photofocus post on moving to Lightroom which might be less of an ordeal than migrating to Photos. I chose Photos because of my distrust of Adobe and my knowledge that any commercial photo management solutions is a hard and eternal lock-in. [1]

Trial 1/9/2022

For my first trial I created a new project from a single JPG image and one version using Aperture under Mojave. I copied it to an Intel Air running Monterey 12.1 and opened it in Photos.

Photos said it could not open the Aperture library. It did not say why.

I then regenerated full sized previews per Apple's directions and I tried again. This time the import proceeded. I could see the Version of my original photo but not the original. I then chose Revert to Original and Photos displayed the original image. Then I tried "Undo Revert" but that did nothing. My Version was lost.

Opening the package I could see a file in the originals folder named with a GUID and a .jpeg extension. It had a size of 2.3MB which is the size of the Aperture original.

I tried to repeat the process but Photos declared that "the content of the Aperture library ... has already been migrated to Photos". The famous léonie of Apple forums explained - the conversion process changes the extension to "migratedphotolibrary", I renamed the extension to .photolibrary and I was able to repeat the import process.

On repeating I saw my Version image. I then tried to edit it and got this message:

"Cannot Start Editing Photos cannot edit this image because it uses an unsupported format"

I explored the new Photos library (Package Contents) and could not find any JPG other than my original. I am puzzled as to how Photos generated the version I could see.

There does not appear to be any way to see the original image other than by reverting to original -- which cannot be undone. There also doesn't seen to be any indication that an original exists! (This appears to be a global Photos problem however).

If one wanted to see both a Version and the Original one would need to export the latests version then open that in Preview, then in Photos revert to Original, and export that. Then compare in Preview.

Update 3/31/2023

I finally resolved the stacks Aperture created when I migrated from iPhotos (original image and iPhoto edited).  The trick was to find all photos with keyword iPhoto Externally Edited (or similar) then save them to an album then in that album remove any that weren't stacked then unstack, delete externally edited, and re-edit originals (non-destructive).

Then I did "Reprocess Originals" for about 8000 RAW images; about 6,000 were reprocessed and about 2,000 didn't need it. That leads Aperture to slowly regenerate previous/thumbnails. It's been about 24h and that is mostly done, I find that I have to periodically pause the task then restart Aperture (or my laptop) and resume. If I really need to regenerate all full sized previews that processing may take a week or so.

Update Preview is also very slow, it in fact may the rate limiting step since I assume "Reprocess Originals" secondarily also updates previews.

Apple's Published Recommendation (for Mojave)

Choose Aperture > Preferences, click the Previews tab, then change the Photo Preview setting to Don't Limit. Close the preferences window.

From the list of projects in the Library inspector, select all of your projects. For example, click the first project listed, then press and hold the Shift key while clicking the last project.

Click the Browser layout button in the toolbar, so that all photos are shown as thumbnails.

Choose Edit > Select All to select all of your photos.

Press and hold the Option key, then choose Photos > Generate Previews.

Aperture now generates full-size previews for every photo in your library. To follow its progress, choose Window > Show Activity from the menu bar. Quit Aperture when processing is complete.

Open the Photos app, then choose your Aperture library when prompted, as pictured above. If you aren't prompted to choose a library, press and hold the Option key while opening Photos. If your Aperture library isn’t listed, click Other Library, then locate and choose your library.

In my testing "Generate Previews" goes very quickly through JPEG originals. DNG images take longer. RAW images may take longer but I think previews were done when "reprocess originals" was run.

I'm going to partition my images by types (including other) to better understand this step. I think unstacking all photos is also wise.

Update 4/8/2023

In testing I found Photos.app (Monterey) only needs Aperture previews for edited images. I've already created previews for all RAW images. Plan is:

1. Remove previews for all images without adjustments including RAW.
2. Regenerate high res images for all images that have had adjustments (I already did RAW so don't need to redo those). I think quick view requires previews so I may be able to use that feature to confirm which have previews and which don't.

Update 7/4/2023

What happens to metadata during conversion. Aperture claims it can write IPTC data to images that can handle it. There are many oddball IPTC metadata tags, I've listed only the ones I sometimes used below. Some of these conclusions are based on studying the Photos.app SQLite database.

[3/27/2024: I learned more much later. iOS does not provide access to Photo Titles, only macOS Photos.app shows them (I'm too tired to be angry about this). Captions in both cases are seen only in the Information screen. Contrary to what was documented by Apple below Photos.app uses the term Caption, not "Photos Description". Were I starting over I'd have copied the Title to Caption when I had no existing Caption value set.]

Aperture                   Photos

Version Name               There's no documentation on how

                           Photos is handling Version Name and 

                           suspect it varies by macOS release.

                           In Monterey my test show the Version 

                           Name is at least sometimes used as the               

                           Photos.title rather than the 

                           Aperture title or the filename.

Filename                   Copied to Photos

Title (IPTC)               Copied to Photos

Caption                    Copied as Photos Description

Keywords                   Hierarchy (path) flattened, 

                           and each string becomes a kw

Project Name               Becomes an album (iPhoto Events)

Project Path               Unknown at this time.

Flag                       Not converted

Date                       Seems to use Aperture date

Badges                     Not converted

Location                   This is saved but location name is lost

Named faces                Saved

Star rating                Become keywords (1 star, 2 star)

Headline (IPTC)            Lost but can write to JPEG

Location (IPTC)            Lost but can write to JPEG

City (IPTC)                Lost but can write to JPEG

Creator (IPTC)             Lost but can write to JPEG

Date Created               Unknown at this time

Date (adjusted vs. file date) Image date is used

Last Modified Date        Not converted

I was able to retrieve Apple's documentation from the wayback machine (https://web.archive.org/web/20190515155043/https://support.apple.com/en-us/HT204478) to see how conversion proceeds. Here's a plain text version:

Photos and organization:

Photos and movies migrate without changes. Photos migrate with the adjustments and filters that you applied in Aperture. You can't change the adjustments in Photos, but you can revert to the original image.

Projects and subfolders migrate to folders that are labeled "Aperture Projects" and "iPhoto Events" in the Albums view.

Photos preserves albums.

Photos preserves most Smart Albums. Smart Albums with selection criteria that Photos doesn't support usually migrate to Smart Albums with "(modified)" added to the Smart Album name. If Photos doesn't support any of the Smart Album criteria, it doesn't migrate the Smart Album.

Photo books migrate to Albums.

Photos preserves slideshows. If a slideshow theme isn't available in Photos, it uses the default Photos slideshow theme. 

Photos preserves Faces data.

Keywords and other metadata: 

Photos preserves keywords.

Photos preserves user-defined titles. Images without user-defined titles show as "untitled" in Photos.

Star ratings migrate as similar keywords in Photos, such as "1 Star," "2 Stars," and so on.

Flagged images migrate with the keyword "Flagged" and appear in the "Flagged" Smart Album in the Albums view.

Color labels migrate as keywords, such as "Green" and "Purple."

Photos preserves captions.

Most metadata is preserved in Photos. Some IPTC metadata, including Copyright, won't appear in Photos. This metadata is still associated with the image, and you'll still see it in other applications that show IPTC metadata, like iPhoto and Aperture.

Custom metadata fields don't migrate to Photos. 

It's possible to use AppleScript to copy Version Name to Title. As of 7/2023 the following script was copied from Github

tell application "Aperture"

  set selectedImages to (get selection)

  if selectedImages is {} then

    error "Please select an image."

  else

    repeat with i from 1 to count of selectedImages

      tell library 1

        tell item i of selectedImages

          set versionName to (get value of other tag "VersionName") as string

          log versionName

          make new IPTC tag with properties {name:"ObjectName", value:versionName}

        end tell

      end tell

    end repeat

  end if

end tell

These were PhotoJoseph's 2014 directions for use of this AppleScript

Launch AppleScript Editor and create a new script (File > New).

Paste your clipboard contents into the new window, and click the Compile button to ensure it's all good. If not, just make sure you've copy and pasted all the content from the github page.

Save the AppleScript to the folder ~/Library/Scripts/Applications/Aperture (if any of those folders don't already exist, just create them). Name it something clever like “Copy Version Name to Title field”.

Open the AppleScript Editor preferences and ensure that “Show Script menu in menu bar” is enabled. Without that, you won't be able to access the script from within Aperture. [I think there's a Mojave bug such that this setting may not stick.]

On my not-too-fast Air it takes about 2.5 minutes to process 1000 images. I've set it to do 2000 at a time and check back after 5 minutes. I tried doing 8,000 at a time but that seems to cause Script Editor to hang so the maximum is between 2000 and 8000.

For best performance sort list view by Version name (since it doesn't change), show the Info panel, start at the bottom and select above (2000). The Info panel updates when the script reaches the last row selected.

Update 1/17/2024 - A retrospective note on keywords

This post is basically a chronologic dump of my Aperture slog, but I'm going to stick this update back in time. After I was done with the major migration I had a lot of keyword cleanup to do. Photos can't merge keywords, so this was tedious. Photos also can't handle Aperture keyword hierarchies, it only keeps the keyword itself. Prior to migration it's best to flatten the hierarchy and look for keywords to merge (variation in case, etc).

Update 7/6/2023

From an Apple Discussion post I learned that Apple's conversion process will write the output to the same drive as the input. A tech article on moving the photo library is also helpful to review. Because of the way macOS hard links work (certainly in Monterey and later) the output takes up much less space than one would expect.

I'll be working with a 2TB USB-C/Thunderbolt drive so this is good news.

Fat Cat Software's PowerPhotos also does Aperture conversion. I own it (it mitigates the pain of losing Aperture) so I asked the author about metadata. Note PowerPhotos can browse the original Aperture Library and do conversion even on versions of macOS that don't support Aperture. Here's his response:

Most of that metadata will come through as expected using PowerPhotos. The things I would note:

Version name won't be retained

Aperture projects are stored in the database analogous to iPhoto events. The PowerPhotos conversion mimics Photos' previous migration process, and creates a folder named "iPhoto Events" in the album list, and creates an album for each event/project coming from the source library.

Dates should be retained, though in certain cases there can be oddities with time zones. When you add the Aperture library to PowerPhotos' library list though, you can use the View > Show Subtitle > Date menu item to show the photos' dates and browse through them to see if PowerPhotos is reading them as expected.

For locations, the latitude/longitude will always be retained, but if you had assigned any custom names to particular places in Aperture (e.g. "My House" or whatever), those won't come across.

Star ratings become keywords just like in Photos migration

You're correct that those other IPTC fields will be ignored

Photo stacks should appear as multiple individual photos in PowerPhotos, but again you can double check by browsing through the library first in PowerPhotos to see how they show up.

PowerPhotos is really designed to migrate iPhoto Libraries, it only works on Aperture because they shared a common database. Parts of Aperture that are not supported in iPhotos don't migrate. This includes folder relationships (nested folders are flattened) and all Folder/Event relations. Events end up in a flat list.

Update 7/10/2023 iPhoto Externally Edited Cleanup

I have reviewed all the photos that were edited in early versions of iPhoto (previously I'd done a key subset of these); these were stored in Aperture as stacks with the edited version as the "stack pick". I unstacked and either deleted the non-pick or reedited the original and deleted the edited image. There were about a thousand of these so this took a lot of work.

I have learned that stacks will become separate photos in Photos app with the keyword "stack pick" assigned to the pick. 

Version name assigned to title when title empty

It took about 30-40 repeat applications of the above referenced AppleScript but all empty titles now hold the version name. I could only do about 2,500 images at a time without crashing Aperture and I went through about 58,000 images.

7/18/2023

Consolidate referenced files

I created a smart album of all referenced files and then consolidated them into the database.

7/19/2023

Update or regenerate previews of all images that Aperture identifies as adjusted. This took a very long time to do for RAW, JPEG was fast. Several hundred were identified as needing updated previews. Images I have not adjusted I'm ok with Photos managing with a preview. Before you do the conversion step redo "update previews"; if previews are current it should go quickly. In my case however, after a database rebuild, some had to be redone. I don't know if that was user error or something weird.

Identify other file types: At one time iPhoto/Aperture could hold audio files and even PDFs. Standard Aperture search shows audio files, I exported those and delete date originals. Search does not support PDF file types but I searched on "PDF" and found 5 images. I removed these from Aperture, a few I converted to JPEG and restored to Aperture. In my testing JPEG2000 images were supported as well PSDs, but I converted the latter to JPEG anyway.

Update 7/25/2023

Preparing for merger with existing Apple System Photo Library. I submitted my plan to Apple Discussions for review, basically I plan to make my converted Aperture Library the Apple System Photo Library which should merge with the smaller older iCloud Photo Library I have now.

Photos Preparation (Current System Photo Library)

• Download all images to my Apple System Photo Library. When you change the download setting in preferences Photos will start bringing down the image files. You can switch to Moments and scroll to bottom of screen to see a transfer progress dialog. A mere 2000 files takes an hour or more on an Intel machine. (Later I did this on an M2 where conversion of a 50,000 image library takes about 3h; that's then followed by Photos' processing that can take days. The conversion is probably 10 times faster on an M2 though.)
• Backup my current Photo Library
• Remove images from the Photo Library that have been exported previously to Aperture (I tagged them with keyword) then purge them from iCloud. Allow sync to happen
• Remove albums that are empty, in general cleanup of the Photo Library

Update 7/26/2023

Repair Library Database: Cmd-Opt on launch gives the option to repair the Aperture Library Database. This is a good measure to make prior to conversion. You can also rebuild the database. In my testing rebuild seemed to work well and I did not have to regenerate Previews for RAW images. It may be wise to do a full database rebuild followed by a database Repair prior to conversion. Look for a Recovery Album to see if anything shows up.

Identify Problematic Video: I have been told it will take weeks to synchronize the ApertureConverted Photos Library. Some old videos may cause the process to hang.  A support article has some advice. As of 7/2023 though I exported the originals of 170 AVI movies and after a bit of experimentation it seemed as though most were fine in both Photos and iCloud. Only 4 that didn't show any preview icon in the Finder needed to be opened in QuickTime ("converted") then saved as .MOV and then they worked. So I will proceed for now without additional video work.

I come across a 2012 blog post that shows I went through the same movie/ video transcoding problem converting from iPhoto to Aperture.  I see that in 2012 I could use QuickTime to change the video file wrapper from ?? to MOV without transcoding and that appears to be what I did for 4 small AVI files.

7/29/2023 - Legacy Media

A comment by léonie (192K "points" on Apple Discussions, I wonder if it's a single person!) sent me to their article on How to Weed out Legacy Media in Photos. Here I see that support for PDFs and audio files was removed, so my intuition to pull those was correct. The problematic items query is excellent. I'm copying his shared image here for posterity. Some of these queries can be done in Aperture but "unable to upload" is Photos.

In Aperture I used this query. I was surprised to find some PSDs and some jp2! However, in my testing Photos including iCloud sync handled both file types (added externally, not through an import).

7/29/2023 - 600 items missing?

UPDATE: I've left this in place but you can skip to the next update for now

I did a test conversion, only took 3h on the M2 Air. Sadly there are 600 fewer items in Photos than in Aperture. I'll document my investigations here using friggin' spaces and a fixed width font because tables aren't supported anywhere any more...

File Type             Aperture       Photos

RAW                     7333                7,333

JPG/JPEG           48,576           48,031

After a full rebuild of the database the JPEG count remained 48,576 and total count was 57,854. After import to photos the result was again 48,031 -- so it's very consistent.

I'm starting my research by focusing on the missing JPG/JPEG. 

It's possible to get a list of all filenames in Photos Library using the Photos SQLite database (see also and a forensics post.) [Update: Simon Willison has the best post on this, including a link to the dog sheep-photos tool and osxphotos]

Sadly, based on my Google Searches and ChatGPT questions there's no way to do this in Aperture. However you can use Aperture's metadata export to create a tab delimited list of some metadata for a list of Aperture images. (Most of the columns are null and I found rows where the title in the list didn't match the title in Aperture or in Photos but it was the best I could do.)

Once I used BBEdit to eliminate duplicates I used an online tool, https://comparetwolists.com, to show me what was in the Aperture list but not in the Photos list. I took that result of 127 rows and then looked for a sample in Photos. Most of the time I could find them, though in at least one instance Photos was using the Aperture Version Name or File Name rather than the Title. I wondered if there was a length limit enforced during conversion (Aperture and Photos have no documented Title length limits.) but I don't think that's the case. I'm continuing to investigate.

Update 8/25/2023: Image count discrepancy - Bursts and Live Images

At least part of the above image count discrepancy is because of the way iPhone "Live Photos" and "Bursts" work. In both cases an Apple proprietary metadata identifier is used to tie files together. In a "Live Photo" it's the assetIdentifier, in a burst it's the burst's BurstUUID. These metadata values are set by iPhone when the image files are created. Aperture doesn't know about these identifiers so in Aperture images that were part of burst and files that make up a Live Photo are all counted separately. On import to Photos they are recognized and the "burst" is counted as one image. If you know how to expand a burst in Photos (it's actually less obvious than I'd like) you'll see the images are still there.

After additional date partitioning to compare mismatches of a limited number of images over a problem date range I found no other causes of image mismatch. I decided the conversion process was not losing images.

Update 8/28/2023: Do all my images have previews?

After all the work done to get reapply legacy (ancient) iPhoto edits I want to be sure every image that needs a preview has one. The gear menu has for the Library has a menu item "Maintain previews for all projects". Despite my preview work it showed a dashed line rather than a check mark. When I chose it about 1600 images were identified as needing previews and those were generated. When I selected ALL 54,000 images and chose "update previews" 1387 were identified as needing a preview update. Even thought those seemed to update normally when I repeated "update previews" I got the same number.

So there's something wrong there. I wasn't able to find out what however. I checked all images "with adjustments" and preview update reported they were all current. I then did a database repair, which caused regeneration of thousands of previews and thumbnails. Despite preview update for all photos always identified 1387 items as needing a preview update. I decided to just live with this.

Update 8/30/2023: Conversion day attempt (did not succeed)

I moved the external SSD with my Aperture Library on it to my Monterey M2 Air and copied it to the Air's external SSD. Then I ran the conversion for the 56K images, it took about 2-3 hours. When completed I could see I had about 170 burst images; that fits the "missing" 600 photos. I'll be going through them and choosing images to save while removing the others.

I then turned off Photos iCloud sync on my legacy Mojave machine and, after confirming I had my copy on the new machine and backups, I deleted my Mojave Photos.app Library. I'll leave that machine on Monterey for now and Aperture will be untouched.

The next step will be to merge my converted Photos.Aperture Library into my existing Photos.System Library. I will wait until Photos is done processing the Photos.Aperture Library; that may take a day or so. There will be backups.

Update 9/12/2023: Burst delete bug (see Apple Discussion)

There's a bug in Photos Monterey and Photos Ventura with managing "Burst" media images generated from an iPhone and stored in Aperture. It may be related to changing either the file name or version name in Aperture. I suspect it's related to version name changes.

The Burst appears to open normally, but count of selected images may be anything from 1 to all and it may not change when an image is selected. Essentially Photos doesn't correctly match the user's selection (user interface) to its internal list of the selected image. I've seen all of the following:

1. Nothing happens. You select images, click done and you don't get the usual save all vs save selected dialog.
2. You get the save all or save selected dialog but whatever happens next is unpredictable. Some may be saved. In some cases, perhaps most often, all images in the stack simply disappear. The "recently removed" folder does not appear and the images are not in the system trash. If you rebuild the Photos database they may reappear.

I was able to demonstrate the bug to Apple Level II media support. They confirmed it as an "issue" but said there was no hope of a fix. I was able to show it happen in a Library converted by Monterey in Monterey and also when moving the same Library to a Ventura machine.

I have not found a workaround. I may work within Aperture to identify the bursts there and remove all but one image from each burst. I may try to use ExifTools to remove the MakerNotes (or just Burst ID) from those that remain in Aperture.

[In later testing I tried using the native Photos.app import tool to convert my large Aperture Library. It failed midway through repeatedly. So PowerPhotos became my only option.]

Update 9/12/2023: Eliminating Bursts

From Photos Media Bursts folder I identified the date of oldest and latest burst. Then in Aperture I created a query for that date range restricted to .JPG and camera has iPhone. Then I set Aperture to auto-stack any images .09 seconds apart. I then had about 6,900 images to review. I put those that had a stack count > 0 into a "Probably Bursts" folder. This took about 20 minutes to do 1/3 of the work.

Once I've done a couple of scans this way I'll remove all but one image from each burst. Then I hope to use ExifTools to remove the Burst ID.

Update 9/13/2023: Thoughts on eliminating the burst problem

Ben Smithett wrote a 2020 article on how Apple implemented bursts using Exif.Photo.MakerNote: "key 11 ⁠— this UUID is shared by all photos taken in a single iPhone burst and is what the Photos app uses to identify and group multiple photos into a single burst." He mentions an iOS app - The Photo Investigator and this post. He provides a set of examples for making a set of images appear as a Burst in Photos.app:

exiftool -tagsfromfile ./iphone_burst.jpeg -makernotes ./target_photos

exiftool -G ./target_photos/gopro_burst_1.JPG

exiftool -BurstUUID="$(uuidgen)" ./target_photos -overwrite_original_in_place

exiftool -BurstUUID ./target_photos

I need to look at the MakerNote for a non-Burst iPhone image to see if I need to remove key 11 or set it to some non-burst value (ex: 0, NULL, etc).

My tentative plan of attach:

1. In Aperture eliminate all but one image for each burst (as above).
2. Use keywords to tag these images.
3. Revert to original and remove previews.
4. Review version names and update filenames to match version names with a distinctive filename suffix like _iPhoneBurst.
5. Export as test library
6. In test library open package and view folders of masters and identify images.
7. In test library run ExifTools to make changes as needed across these files using the -r option. I believe I will need to "clear" key 11 with something like [exiftool -TAG= /path/to/files/]. The -r option is not compatible with wildcard filenames but there's an if option: -if '$Filename=~/_AppleBurst/'.
8. Move Aperture test library to Monterey machine and convert to Photos Library.
9. Review behavior.

Update 9/15/2023 - ExifTools

Progress today:

1. Created test library 
2. Installed ExifTools DMG, no Perl install required
3. Found that iPhone non-burst images lack the BurstUUID Tag completely
4. ExifTools can't remove individual tags in MakerNotes but it's possible to remove the entire MakerNotes: exiftool -MakerNotes= noMakerNotes.jpg
5. Created a test library and tried converting that on Monterey Photos. I found the same bugs as before with burst photos but the image I removed the MakerNotes from shows as a regular image. Sadly the burst version wasn't imported, perhaps because Photos.app considered it to be a duplicate and excluded the entire burst. See a later comment on duplicate photos and bursts.

My current version of the ExifTool command is:

exiftool -r -overwrite_original -P -progress -MakerNotes= -if '$BurstUUID ne "" ' ./TestSuite.aplibrary 

The -r flag means it will recurse over all subdirectories of the TestSuite Aperture Library, overwrite_orginal manes that the original image is removed, -P keeps the modification date of the original, -progress gives me a list of changed files, -MarkNotes= means the MakerNotes group of tags is removed, the -if statement tests for presence of the BurstUUID. 

The -progress output can be copied out of terminal and with a bit of regex I was able to extract file names (not guaranteed unique!) for 1,564 images stored in Aperture that had a BurstUUID.

You want to remove the MakerNotes from both images and previews. You do want to create previews for every image despite my earlier experiments with not creating previews for non-edited images.

Despite removing the MakerNotes about 68 images still rendered as Bursts. On inspection with ExifTools they don't appear to have MakerNotes and thus no BurstUUID visible to ExifTool. I am mystified. My best guess (see my stackoverflow comment) is that Aperture cached the EXIF data and Photos read it from that cache during the import process.

I will now evaluate PowerPhotos migration of an Aperture Library.

Update 9/15/2023 - Photos Conversion Duplicate Delete Bug

In a test library I had two versions of a single image in a test album. One had the MakerNotes removed. They also had different titles. After converting to Photos the image with MakerNotes removed was missing, but it's container album said two image were present. After rebuilding the Photos database the album reported one image present.

I don't know if this is "normal" (albeit buggy) Photos behavior or if it's an indirect result of the Burst Bug. I fear it is normal behavior for converting an Aperture Library; that metadata is ignored and perhaps edits and versions are also ignored.

On the other hand with non-Burst images in my testing Photos did retain multiple Aperture Versions from one Master image as long as the images differed.

Update 9/17/2023 PowerPhotos Migration of an Aperture Library

My sanity feels restored. I used PowerPhotos to convert my Aperture Library. This can work with post-Monterey macOS as well as Monterey. The results are reassuring.

PowerPhotos reported 97 images in my Aperture library and 97 images in the post-conversion Photos Library. When I opened Photos it showed 87 images and 10 videos. For the first time the counts agreed!

Most importantly - NO BURSTS. All the burst images are now independent. So there wasn't anything in the Burst images or the Previews; the (flawed) resurrection of the Bursts in the absence of the MakerNotes must have been related to copying some unknown data source (independent cache of EXIF data?) from Aperture. (I did not find BurstUUID in the Aperture.xml file stored in the Photos package by the Photos conversion process.)

I am further evaluating this process but my initial take is that PowerPhotos migration is vastly better than trying to use the native Photos conversion.

Update 9/17/2023: Puzzling test results, need remove BurstUUID and use PowerPhotos conversion

I created a single Aperture test library with a single known burst in it called MicroBurstTest. I also retested my larger BurstTestLibrary.

The results surprised me. To get a reliable translation I needed to use both the ExifTool step and I had to use PowerPhotos.

MicroBurstTest NoBurstUUIDRebuilt: FAIL with Photos Conversion

MicroBurstTest: PASS with PowerPhotos Conversion, 4 images in burst, can edit

MicroBurstTest NoBurstUUID: PASS with PowerPhotos Conversion 4 separate photos

BurstTestLibrary 2: FAIL with PowerPhotos Conversion, image selection bug.

Everything failed when I use Photos native conversion, even if I remove the burstUUID (MakerNotes) from all images and then fully rebuild the Aperture database. I thought that would deal with a hypothetical caching bug but it made no difference. (I didn't try creating an export Library then importing the Export Library into an empty Aperture Library -- mostly because I'm very tired of this and I may have a fix.)

PowerPhotos conversion seemed to work in my small library even if I left the BurstUUID intact, but when I tried a larger library the bursts I tested sometimes behaved properly and sometimes the selected image vanished

I'm hoping I can move on, but for now I'm thinking there are two issues/bugs:

Burst image selection bug: This doesn't happen to every image. I wonder if it's something triggered by version renaming or title or file renaming in Aperture. The image you select can vanish and the count of selected images doesn't match one's selection. I have a few burst images that work properly following a PowerPhotos Conversion but many do not.

Burst reconstruction:  a Burst is reconstructed even though the BurstUUID is gone, I think this is due to Aperture storing EXIF data from an image and not updating that even with a complete database rebuild.

I'm now proceeding to testing running ExifTools against the entire library, then regenerating Previews, then trying the bug library with PowerPhotos.

Update 9/17/2023 PowerTools import of entire library: 57,844 images

After using ExifTool to remove all BurstUUIDs and converting to Photos using PowerPhotos I had 57,844 images imported with one error in the log that I need to look at. Likely a corrupted or incompatible video. The process was slower than Photos native conversion. There were no Bursts in Photos after the import. This number lines up with what I expected.

Update 9/20/2023 Testing ok so far, Aperture clean-up continues.

So far testing is good. I'm cleaning up Aperture for the final migration; I may test Photos in Sonoma to see if the Burst Bugs are any different.

There's a feature that lets you write the modified date into the exif header and maybe the file date; I realize I should carefully do that in Aperture for some of the photos I've re-dated to the time they were taken (vs. when I scanned them).

I posted this summary to the Facebook Aperture Group, when I'm done it will end up as part of an introduction to this post.

If you haven't migrated off Aperture yet  and you hope to keep any of your titles, version names, dates etc time is running out. You should always be able to get your Originals out, but access to edits (Previews) and especially metadata (names, titles, descriptions, folder relationships, etc) is at increasing risk.

So far testing from my latest attempt to migrate to Photos.app is good. As per my prior notes I needed to use:

ExifTool to work around the Burst bugs.

PowerPhotos to do the migration (because of bugs with Photos native conversion)

An Apple Script to copy Version Names to titles. (Apple script itself is on the edge of extinction btw)

There was also a lot of review and cleanup including removing files not supported by Photos (esp iCloud) or most other media apps.

I don't know how migration to anything else would work. Life is way too short to test. I chose Photos seeking safety in numbers. I wish a competitor had a solid "exit" strategy.

If there's any chance you will need to use Photos you should buy PowerPhotos (https://www.fatcatsoftware.com/powerphotos/) *now* and let the dev know of your interest in the Aperture library conversion feature (works in Sonoma). For him the Aperture conversion was very much an afterthought (built atop some old iPhoto conversion). He's a much better dev than the poor intern who got suck with the Photos conversion. PowerPhotos also provides core functionality that should have always been a part of Photos.

I'd recommend saving a copy of my blog post on the process [1]. I'm not immortal, it could go at any time. I'm sure it's wrong and flawed but this kind of knowledge is dying fast. Some things I found in only one old, almost forgotten, web page. You should also assemble the tools I used just in case you end up on Photos somehow.

If you want to send a bill for your time to Tim Cook please do. I figure $20,000 would be about right for me [2]. Apple could have prepared a high quality Aperture export to a well documented public intermediate data model that Lightroom or others could read from with good results. Including documenting Apple's edit rules. They didn't because they didn't (and don't) give a sh*t. (sorry, expletive earned).

I'll clean up my blog post when I'm all done. For now it's relatively agreeable cleanup of existing content in Aperture prior to repeating my migration.

Backups? Oh, yeah, I got backups 😳.

[1] https://tech.kateva.org/2022/01/converting-from-aperture-to-apple.html

[2] Plus a pittance for PowerPhotos and a donation to ExifTool.

Update 10/16/2023: Another migration attempt only partly successful

Today I am making a hard push on the migration. I did not rerun the AppleScript that copied version name to 

In Aperture

1. Remove all flags from images.
2. Empty rejected and trash
3. Regenerate previews for my Video search folder for which many were lacking previews for some inexplicable reason.
4. Document photo count: 57,518 items
5. Update previews - this always seems to require regeneration for about 1300 JPEG for no good reason
6. Keyword cleanup - review the keyword hierarchy
7. Run repair database (again) - recheck image count, check for creation of "recovered ..." album/project. (no changes found).
8. Update Aperture Vault.

In Finder

1. Lock Current.aplibrary to prevent further changes to it.

On target system

1. Unlock Current.aplibrary and rename it Current_20231016.aplibrary
2. Run: exiftool -r -overwrite_original -P -progress -MakerNotes= -if '$BurstUUID ne "" ' ./Current_20231016.aplibrary
3. Run PowerPhotos Conversion
4. Manually recreate folder hierarchies and folder event relationships. This is bad but it isn't quite as bad as it sounds because when you select a folder on the left panel the albums and subfolders display on the right side. You can multi select those and move them together. I marked my 4 and 5 star (keyword) photos and my slideshow photos as Photos.app "favorites".

Update Oct 29 2023

After the Oct 16 post my big surprise was loss of folder/album relationships due to limitations of Power Photos (updated above). I had to recreate album/folder relationships. It's doable because in some views you can select multiple albums and move them between folders by drag and drop. The PowerPhoto conversion process turns Aperture/iPhoto Folder/Album relationships into keywords with an internal | delimiter. I didn't find the keywords useful but they are a guide to how Albums and Projects in Aperture were organized; I used them to selectively apply the same folder/album relationships in Photos.app. Then I deleted them.

Some photos have incorrect orientation. I think this is related to a bug in an old version of iPhotos (exif orientation tag bug). I believe they were fixed in Aperture but that appears to have been lost in conversion.

Otherwise I've been doing cleanup. Both PowerPhotos and Apple Photos.app will identify and resolve duplicate images but I'm waiting to understand Apple's undocumented rules for metadata management.

I have not yet merged my new library yet with my iCloud Library. I'll be organizing and cleaning up for a while. Keywords alone will take a bit; PowerPhotos made parent albums into keywords so there's a lot to remove.

When I do merge I will

1. Verify current system library has all images local.
2. Organize current system library so all albums are in one folder
3. Backup current system library.

Update 1/17/2024: Post-migration PowerPhotos hierarchy Keyword Cleanup

Since Photos.app was unable to migrate my Aperture Library I used PowerPhotos. That seems to have worked pretty well although since PowerPhotos was made to convert iPhoto Libraries some album/folder/project relationships were lost (however the albums themselves were preserved). PowerPhotos generates keywords that have folder/project/album relationships so these can be used as a guide to creating folder/album relationships in Photos.app. They can also be used as keywords of course, but I ended up with hundred of them so I used them as a guide to recreating folder/album collections that I found useful. Then I deleted them.

This is roughly how I did that

1. search for album in Photos search.
2. When click into the album the left pane will open the folder that contains the album (but it doesn't highlight the album -- that's just dumb)
3. Decide if there's a folder I should create to group things. If not then delete keyword.

In practice I deleted most of these hierarchy keywords. When several albums shared the same "path" then it was often useful to create a container folder. You can select multiple keywords and delete at once.

Some keywords were from albums that were organized in Aperture under Projects. In Photos both the Project and the child Albums become peer albums. In this case the keyword path will include the Aperture Project name and the Album name. Sometimes you might put them both in a folder.

The single window Photos.app UI makes reorganizing folders and albums pretty frustrating, but this was a flaw in Aperture too. To navigate long lists of albums I found sort by name helpful.

I did end up with collections of albums that belonged to an Aperture Folder that were better moved into other Photos.app hierarchies but that can be done over time. Since one cannot search on Folder names in Photos.app 2024 I have to rely more on folder hierarchies than I did in Aperture.

Note you cannot merge keywords in Photos.app. There's a workaround.

Update 1/17/2024 - Comment on Gamma and Color Profiles.

I realize all my images look darker than I remember. I suspect it's because of Aperture's ability to adjust images for "gamma", screen type and so on. Of course Photos has none of Aperture's extensive color profile and gamma support.

Update 2/6/2024 - Garbage keyword cleanup

Following migration I had multiple instances of the same keyword as upper and lower case. I'm not sure keyword selection is actually case specific in Photos. In all cases the lower case version was not actually used by any images and I removed them. There were about 10-20 oddball keywords I don't recall using in Aperture.

At this point I need to confirm some backups and then, after testing my backup Libraries I'll begin the iCloud merger between my migrated Aperture Library and my original System Library.

Update 2/28/2024 - Duplicate cleanup - time zone bug in conversion

During the conversion process a number of duplicates were created in addition to genuine older duplicates. It looks like correcting times in Aperture (esp timezones) somehow resulted in duplicates during the conversion. I'm using PowerPhotos and native Photos.app duplicate detection to slowly work through the duplicates and decide which to keep.

I think the way Apple's duplicate selection works marking one as a 'Favorite' will bias selection to that one. I saved all my Duplicates to a local folder then inspected them and sometimes marked on as a favorite and the other not. When done I removed the Favorite tag from the images retained in the folder I'd created from Apple's Duplicate list.

Update 3/1/2024 - Syncing with iCloud continues

After resolving duplicates I tested my backups (full image locally) and then made my converted Aperture Library the new System Library. For about 12-24 hours it simply said "Syncing with iCloud". After 1-2 days it started saying "Syncing #### from iCloud". Activity Monitor Network Activity showed lots of traffic with bursts of 20-30sec activity followed by longer intervals of no traffic. I assume it's throttled by Apple's servers. I assume this will take a week or two.

On the machine holding the former Aperture Library the image count hasn't changed but the video count it going up. Other devices show the same Folder/Album and images as before but the total image count is going up. Based on that count I'm guessing I'm about 2/3 done after 3 days. The delays are all on Apple's side.

Update 3/3/2024 - completed syncing with iCloud

It's possible that turning off iCloud shared folders helped the sync go faster. I'll turn them on again when done. [3/27/2024: I recommend turning off shared albums then turning them on when iCloud synchronization has completed.]

After 3-4 days of intermittent bursts of throttled iCloud synchronization I have one large collection that appears to include folders, albums, keywords and images. I merged a small number of new duplicates -- two video and one still. I can't confirm metadata has survived yet.

Update 3/27/2024 - After the migration

It's been several weeks since I turned on iCloud synchronization. That seems to have worked as promised -- which is unusual for anything related to Apple Photos.app. I did discover that iOS does not display Photo Titles; much of what I have now in Photo Titles could be copied to Caption and I am looking into ways to do that especially where Caption is empty. If I succeed that will be a separate blog post. Titles do display (optionally) in macOS photos.

The real strength of Apple Photos.app is making a pile of unorganized photos more valuable to the user. Any active management of Photo metadata or organization, even attaching a comment to a photo, is poorly supported.

I would say the migration succeeded with probably 97% of information salvaged -- at enormous cost. I have thoughts about what this means for, say, Apple antitrust litigation. I believe Apple's de facto monopoly ("Hotel Cupertino") has caused it to lose interest in legacy customers. In retrospect I don't think Apple every worked on an Aperture migration path, I think they just did an iPhoto migration path. Aperture migration was sort-of supported because of the shared iPhoto/Aperture Library format.

Post migration I evaluated Houdah's Photos Workbench. I took me about 5 minutes to go from trial to purchase. With the direction Apple has taken both PowerPhotos and Photos Workbench are necessary purchases. I am also evaluating osxphotos and I will work further on AppleScript supplementation. I believe both osxphotos and Workbench use Apple's PhotoKit APIs.

Historical posts related for shared misery

• 2012 post on iPhoto to Aperture - video issues
• iPhoto 8 to Aperture 3/17/2012
• Aperture wouldn't show iPhoto event/album comments in shared libraries 1/12/2012
• Video format choices sucked in 2011 too. 7/2/2011 (How should I?)

Footnotes

[1] Laughably I chose Aperture because I figured Apple's user base and resources were large enough that if Aperture died Apple would provide a good migration path. So my choice of Photos is very debatable. What we need is a media management solution that's open source and used by multiple commercial vendors. Ask me if you want the requirements and design spec for this :-).

Microsoft OneDrive automatic update disables OneDrive on Catalina and Mojave. Catalina fixed but Microsoft has dropped Mojave support.

My attempt to fix Microsoft Update disables OneDrive on Catalina and Mojave (and perhaps more) is in process.

 Fix from Microsoft Answers

1. Download last working version - Install version 23.002.0102.0004.

2. Disable autoupdate for OneDrive preferably by blocking network access to g.live.com IP 20.205.115.102 Port 443 TCP. Port 443 is standard port for https traffic.

3. Remove current version, install last good version.

As of April 20 we are to watch the release notes page for a possible fix.

May 27, 2023 Update: A possible new fix for Catalina, but it's not working on Mojave.

May 29, 2023: Microsoft has decided it will not fix Mojave and has retroactively dropped Mojave support. From a support email written by a non-native English speaker.

In regard to the issue, we would like to inform you that OneDrive sync support has ended on MacOS Mojave.

It is equally sad for us to deliver such a hard message. However, we recommend you to update your operating system to MacOS 11 (Big Sur) or later. This will also help you to get a more secured Operating System with enhanced features and more security.

You may always use OneDrive on Web ( https://onedrive.live.com ) as a workaround to upload and download files from OneDrive.

It may be possible to install the last good version then block the autoupdate. Personally I'll move to using iCloud rather than OneDrive.

Home sleep monitoring with Apple Watch and Google (Nest) Home Video in adult with limited ability to describe a nocturnal health issue

This post content was moved to:

https://notes.kateva.org/2023/05/home-sleep-monitoring-with-apple-watch.html

Apologies for the error!

iOS App Update hangs without an error message in infinite download: a general approach

Apple's FairPlay DRM management is notoriously fragile. It can be confused by family sharing, Screen Time controls, payment method changes, and, heaven forfend, mixed Apple IDs on a device.

Once Apple's DRM gets confused there's often no user accessible error message (PS. This is a bug [1]). The app just hangs. So when I realized my (manual) App Store updates were not completing I was not completely surprised. Recently I had:

1. Changed payment methods. I made my Apple Card's award balance (1-2% transaction) the default payment method (so it always gets emptied)
2. Enabled Screen Time account change restrictions to mitigate the harm of Apple's biggest current security issue.

I fixed the problem in the usual way (see Apple's article on this as well):

1. [Switch to manual update if you've been using automatic]
2. Verify Apple ID payment methods look correct
3. Turn off Screen Time [Apple doesn't mention this.]
4. Restart phone (power off/on)
5. Download a new free app from App Store [An old method, still useful]
6. Verify I can now update one of the pending apps.
7. Update All
8. Turn Screen Time back on.
9. [Turn auto update back on if you like that.]

-- 

[1] Failure to generate a user notification of a failed interaction is, of course, a bug. Regardless of whether there's a bug in the interaction processing (which there is, so that's another one).

iPhone Recovery Key attack vector kills your iCloud access: Workarounds pending an Apple fix including Apple ID protection

Someone who has your iPhone passcode can lock you out of your Apple iCloud and Apple ID services -- as well as take control of your iPhone and have access to all passwords stored in Apple's Password Manager (iCloud Keychain).

This can happen when someone steals your phone and obtains your passcode by the simple measure of threatening to kill you. Or they might see you enter your passcode or surreptitiously record entry. In bars drugs can be used to facilitate the process. This is often done as part of "borrowing a phone" for an "emergency call". (Never let anyone you don't trust with your life and wealth touch your phone. If it's an emergency make the call for them but ensure they don't record your passcode and don't let go of the phone.)

Once the thief has your phone and passcode they can change the victim's Apple ID password. This prevents the victim from locking the iPhone. The victim could still do the Apple ID password recovery process, so to get more time with the phone the thief can set a Recovery Key. If a Recovery Key exists they can change it. Setting a Recovery Key this way disables Apple ID password recovery. This gives the thief an unlimited time with the phone. It also locks the user out of all their Apple ID associated services and products including video, music, personal photos, personal documents, family sharing, other Apple devices, and the like. From the thief's perspective the Apple ID lock out is merely a side-effect. They may even feel a tiny qualm of sympathy for their victim. They do it to prevent iPhone lockout.

This is an Apple design problem. They need to fix it. Basically the iPhone passcode has far too much power -- especially since it has to be tapped in far too frequently and thus relatively easy to enter. Secondarily the benefits of the Recovery Key are limited to a few people and the with this technique in common use the risks dwarf the benefits. Apple should disable creation of new Recovery Keys immediately while they come up with a better fix.

TidBITS has one of the best descriptions of the problem following a somewhat confused WSJ article. I suggest also reading TidBITs preceding article on the problems with iCloud Keychain.

I was aware of most of these issues, but the Recovery Key hack is new to me. Again, if an attacker has control of your iPhone they can change your Apple ID password, locking you out of your photos, documents, Apple services, Apple media you've purchased, subscriptions, software, and more. At this point you can ordinarily reset your Apple ID password [1] through a tedious series of authentication steps or with the help of a previously specified Recovery Contact [2]. However, if you have set a Recovery Key you can't use these methods. You have to know the Recovery Key. If a thief sets or changes the Apple ID Recovery Key to prevent locking of the stolen iPhone you are truly screwed. Once you set the Recovery Key yourself Apple no longer stores it [3]; they can't recover your Apple ID even if they wanted to.

Apple has to fix several things here. It's insane that a six digit iPhone passcode allows access to all of the iCloud Keychain (Apple Password Manager) and setting up a Recovery Key. The power and risk of the Recovery Key is a separate problem and creation of new Recovery Keys should be disabled until there's a better fix.

In the meantime we've taken two steps on our our iPhones:

1. Emily and I set each other up as Recovery Contacts to facilitate doing an Apple ID password reset in the absence of an Apple Device.
2. Follow the recommendation of TidBITS to use Apple's Screen Time feature to prevent Account Changes. This requires setting a separate 4 digit ScreenTime code (PIN). When you do this Apple seems to require entry of Apple ID credentials that can be used to reset the ScreenTime PIN, but if you tap "cancel" you can continue without this step. That means an attacker can't use the Apple ID credentials they've stolen to unlock the account settings; they can't change an Apple ID password and they can't set a Recovery Key. (I think this can trigger an Apple Bug with App Updates and mixed Apple ID - see this article.)

I have not yet deleted all of my iCloud Keychain entries. I will go through mine and delete a few key ones. Apple really and truly needs to secure iCloud Keychain with an optional separate credential [4].

I do NOT recommend setting a Recovery Key.  An attacker with your iPhone passcode can change it anyway, and you won't be able to use Apple's standard Apple ID password recovery method.

- fn- 

[1] One time I tried to use login with Apple on a calendar service provider (Stanza). Apple evidently decided that was a bad idea and instantly locked my Apple ID. I had to follow the password recovering steps. If I'd set a Recovery Key and did not know the Key I'd have lost access to my Apple ID content (photos, etc) for all time.

[2] Setup a recovery contact NOW.

[3] I presume that when you do a standard password reset, or a Recovery Contact does a password reset for you, that behind the scenes Apple is using the Recovery Key they keep.

Mastodon wishes: topic tags that actually work

The mastodon social network (I'm https://appdot.net/@jgordon) lets me follow people at any Mastodon community (instance). Mastodon is person-centric. Reddit, by contrast, lets me follow activity on predefined topics.

I'd like Mastodon to have better topic support; I'd like to be able to follow both people AND topics.

In theory Mastodon has support for topics through hash tags. In practice, particularly if you are on a smaller Mastodon instance, the tags are not very useful. They only "know" about posts that have been pulled into a user's home instance, most often because someone on the instance follows the post author.

I'd like to see "topic tags" that were predetermined and worked across the Mastodon part of the Fediverse. I imagine a registry of topic tags that's updated by an instance daily based on instance posts using the topic tag. There are likely better models for how to do this.

Apple's App Store Apple Account balance: updated due to transition to Apple Cash

Update 5/1/2023 - a few months after writing this I realized Apple is in the midst of a very poorly documented multi-year transition.

There are currently two "gift cards" - Apple Gift Card (AGC) and App Store & iTunes (ASIGC) gift card.  The AGC used to be only useful for buying things at Apple Stores (or online equivalent) but sometime in the past few years the AGC could be used to to buy apps and media. 

The ASIGC works as below. Apple's check balance advice remains incorrect; the balance displayed in the App Store UI is not updated reliably. However, I don't think the ASIGC is long for this world. Which probably explains why Apple hasn't fixed the balance display or the use of the old term "iTunes".

The AGC can be purchased through a web interface - https://www.apple.com/shop/buy-giftcard/giftcard. When I bought it for my son using his iCloud email the balance showed on his phone wallet as Apple Cash. The Apple Cash balance also shows under his Account as did the prior ASIGC balance, but in a different location inserted at the top of the screen (it doesn't show there on my iPhone!).

The Apple Cash account is also used to hold purchase rebates (1-2%) from an Apple Card. I see my Apple Card balance there. I use the payment method selection control; my first payment method is Apple Cash, the second is Apple Card. When I view my son's payment methods at appleid.apple.com I see Apple ID (that is in fact holding his Apple Cash balance) and the fallback payment is "Apple Cash", but it's MY Apple Cash not his. (In fact there's a 3rd payment method --after these two charges roll over to me via my Apple Card, but the current UI can only show two.)

Note the weirdness here. In the case of a non-organizer family member the Apple Cash balance shows up here labeled Apple ID rather than Apple Cash!

I have found charges do go first against this "Apple ID" (his Apple Cash) then against my Apple Cash (currently $41.71). Incidentally, note if you can connect to a non-child family member's appleid you can see their balance and they can see the family organizer's cash balance.

If my son were a minor I'd have more options to manage Apple Cash. As it is this is a big improvement on a few months ago (below). It's obvious that in the US at least the ASIGC is obsolete and Apple will transition to the AGC. They still have a ways to go; they have added a savings account feature to Apple Card; I wonder if they'll add one to Apple Cash.

--------- original post

Apple's "Apple Account" holds cash that can be used to purchase apps, media and subscriptions. Money is most often added to an Apple Account through App Store and  Gift Cards iTunes gift cards. Users can also directly add money to their personal Apple Account from a payment method, but there are few times that makes sense. If a user is a member of a "Family" then the money comes from the Family Organizer's payment method (usually this is a bad thing). If a Family Member purchases something it will come out of their Apple Account balance first then any residual charge will come out of the Family Organizer's payment method (not the Family Member's payment method).

Apple Accounts are poorly documented, especially when they intersect with Family Sharing. Sometimes the support documents are incorrect or incomplete. For example, the check balance article for Mac tells users to look below their name in the App Store app:

That doesn't work very well though. You can see the problem in this screenshot taken from my son's account

His account shows $150 as a balance, but that's wrong. If you click on Profile and drill down to this Accounts page (requires authentication) you will see the correct amount of $135.37. Evidently the amount displayed on the App Store screen is copied there from another system and there's a time lag. In my testing I've found that the lag is at least a day and I suspect it only updates when one checks the Apple Account (requires authentication). So, in reality, the Apple Account is the only way to know this number.

Apple doesn't mention this, but you can also get to this Accounts page (which has the accurate numbers) from iTunes/Music. You can't get to it from the web however; appleid.apple.com doesn't have this data. My guess is that Apple is still using their 20yo iTunes infrastructure for the "Apple Account" (authentication doesn't support Apple Passwords OR biometrics) and that the display in the App Store is a bit of a hack. 

The Apple Account is a legacy system that is much older than Family Sharing and doesn't support it very well. I'm guessing Apple has been trying to replace the iTunes backend for a years and that the version we see is in maintenance mode. Perhaps they will transition to the emerging Apple Pay infrastructure. For now we have to workaround the issues.