I had a longish holiday break this year, so naturally I spent part of it fighting with another technology regression.
This regression was about getting photos from my Mac to my iPad; I use the iPad as a digital photo frame randomly displaying my Aperture (running on Sierra) images tagged as “slideshow” (this drains the battery fairly quickly, so I usually have the iPad charging). Years ago this was painless, but over the last few months I’ve found that only a fraction of my tagged photos make it over to the iPad. Often the process seems to die.
This go round I tried an old Aperture tip to delete the iPod Photo Cache (Apple approved) [1]. That had one effect — image transfer died completely! Instead of telling iTunes to transfer all my Slideshow tagged images (9,019) I had to select year-specific smart album slideshows one at a time (incremental add):
That generally worked, though the ghostly counts shown above in the iTunes 12.7 UI came and went inexplicably. Sometimes a mouseover seemed to trigger an appearance. If they didn’t show up that was a clue that something had gone wrong with the sync. (Infamously iTunes does not log errors to Console. Perhaps because they’d overwhelm console?)
Some part of the OS (or Aperture?) has to regenerate the .ithmb files [2] that are shared, that seems to be a slow process. I think the build process might make use of some data Aperture saves on exiting; that data is also used by the old OS X media browser (itself related to Apple’s old iLife code) [3]. I’m pretty sure some of the .ithmb creation is by an OS thread thought, and that seems to have a low priority. Maybe part of what causes a complete failure with large numbers of files is that iTunes has an arbitrary timeout placed to work around some other iTunes bug [4].
Once I broke down my slideshow into smaller bits and added them incrementally I seemed to have some success. The count (erratically) displayed in iTunes matched what showed on the iPad album counts and the Aperture smart album counts. Then things went south. The numbers transferred started to be a fraction of the Aperture counts. When things settled out I seemed to be stuck at 7,811 on the iPad (same count in iTunes if I mouseover the iTunes content type distribution bar) out of 9,109 in Aperture [5].
I did some half-hearted testing to try to figure out if this was related to filename characters or the like, but I haven’t gotten very far. I did confirm that RAW files don’t have an obvious problem.
The great news is that I learned today that I’m not alone and it’s not an Aperture specific problem! There’s a Macintouch thread on this by riley (quotes others, emphases mine):
So here's an update on the issue with syncing large Photos databases between iOS devices and iTunes.
After posting a number of followups to my Apple bug report, the response I got back from Engineering was that this is a known limitation in iOS, and "we are working to improve this in a future iOS release."
Their suggestion was to sync by starting with just a few albums, and then repeatedly re-sync, adding a few more albums each time. "On subsequent syncs, iOS will not add additional photos unless the sync preferences in iTunes are different from the previous sync, which is why it is necessary to keep adding selected albums each sync, instead of all at once."
… There is a huge discussion of this over on the Apple forums (see “Photos app on iPhone not syncing in iOS 11 ").
Various people have been told various things. Some have been told it is an iOS issue, some an iTunes issue. Personally, I think there are issues in both that are interacting, which are preventing reliable syncs with large photo albums over USB between iTunes and iOS devices.
Two years ago, our then ~40k photo library in Photos would sync the selected items (maybe around 30k photos in various albums) without problem. Perhaps around the time of iOS 10, the syncs needed to be restarted to complete, but eventually it would work. Since iOS 11, some sync, but many do not, and the sync just cancels with a variety of error messages at various points. Sometimes it is a device error, sometimes nothing. Sometimes many of the photos will be on there, but the albums they should be associated with are empty…
… The last update I heard from the person who was helping me was that it was an iTunes issue (at minimum, perhaps iOS too). She said that the Apple engineering team is aware of the issue and working on finding the problem so it can be fixed.
… I also see that since the latest tvOS update and the latest iOS 11 update, many albums are no longer being shared with the Apple TVs.
Hmm. This suggests to force updates one should add or remove some album with every sync. That did not work quite as expected. After I tried using the Media Finder code [3] and the ‘add new album’ trick iTunes said it was synchronizing another 1000+ files (that’s good!) but then the counts in iTunes and iPad dropped from 7,811 to 7303. So that’s just great.[6]
Since I now know this is not just an (abandoned) Aperture problem I added an iTunes feedback comment about the problem. Rumor is Apple processes these, presumably doing some algorithmic text extraction based on volume. So add your voice if you’re interested.
- fn -
[1] It took hours to empty the trash after deleting iPod Photo Cache — despite files being on an SSD. As of 7/2020 the links are all broken, but this is what you do: "find your Aperture or iPhoto library in the Finder. Right-click and select “Show Package Contents” then find “iPod Photo Cache” and delete it."
[2] I changed extension of a copy to .jpg and it rendered as expected. They are fairly high res images, in a small sample I examined one was a 3MB jpg.
[3] In Sierra’s Open dialog the Media Finder still shows in the left nav pane! Maybe invoking it might trigger some updates? Worth a try to open it and let it run for a while.
[4] It’s widely assumed that iTunes source code is an impenetrable mess with bits left over from pre-Apple days.
[5] Incidentally “Stacks” are a problem. One of the many bad things Apple did in bridging iPhoto and Aperture was to treat an old iPhoto Edit as an Aperture Stack. This threw away key information — that one image was the new version of another. There are lots of downstream implications. Years later I’m recreating thousands of iPhoto edits in Aperture. One of the problem is that when you tag an image as “slideshow” then transfer to iPad both versions go. I think this can cause the iPad count to be higher than the Aperture count, but of course the problem I haves is in the other direction.
[6] I removed the extra album and then the count went UP to 7,811 again! That album did reference photos that were in other albums, so maybe part of the bug has to do with photos that are in more than one album. I do think there are multiple bugs here though.
Update 12/31/2017 I
There are some JPEG images in my older files that do not show up in either OS X Media Browser (old iLife code) or Sierra’s Screen Saver Library browser and do not transfer to the iPad via iTunes. However, if I export the original and drop it back in again then the image appears normally — this doesn’t appear to be problem with the JPEG. They also render normally in several different apps and EXIF viewers.
Rebuilding the Aperture database does not make a difference.
So this is curious, and potentially an important clue, but to go further I need to understand how Media Browser works. In my case the bug doesn’t only impact image transfer to my iPhone; it impacts Media Browser as well. I’m now poking around at ~/Library/Application Support/iLifeAssetManagement and ~/Library/Application Support/iLifeMediaBrowser, especially ~/Library/Application Support/iLifeAssetManagement/state/albumshare/Daemon.sqlite.
Update 12/31/2017 II
Finding more “invisibles” but no clear pattern to what’s omitted. Some are quite small - from the days when 100K was a good image size. It’s not size alone though; of a pair of similar old images a 64kb one made it across and a 70k one didn’t.
Update 1/1/2018
Same sync, same everything — but now iPad acts as though it’s been syncing to a different computer. All images removed! It took several tries to get one album to sync, but after it started working I could do about @1,200 at a time. Now have about 7,700 on iPad out of 9,100 on Mac. I won’t sync iPad to iTunes again until there’s word of a bug fix. Just stick with what’s on there now.
Update 6/9/2018
I thought this was fixed — but it’s broken again. On latest versions of iTunes and iOS 11.3.1, macOS Sierra.
Update 6/9/2018b
I was able to sync 7777 of 7771 (different apps count photos differently, not least due to Aperture’s brain-dead “Stacks” handling of iPhoto imports) photos after installing iOS 4. I don’t know if the bug is fixed or if it’s just that process of installing an iOS update cleared something out. In the latter case it might return.
Update 7/4/2020
It's been a year or two since I last tried syncing. At this time I'm still on Aperture and Mojave. The sync was worse than ever. I wiped my old iPad and restored from iCloud backup and sync actually seemed to work! It saw only 7992 of the 9413 images in my Aperture slide show, but it did import those.
I tried the trick of deleting the iPad Cache in Aperture [1] but on resync it was still 7992/9413. Consistent at least.
I think there's a bug in Aperture with Sierra and later with generating the images in the iPod Cache. Maybe something iTunes or OS X was supposed to do.
I suspect the iPad flash memory is also in bad shape.
The lack of logging or error handling is appalling.
