Google sent my family into email Hell. This is how we climbed out.

… we’re living in a time when algorithmic software is just good enough to eat the world and still bad enough to be endlessly frustrating. (Daniel Genser, via Twitter)

A few days ago I sent an email to Emily and a few seconds later this came back:

Mail Delivery System <MAILER-DAEMON@homiemail-mx1.g.dreamhost.com>
Tue, May 8, 9:52 PM (4 days ago)

This is the mail system at host homiemail-mx1.g.dreamhost.com.

I'm sorry to have to inform you that your message could not
be delivered to one or more recipients. It's attached below.

…

                DreamHost Email Support

   <emily@googleappdomain.com> (expanded from <emily@redirectemail.com>): host
    ASPMX.L.GOOGLE.com[173.194.202 .27] said: 550-5.7.1 [208.97.132.209      12]
    Our system has detected that this message is 550-5.7.1 likely unsolicited
    mail. To reduce the amount of spam sent to Gmail, 550-5.7.1 this message
    has been blocked. Please visit 550-5.7.1
    https://support.google.com/mai l/?p=UnsolicitedMessageError 550 5.7.1  for
    more information. q3-v6si11849599pgn.272 - gsmtp (in reply to end of DATA
    command)

Google had sent me to email Hell. Everything I sent to Emily was rejected. It was the same story with emails sent to my children. Whatever I sent, Google rejected.

I think the problem, as far as Google was concerned, was that I was using redirects. This is old net tech. If you own a domain, say “kateva.org”, your hosting service will usually let you define as many redirect “email addresses” in the domain as you might want. They aren’t real email addresses though, they’re redirects. You have to define a true email recipient for each one.

Google didn’t see the email I sent Emily as coming from my personal gmail account, it saw it as coming from the SMTP service that works with my domain (which I’ll call “kateva.org"); in this case homiemail-mx1.g.dreamhost.com. It appears either my domain, or that sending service, had bad Google-karma [1]. Google sent the rejection to “homiemail” (I think!) and homiemail sent it to me.

We were in trouble. It was likely that email sent to our family members was going to be bouncing back in a very confusing way.

I tested sending from various email addresses and using various software (web client vs. iOS Mail, etc) to see which was rejected and which accepted:

Sender	Redirect	Result
GSuite (web client)	Yes	Success
Yahoo (web client)	Yes	Success
Gmail (web client)	Yes	Success
GSuite (macOS mail, iOS mail)	Yes	Success
Gmail (macOS mail, iOS mail)	Yes	FAIL
Gmail (macOS mail, iOS mail)	NO	Success

It turned out my mail was only hitting Google’s spam threshold when I sent it from my personal Gmail (not GSuite email) using iOS or macOS Mail. Everything else worked, including using my personal Gmail by web interface.

No, I don’t understand what’s happening here. I can speculate that Google rates sending services roughly as follows: GSuite_web > Gmail_web, Yahoo_web > GSuite_client > Gmail_client. The combination of Gmail_client (macOS Mail) and the redirect dropped my email score below Google’s internal cutoff.

There’s not much to do about this. It’s not like Google is going to help. I was on my own and we were in trouble.

The first thing I did was change the redirects to send all the mail to an old school Dreamhost local webmail box. That stopped the bouncing. I could forward from that box to my gmail and reroute important emails manually to other addresses for Emily and the kids.

Next I discovered, contrary to their documentation, that Dreamhost had not enabled DKIM or SFP on my domain. I fixed that (separate post pending) to see if it would improve the reputation score for my domain but it had no effect. I suspect the reputation that mattered was that of homiemail-mx1.g.dreamhost.com.

That left me with these options …

1. Setup Dreamhost mailboxes for each person and create redirects to them.
1. Set up a forward from the DH mailbox to another of our family GSuite emails (I picked several Google Apps/GSuite things when they were free)
2. Have our family GSuite emails do POP retrieval from the DH mailbox.

Dreamhost recommended the 1.2 (second). But I had a third option…

… even though I’d not used it, the problem domain (“kateva.org”) had an old legacy GSuite. I went into that GSuite, defined an account for Emily and kids, and then switched Dreamhost MX to use the GSuite email rather than DH email [2]. I had lost some faith in DreamHost by that point and I figured that since Google rules the net I was better off inside their castle.

And that’s where we are at the moment.

One more weird thing. For 2-3 family members I received a notice that a non-apps Gmail account already existed for them. I didn’t create those accounts but maybe my kids did? (It’s complicated, but somehow when I did Google Voice for them it did … something … brain shutting down ….) Apparently Google lets one do this! They wouldn’t have received email though, DNS records were using DreamHost MX redirects. If you own a domain, and create Google Apps account that matches the existing address, it gets renamed to something like “username%domainname@gtempaccount.com” email. Wow. [3]

It’s Google’s world, we just play in it.

Oh — and don’t use redirects. Google doesn’t approve.

- fn -

[1] Dreamhost denies that they’re in any kind of trouble with Google — but really, how would anyone know. Maybe it was my domain that Google didn’t like — we get a lot of spam and in this setup all that spam would seem to be coming from my domain.

[2] This wipes out all the past redirects. GSuite supported aliases so I moved them into there. Fortunately I’d saved the redirect records before making the MX switch.

[3] Google has an article on conflicting accounts, here is how they are resolved. I think Google Voice was the primary route for creating conflicting accounts — an artifact of how Google switched Voice credential systems post acquisition. 

Appigo Todo Cloud.app - don't forget to unsubscribe when you leave

Even since iOS 11 editing in Appigo’s Todo Cloud.app has been buggy for me on iPad and iPhone alike. Just aggravating. Feels like they failed to revamp something. I wasn’t delighted with their sync technology, but I could live with that. The editing bugs finally broke me.

So I decided to exit. Somehow I remembered Todo Cloud is a subscription service. I found my way to the somewhat hidden account settings and disabled premium. Turns out that turns off auto-renewal. 

Hope it really works.

Appigo is a textbook example of how subscription solutions can disappoint. They never provided a good export strategy, so there’s a strong data lock. Then they failed to do minimal maintenance but continued to collect subscription revenue and sell the app.

So what will the replacement be? I’d like a product that

1. Did what Appigo ToDo Cloud did but actually worked
2. Had a web client as well as Mac, iPad and iPhone client
3. Supported family sharing
4. Had good data export (exit strategy).

I evaluated Things.app and OmniFocus. Things got #1 and 3. OmniFocus got #1 and 4. Neither got #2.

Hmm.

On the other hand, Reminders.app for iOS got #2 and #3 and it’s free. So it’s weirdly in contention.

For now I’m using Reminders.app for tasks and Trello for projects. I manually copying over tasks that had dates, it’s not too bad. The backlog of ‘someday’ tasks I’ll gradually slog away at.

If Things gets some data export I’ll probably buy it, but it’s expensive since it’s not a universal app. If OmniFocus gets family sharing I might buy it. Meanwhile I’ll see what I can make Reminders do.

I think this is my first significant iOS functional regression.

Update 5/22/2018

I just discovered I wrote about this in 2011 …

There are no great task managers for the iPhone - but there's hope for 2011

… Neither Things, nor Appigo’s ToDo.app (which I have used incessantly since 2008), nor OmniFocus, nor Remember the Milk.app nor Toodledo.app are a great solution. They all fall short…

Seven years later and ToDo.app is moribund (I’ve been using it for 10 years!) and both OmniFocus and Things are still flawed.

Since I first wrote this I’ve run into issues with Reminders.app — including sync bugs and even text editor bugs. On the other hand, OmniFocus is promising a web client. I’m going to transiently switch back to ToDo.app and see if WWDC providers some kind of family sharing for subscriptions. That would make OmniFocus pricing less extreme.

MarsEdit - don't enable all post sync if you have a very large number of posts

MarsEdit 4 added a new feature — the ability to sync all posts. It didn’t work for my Blogger posts, somewhere around post 3,000-4,000 Google Blogger dies. It looks like a Google bug.

It did work for kateva.org/sh though — that has over 30,000 posts (it mirrors my pinboard shares and twitter tweets). Unfortunately that slowed MarsEdit launch severely — it took up to 3-4 minutes to start. Turns out MarsEdit loads all posts into memory and it doesn’t scale to that size.

I reset the kateva.org/sh sync to ‘last 50’ and startup is ok now.

Two possible bugs in iOS 11.3 to watch out for: enable purchases by Touch ID, change iTunes & App Store account

Our kids' phones were setup to use my Store Apple ID (it’s an old practice — see a blog post about undoing this). The phones were set to require the Store Apple ID password prior to purchase. When our kids wanted to buy we’d enter the Store Apple ID password for them. Touch ID was not enabled for purchases. 

After the 11.3 update two of the phones had Touch ID enabled for purchases. Normally you need to enter the Apple Store ID to enable this. One child (special needs) ran up a $10 bill which he paid for. (He gleefully shared his new power with us.). On one affected phone I turned it off again, when I turned it on the Store Apple ID was requested as expected. (Another phone I migrated to using its own Store ID, a third device wasn’t affected.)

On 1 of the 3 phones the Store Apple ID was changed to the user’s iCloud Apple ID. The problem was recognized when my daughter was unable to view movies not on her phone — it took me a few minutes to figure out what happened.

It’s interesting how much behaviors varied between the 3 devices.

Why you should change your router DNS to Cloudflare and Quad9

I wrote this one for Facebook friends - reposting here. The TidBITS review is excellent.

—

This is roughly how things work ...

1. Someone in your home visits a certain web site.
2. Your home internet provider (ex: Comcast) associates that site with the identity of the Comcast account holder. They sell that information to the universe. Facebook buys it.
3. Facebook shows the Comcast account holder ads based on the visited site.

This happens because your internet provider is your default internet directory service (DNS provider) - Comcast knows all the sites you visit and (as best we can tell) they sell that information.

It's basically an internet design flaw.

If you're a bit geeky you can change the DNS settings in your home router to someone more trustworthy than Comcast (which would be anyone else, really). In our home that's an Apple Airport. I have our settings below but your settings will vary.

We use:

Cloudflare: 1.0.0.1 (their 1.1.1.1 address doesn't work with some ISPs)
Quad9: 9.9.9.9 (as backup)

For more information see an excellent TidBITS review. If you truly want privacy, particularly on iOS where DNS settings are a pain, you need to use a trustworthy VPN (see below).

—

PS. It’s easy in macOS Location settings to experiment with different settings. This is particularly important for a laptop that moves between locations. On my standard macOS Location I use Cloudflare, then Quad9, then Google. Sadly iOS DNS settings are a mess (per TIDBits):

In iOS, DNS server settings tend not to work the way most people would want them, which is as in macOS: setting the details once and having them work on every network to which you connect. The settings have to be set for each network. Worse, we’ve found in our testing that after changing DNS values, the settings revert to Automatic and the server IP addresses we entered are tossed. There’s also no way to set DNS servers for cellular connections.

In iOS you need to use trusted (not free!) VPN provider. I use TunnelBear’s free GB option, I wish I could buy blocks of data from them rather than yet another subscription. If you use a limited data capacity VPN on a Mac you should use TripMode to reduce background data use.

Update 4/22/2018: A Gizmodo article reminded me why this DNS control is more important now ….

“The FCC under Chairman Pai changed the rules in the United States for ISPs allowing ISPs to start selling your browsing history to target advertising against you,” Prince said…

They mention two popular VPNs - ProtonVPN and Private Internet Access. Curiously PIA claims TunnelBear does not protect me — which makes me a bit suspicious of PIA. 

Converting from shared store ID to Family Sharing - and what didn't work

Maybe this worked. Or not. See update.

Our five family members have long shared one store Apple ID. We’ve done this before there was Family Sharing. I put off switching to Family Sharing as I figured it would take Apple 3-4 years to get it working.

With iOS 11.3 Apple broke a longstanding purchase behavior. My son’s iPhone no longer required a password for purchases, only his fingerprint. There might be a fix, but I decided instead to move him to Family Sharing. (There is a fix, see below.)

The story went something like this [1]:

• I have an iCloud Apple ID (john.___@icloud.com) and a different Store Apple ID (j____@mac.com) — because I’m old. He has an iCloud Apple ID (sam.___@icloud.com) and my store Apple ID.
• In my iCloud Apple ID he is a family member. 
• I removed my Store Apple ID from his phone and added his iCloud Apple ID.
• I sent $15 to his iCloud Apple ID from my App Store account.

So far he still can access our movies and apps. Now he will make his own purchases that will be associated with his Apple ID. When he runs through his $15 he’ll give me cash and I’ll send more money. Eventually I do need to get a debit or managed credit card on his phone but we’ll start with cash. Alas, it doesn’t work that way. See update.

After the change I checked the (this is broken) two places Apple currently tracks devices associated with an Apple ID

• appleid.apple.com/account/manage: showed 7 devices including an old iPhone my son used to have that I’d previously removed. This also showed on his iPhone Apple ID view. I removed it from both places and it has not returned.
• iTunes Manage Devices showed 8 devices, but not my son’s current iPhone. This, in contrast to past testing, is correct while the appleid.apple.com list is incomplete. It’s interesting that moving my son’s phone to Family Sharing means I’m no longer at my 10 item device limit (if that rule still applies!)

- fn -

[1] He is, incidentally, a special needs adult. I’d have liked to be able to use Apple Ask to Buy for him but that’s not available for an adult. (I wish Apple considered special needs as a disability — they have great support for visual and auditory needs, but not for cognitive.)

Update 4/19/2018

• Seeing purchase histories is really clunky. You can see what apps a family member has purchased by launching App Store.app, logging out and then logging in as the family member. To see both tunes and apps you go to Apps & iTunes in Settings (yeah, this is crazy). You have to log in as the family member — I got the ancient iOS 1.0 un/pw dialog that shows up when you get to a part of iOS that desperately needs a replacement. It did work, but seriously ugly.

Update 4/20/2018

• Subscriptions aren’t Family shareable. So that’s a significant bummer; several of his apps are subscription based. All is not lost though, At Bat.app presented my Store Apple ID username and accepted the password. In-App purchases aren’t Family shareable either — which is bad news for Omni Group. Apple has a list of what’s not shared.

Update 4/28/2018 - what I wish I’d known

My son ran up a $70 bill on a $15 credit — all on my account — because “Any time a family member makes a new purchase, it’s billed directly to the family organizer’s account”. It doesn’t work the way I thought it did. If a family member is under 18 you can activate Ask to Buy, but not for someone over 18.

Family sharing is clearly designed to only work for children. It’s a poor match for a couple that wants to keep separate finances and it’s unsuited to adult children.

I found that the 11.3 update bug didn’t truly break the ability to require an iCloud password for purchases. It only bypassed the requirement to enter the iCloud password to enable Touch ID. I went into Touch ID & Passcode and turned off “USE TOUCH ID FOR … iTunes & App Store”. 

He doesn’t know his iCloud password (so he can’t lose it in a phishing attack!), so this meant he again needed us to enter a password into his iPhone to make purchases. Obviously, Ask to Buy would be far better. If Apple wanted to support users with cognitive disabilities …well, this blog accepts comments. I’d be glad to advise.

We didn’t want to have to memorize another password, so I changed his iCloud password to match my App Store & iTunes password.

"This item was not added to your iCloud Music Library because an error occurred"

This is why Apple Stores are overloaded. I have 254 items like this. No explanation, no hints on fixing it.

Low quality is expensive.

PS. In this particular case it looks like iTunes and iPhone supported .mp4 audio, but iCloud does not. Long ago I ripped these and the software did .mp4.

I found the on drive files and used quicktime player to export as .m4a — a lossless transform that strips out the some of the .mp4 wrapper. Then I deleted originals in iTunes and added these back in.

iPhone aggravation: apps don't appear in Settings:Cellular until they actually use cellular data

Apple could have displayed all iPhone apps in settings:cellular whether or not the app uses any data. Then we could disable cellular data access without having to first use data.

They didn’t. Apps only show up there after they use cellular data.

This is really annoying when trying to stay under the data cap of #2’s $40/year mobile plan. It also confuses the heck out of customers.

I don’t think this will get fixed those. Apple’s technical debt fix list is deeper than Valles Marineris.

Apple has a new problem with DRM and device management

Today one of the family iPhones died. I went to remove it from our quota of devices (you can have a maximum of 10 devices associated with a family account) in iTunes Mange Devices.

I couldn’t. 

There’s a 90 day time limit to change associations, which I don’t recall being enforced for removal, but here you go…

Except it’s not 90 days, because the grayed out non-removable devices were associated as long ago as May 2016.

Things are broken in two ways.

1. What does Apple want us to do with a wiped or lost or broken device?

2. The items I can’t remove are years old.

PS. Yeah, I hate Apple too. But really, everybody does.

Update: I reviewed Apple’s support document. If you have a working device you can remove the device from the DRM control list — but only through one very obscure screen. Logging out of iTunes doesn’t do it. Otherwise device removal requires iTunes, which, for me yesterday, showed this error.

Today I rechecked, and all the devices with “1 day remaining” are still “1 day remaining”. It’s broken.

Once this type of blunder would have been a bit of a deal, but now we’re so numbed by Apple’s quality collapse even I can’t put much energy into it. All the money in the world can’t replace culture, and Apple’s culture is broken.

Update: Added to Apple Discussions, asked @AppleSupport on Twitter.

Update 3/23/2018. On my second Apple Support call the “Senior Advisor” and I found a fix. We think iTunes, or the database it accesses, is broken/deprecated. From my Apple Discussion post:

We have a fix. On my second try with Support I called iCloud support and was escalated to the "Senior Advisor" level. Andrew and I worked the problem and found that you now need to work with https://appleid.apple.com/account/manage. There's now a section called Devices that lists devices signed into. In my case it listed all 10 devices that use the same iTunes Apple ID, so by "signed in" it means "signed in with Apple ID for iTunes/DRM".

Click on device and you get a remove option.  If the device is in use and signed in then it may reappear. You can restore a device that you have removed by signing out of the iTunes Apple ID, then signing back in again.

The iTunes Manage Devices (Account:View My Account:iTunes in the Cloud:Manage Devices) screen did not update after doing this, it still showed the device I removed. I think it’s mostly broken. (Mostly, because I was able from there to sign my iPhone 8 out of iTunes and that reenabled the Remove button, albeit with the broken ‘1 day remaining’ screen, and after signing it back in the Remove button is still active.)

PS. It’s not clear if Apple is still using the 90 day limit for switching Apple IDs. It doesn’t show up in the new iCloud UI.

PPS. Maybe iTunes in the Cloud is using a different database than iCloud to manage DRM, and that the two databases are supposed to synchronize. The iTunes database may be on the way out, so it didn’t get updated when it needed to be…

You can drag and drop some links into a Google Doc and create a hyperlink text

This is one heck of a time saver. Didn’t realize Google Docs allowed this.

Error in Dreamhost WordPress one-click installs - missing code in .htaccess causes 404 errors

I’m far from a WordPress expert. I’m not even a WordPress fan — it’s too complex and powerful for what I do. That complexity means it is also very vulnerable to attack.

So take what I write here with some doubt.

As best I can tell the default Dreamhost One-Click installation of WordPress has an error. At least it did for me. My hpmtb.org site was giving 404 errors. I installed the Redirection plugin and it wasn’t doing anything.

A blog post had a fix that worked for me. I used Transmit for macOS to open the install folders .htaccess file and added this code:

# BEGIN WordPress
<IfModule mod_rewrite.c>
RewriteEngine On
RewriteBase /
RewriteRule ^index\.php$ - [L]
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteRule . /index.php [L]
</IfModule>
# END WordPress

The 404 error is gone now and I suspect the Redirection plugin will work if I need it. I have a hunch that using “pretty permalinks” (not the default of “id=“) might also fix this file — based on some documentation here: 

WordPress uses this file to manipulate how Apache serves files from its root directory, and subdirectories thereof. Most notably, WP modifies this file to be able to handle pretty permalinks.

This page may be used to restore a corrupted .htaccess file (e.g. a misbehaving plugin).

Did I mention that WordPress is too complicated?

The state of iOS parental mobile device management is bad

I recently reviewed Qustodio and found it unusable because the mandatory VPN is not compatible with modern encrypted connections (https).

After that I revisited MMGuardian. Things there are almost as bad. They haven’t implemented password security on their MDM profile; the vendor claims Apple doesn’t support the functions they rely on with a locked profile.

I also learned that MMGuardian’s primary app control, which is to hide all non-default apps, also deletes their folder and icon arrangements. So when they are restored they are no longer in their original locations.

MMGuardian can’t report on device usage, probably because it doesn’t have a VPN option.

Lastly, while MMGuardian can hide Safari, that’s a binary setting. It can’t schedule Safari to be active or disabled the way it can schedule other apps. 

In my 2016 review Qustodio and MMGuardian were the only candidates to meet basic requirements. As of 2018 there are no longer any useable iOS parental control applications.

So now we wait to see if Jana Partners and the California State Teachers Retirement System can push Tim Cook to add remote MDM to iCloud. I doubt they’ll succeed. Next it’s up to the EU.

Update: Apple introduced a Family page and branded it as doing something in the general direction of supporting vulnerable users. That’s not a positive sign.

Qustodio parental control software for iOS is obsolete.

I did a review of Qustodio’s parental control product for iOS on my special needs blog. It wasn’t a positive review. The product routes all traffic through their VPN — and the VPN can’t handle SSL traffic. That might have been acceptable in 2010, but it won’t work today.

I’m feeling grumpy about the time I spent finding this out — not to mention the $10/month subscription I paid for. The lost time was the bigger deal though. The reviews I’d read led me to think the base product worked, so I spent time checking out other features.

Qustodio must know about the SSL issues, but they’re continuing to sell the service. That’s not nice.

Salvaging the wreck of iOS Podcasts.app

iOS 11 Podcasts.app isn’t as broken as iTunes mediated iPad photo transfer or the US government, but it’s not far behind. The current version is not intended to be used with iTunes. It is designed for people who listen to Podcasts they way we used to listen to radio. Pick a podcast or collection of podcasts (a “station”) and stream it over always-on cellular.

That’s not how I listen to Podcasts. I select a topic of interest and learn from it. I used to use iTunes to organize my subscriptions and local stores; smart and manual playlists queued up my programs. In iOS 11 Apple broke all that.

Here’s what I do now. 

I disabled iCloud sync in both iTunes and iOS.

In iTunes I select about 10-20 episodes from my queue. I push them to my iPhone from iTunes. In Podcasts.app I enable the ‘Downloaded’ episodes menu. That’s all I use now. I pick episodes off that queue. One at a time, because play-next doesn’t work any more. iOS Podcasts.app deletes an episode after I listen. Every week or so I add another 10-20.

It’s very simple, it’s very stupid, it’s Apple 2017.

iTunes Mac to iPad photo sync is broken

I had a longish holiday break this year, so naturally I spent part of it fighting with another technology regression. 

This regression was about getting photos from my Mac to my iPad; I use the iPad as a digital photo frame randomly displaying my Aperture (running on Sierra) images tagged as “slideshow” (this drains the battery fairly quickly, so I usually have the iPad charging). Years ago this was painless, but over the last few months I’ve found that only a fraction of my tagged photos make it over to the iPad. Often the process seems to die.

This go round I tried an old Aperture tip to delete the iPod Photo Cache (Apple approved) [1]. That had one effect — image transfer died completely! Instead of telling iTunes to transfer all my Slideshow tagged images (9,019) I had to select year-specific smart album slideshows one at a time (incremental add):

That generally worked, though the ghostly counts shown above in the iTunes 12.7 UI came and went inexplicably. Sometimes a mouseover seemed to trigger an appearance. If they didn’t show up that was a clue that something had gone wrong with the sync. (Infamously iTunes does not log errors to Console. Perhaps because they’d overwhelm console?)

Some part of the OS (or Aperture?) has to regenerate the .ithmb files [2] that are shared, that seems to be a slow process. I think the build process might make use of some data Aperture saves on exiting; that data is also used by the old OS X media browser (itself related to Apple’s old iLife code) [3]. I’m pretty sure some of the .ithmb creation is by an OS thread thought, and that seems to have a low priority. Maybe part of what causes a complete failure with large numbers of files is that iTunes has an arbitrary timeout placed to work around some other iTunes bug [4].

Once I broke down my slideshow into smaller bits and added them incrementally I seemed to have some success. The count (erratically) displayed in iTunes matched what showed on the iPad album counts and the Aperture smart album counts. Then things went south. The numbers transferred started to be a fraction of the Aperture counts. When things settled out I seemed to be stuck at 7,811 on the iPad (same count in iTunes if I mouseover the iTunes content type distribution bar) out of 9,109 in Aperture [5].

I did some half-hearted testing to try to figure out if this was related to filename characters or the like, but I haven’t gotten very far. I did confirm that RAW files don’t have an obvious problem.

The great news is that I learned today that I’m not alone and it’s not an Aperture specific problem! There’s a Macintouch thread on this by riley (quotes others, emphases mine):

So here's an update on the issue with syncing large Photos databases between iOS devices and iTunes.

After posting a number of followups to my Apple bug report, the response I got back from Engineering was that this is a known limitation in iOS, and "we are working to improve this in a future iOS release."

Their suggestion was to sync by starting with just a few albums, and then repeatedly re-sync, adding a few more albums each time. "On subsequent syncs, iOS will not add additional photos unless the sync preferences in iTunes are different from the previous sync, which is why it is necessary to keep adding selected albums each sync, instead of all at once."

… There is a huge discussion of this over on the Apple forums (see “Photos app on iPhone not syncing in iOS 11 ").

Various people have been told various things. Some have been told it is an iOS issue, some an iTunes issue. Personally, I think there are issues in both that are interacting, which are preventing reliable syncs with large photo albums over USB between iTunes and iOS devices.

Two years ago, our then ~40k photo library in Photos would sync the selected items (maybe around 30k photos in various albums) without problem. Perhaps around the time of iOS 10, the syncs needed to be restarted to complete, but eventually it would work. Since iOS 11, some sync, but many do not, and the sync just cancels with a variety of error messages at various points. Sometimes it is a device error, sometimes nothing. Sometimes many of the photos will be on there, but the albums they should be associated with are empty…

… The last update I heard from the person who was helping me was that it was an iTunes issue (at minimum, perhaps iOS too). She said that the Apple engineering team is aware of the issue and working on finding the problem so it can be fixed.

… I also see that since the latest tvOS update and the latest iOS 11 update, many albums are no longer being shared with the Apple TVs.

Hmm. This suggests to force updates one should add or remove some album with every sync. That did not work quite as expected. After I tried using the Media Finder code [3] and the ‘add new album’ trick iTunes said it was synchronizing another 1000+ files (that’s good!) but then the counts in iTunes and iPad dropped from 7,811 to 7303. So that’s just great.[6]

Since I now know this is not just an (abandoned) Aperture problem I added an iTunes feedback comment about the problem. Rumor is Apple processes these, presumably doing some algorithmic text extraction based on volume. So add your voice if you’re interested.

- fn -

[1] It took hours to empty the trash after deleting iPod Photo Cache — despite files being on an SSD. As of 7/2020 the links are all broken, but this is what you do: "find your Aperture or iPhoto library in the Finder. Right-click and select “Show Package Contents” then find “iPod Photo Cache” and delete it."
[2] I changed extension of a copy to .jpg and it rendered as expected. They are fairly high res images, in a small sample I examined one was a 3MB jpg. 
[3] In Sierra’s Open dialog the Media Finder still shows in the left nav pane! Maybe invoking it might trigger some updates? Worth a try to open it and let it run for a while.
[4] It’s widely assumed that iTunes source code is an impenetrable mess with bits left over from pre-Apple days. 
[5] Incidentally “Stacks” are a problem. One of the many bad things Apple did in bridging iPhoto and Aperture was to treat an old iPhoto Edit as an Aperture Stack. This threw away key information — that one image was the new version of another. There are lots of downstream implications. Years later I’m recreating thousands of iPhoto edits in Aperture. One of the problem is that when you tag an image as “slideshow” then transfer to iPad both versions go. I think this can cause the iPad count to be higher than the Aperture count, but of course the problem I haves is in the other direction.
[6] I removed the extra album and then the count went UP to 7,811 again! That album did reference photos that were in other albums, so maybe part of the bug has to do with photos that are in more than one album. I do think there are multiple bugs here though.

Update 12/31/2017 I

There are some JPEG images in my older files that do not show up in either OS X Media Browser (old iLife code) or Sierra’s Screen Saver Library browser and do not transfer to the iPad via iTunes. However, if I export the original and drop it back in again then the image appears normally — this doesn’t appear to be problem with the JPEG. They also render normally in several different apps and EXIF viewers.

Rebuilding the Aperture database does not make a difference.

So this is curious, and potentially an important clue, but to go further I need to understand how Media Browser works.  In my case the bug doesn’t only impact image transfer to my iPhone; it impacts Media Browser as well. I’m now poking around at ~/Library/Application Support/iLifeAssetManagement and ~/Library/Application Support/iLifeMediaBrowser, especially ~/Library/Application Support/iLifeAssetManagement/state/albumshare/Daemon.sqlite.

Update 12/31/2017 II

Finding more “invisibles” but no clear pattern to what’s omitted. Some are quite small - from the days when 100K was a good image size. It’s not size alone though; of a pair of similar old images a 64kb one made it across and a 70k one didn’t.

Update 1/1/2018

Same sync, same everything — but now iPad acts as though it’s been syncing to a different computer. All images removed! It took several tries to get one album to sync, but after it started working I could do about  @1,200 at a time. Now have about 7,700 on iPad out of 9,100 on Mac. I won’t sync iPad to iTunes again until there’s word of a bug fix. Just stick with what’s on there now.

Update 6/9/2018

I thought this was fixed — but it’s broken again. On latest versions of iTunes and iOS 11.3.1, macOS Sierra.

Update 6/9/2018b

I was able to sync 7777 of 7771 (different apps count photos differently, not least due to Aperture’s brain-dead “Stacks” handling of iPhoto imports) photos after installing iOS 4. I don’t know if the bug is fixed or if it’s just that process of installing an iOS update cleared something out. In the latter case it might return.

Update 7/4/2020

It's been a year or two since I last tried syncing. At this time I'm still on Aperture and Mojave. The sync was worse than ever. I wiped my old iPad and restored from iCloud backup and sync actually seemed to work! It saw only 7992 of the 9413 images in my Aperture slide show, but it did import those.

I tried the trick of deleting the iPad Cache in Aperture [1] but on resync it was still 7992/9413. Consistent at least. 

I think there's a bug in Aperture with Sierra and later with generating the images in the iPod Cache. Maybe something iTunes or OS X was supposed to do.

 I suspect the iPad flash memory is also in bad shape. 

The lack of logging or error handling is appalling.