Saturday, February 16, 2013

Google Contact Sync on Mountain Lion and iOS - CardDAV (Project Contacts 2013)

I'm rather late to this party [6], but around the time Google ended ActiveSync support for unpaid accounts [1] they added vCard 3 (1998) based CardDAV support [3]. Up until then one could sync iOS and OS X mail and Calendars via IMAP and CalDAV, but Contact sync was only via Active Sync. [2]

TUAW has some details and Google's iOS CardDAV instructions are here.

As we all know, synchronization is hell. Even if Apple and Google actually used the same vCard spec (probably vCard 3, 1998), and even if they agreed on newline characters, they still have to deal with distinctions between their internal data models and vCard. Some of those data model gaps are non-computable. [5]. Even thought Google and Apple have similar 'Group' models that relationship metadata is ignored [8].

That said, at least CardDAV is relatively Apple friendly. Most importantly, this standard means, for the very first time I know of [7], there's a somewhat supported way to synchronize Contacts between Google and OS X Contacts and iOS contacts.

I've begun using it on Mountain Lion. Since I use Google two factor authentication I had to use one of their totally-defeats-the-purpose-giant-security-hole-not-single-use-god-i-hate-passwords-humans-are-losing-the-battle alternative passwords. There was some kind of transient authentication glitch but on a second try it worked. After a few minutes I had all 634 of my Google "My Contacts" on my Mac. Because of the long history of my Google/Apple Contact sync efforts (See Also, below) these largely duplicate my Address Book/iCloud contacts -- but Mountain Lion Contacts.app seems to merge the duplicates when I search [9].

This looks promising in a terrifying sort of way. You can drag and drop contacts between the iCloud and Google Contact lists; they'll be copied.

So what about Snow Leopard? Many fine machines run Snowie (and should not upgrade to Lion even if they theoretically can). They're (more or less) cut off from iCloud. Does CardDAV sync work with Google now? I wonder about adopting Hsiaio's technique, but so far the answer is no (same with Yahoo CardDAV). I didn't find much research on this question, but I suspect it's a combination of 10.6 bugs and SSL requirements. [9]. There is some limited Google Contacts sync on Snow Leopard that some still use.

- fn - lots

[1] Google grandfathered my half-dozen Google Apps accounts and recently extended the termination date for non-Apps users.

[2] Google says Google Apps accounts should use "Google Sync", which is their implementation of Microsoft's Exchange Server derived ActiveSync (with different bugs one assumes).

[3] vCard over HTTP/WebDAV. vCard has been Apple's Contact/Address Book export option for as long as I can recall, so it may be a good match to their Contact data model. CardDAV is also used by Apple's ancient Address Book Server. vCard 4 2011 has old-style and XML representations, but I suspect Apple, like Google, is more or less on 1998's vCard 3.

Google's vCard implementation has limits; worse the only partly implement the 1998 vCard 3 spec: "why did Google move the ADR extended address "Oak and Pine" into the street address component? And worse still, why did it separate the two using an '\r\n' (CRLF) sequence? The VCard 3.0 specification clearly states that CRLF must be escaped by the single escape sequence '\n', not by '\r\n'".[4]

[4] Reading the last line of [3], by the way, a penny dropped. Apple changed the way they handled CRLF and vCard between MobileMe and iCloud, with near-disastrous consequences for my OS X Contacts. By Darwin, I hate the ancient DOS/unix/mac line newline debacle.

[5] Meaning data loss is inevitable, transferring a Contact between systems won't return the original. Incidentally, under my TrueName, I've spent 15 years dealing with interoperability issues in healthcare - from HL7 to TermInfo to SNOMED/RxNORM and back again via cCCD and the like. If you understand why CardDAV is hard, then you have a glimmer of a smidgen of a hint of an idea why we don't have true interoperability between EpicCare and Cerner -- and why we won't for decades.

I think the people who trudge away on vCardX are heroic figures who should be carried upon our shoulders. Forget Ive, these people do the work I rely on.

[6] The marketing/interest problem is unsolved -- largely because of the spam/advertising counter-current. I'm very interested in productivity software and I follow a number of related feeds; but I still miss news like this.

[7] Check out "See also" - lots of history here.

[8] Looking at Apple's braindead implementation of iOS Contacts and Groups I'd pin this failure on Cupertino and Apple's corporate silos.

[9] A friend of mine has a 10.6 MacBook and needs to use Google's world (as do we all). I think the easiest way to manager her Contract migration will be to get them to iOS/Google and then use Gmail web on Mac. I may put them on an account on my ML machine to help with cleanup.

[9] So which gets edited when one clicks "Edit" in Mac Contacts.app? Don't ask those questions. OS X doesn't say.

See also - via Gordon's Tech

Yeah, I've been fighting this war a long time. February 2009, almost exactly 4 years ago, was a big series of battles, but the 2008 PalmOS to iOS migration was probably the worst (later there were tools). I've read recently that Apple is an intensely siloed company -- that explains why iOS and OS X contract integration is so very bad.

Thursday, February 14, 2013

Facebook albums: How to choose which photo will appear on the timeline view of the album (primary photo, key photo)

I don't put that many albums on Facebook, but whenever I do I spend at least five minutes trying to remember how to change the album picture that appears on the timeline when the album is published.

The problem is there are many edit menus associated with Facebook photos. There are menus that appear on the photo, menus for photo descriptions, menus for editing albums and so on. The one you want is the edit menu associated with the timeline item, not with the album or the photo:

19+ Facebook Timeline Features and Resources You Should Know!

... To switch it with another photo, click at the pencil icon and select ‘Change Primary Photo’. You can then use ‘Reposition Photo’ if your selected picture doesn’t fit in properly."...

Now I might remember.

Thursday, February 07, 2013

Brother HL-2270DW out of toner workarounds and admin screens

I just bought an HL-2270DW to replace a 2140 at drum end-of-life. This review is excellent (here is my short one), Wirecutter recommended it as best cheap laser printer and Brother drivers seem better behaved than most. User manuals are here.

I've have preferred the lower print costs of a higher grade printer, but the Brother model I considered draws too much current for many overloaded home circuit breakers.

Like most Brother printers (all printers?) it will stop printing when toner is "low" -- typically a few hundred pages short of an empty cartridge. With the 2140 I could cover an optical sensor and order a new ($$) cartridge. That doesn't work on the 2270; this page provided 3 options which I've revised here:

  1. The unscrew and rotate method
  2. Press the GO Button 7 times (quickly). " All of the LEDs should flash twice, and the Ready LED should turn on." Repeat to revert.
  3. Use the web GUI to change the Replace Toner printer setting to "continue" instead of "stop"

I like the last one. I used Bonjour Browser to find my printer's IP address, but I don't recommend this [2]. Instead read the Network guide/manual. In OS X  I used Open Print Queue I clicked a button called "Show Printer Webpage ...". That did the trick; OS X resolves the Rendezvous URL to the correct network address.

Click printer settings (un/pw is user/access) then change the Replace Toner settings. There are other settings  and reports of interest as well [1].

I left the settings on Stop -- when the printer stops I know to order a new cartridge; I can change the settings then.

PS. The printer is supposed to come with an optional sticker that explains various flashing lights. I found mine floating about the floor. Watch for this when unpacking.

[1]

  • Drum Unit is rated for 12,000 pages, memory size is 32MB.
  • The Administrator pw is admin/access. You can use this to change both the Admin (admin/access) and User (user/access) credentials if you wish.
  • You can give the printer a static IPv6 address (good luck)
  • It has NetBIOS support (brings back nightmares)
  • BRAdmin uses the printers SNMP services
  • It has an FTP and TFTP server with mDNS and LLMNR (who needs all this in a home printer?!)
  • It has an SMTP service and a POP3 Server?!! - presumably for notifications (but why POP3?). You just know that this would feature in some evil network hack.
From the Network Guide supported protocol list:

(IPv4) (IPv6) 1 10/100BASE-TX IEEE802.11b/g (Infrastructure Mode / Ad-hoc Mode) ARP, RARP, BOOTP, DHCP, APIPA (Auto IP), WINS / NetBIOS name resolution, DNS Resolver, mDNS, LLMNR responder, LPR / LPD, Custom Raw Port / Port 9100, IPP, FTP Server, SNMPv1 / v2c, HTTP Server, TFTP Client and Server, SMTP Client, ICMP, LLTD responder, Web Services (Print) NDP, RA, DNS Resolver, mDNS, LLMNR responder, LPR / LPD, Custom Raw Port / Port 9100, IPP, FTP server, SNMPv1 / v2c, HTTP Server, TFTP Client and Server, SMTP Client, ICMPv6, LLTD responder, Web Services (Print) APOP, POP before SMTP, SMTP-AUTH WEP 64/128 bit, WPA-PSK (TKIP/AES), WPA2-PSK (AES), APOP, POP before SMTP, SMTP-AUTH Wi-Fi Certification Mark License, Wi-Fi Protected Setup (WPS) Identifier Mark License, AOSS Logo

I wonder if it's running Linux.

[2] After I ran Bonjour Browser ML persisted in showing the Rendezvous identifier for the printer as a "SHARED" item in the Finder left bar. I had to run BB again, and twiddle with reload services, to make it go away. I think BB might need an update -- in one config screen it refers to "Jaguar".

Update 2/21/2013: I had to do this when the "starter" toner ran out and the real cartridge didn't arrive. First times I tried setting the web config to 'continuous' instead of 'stop' it failed with a 'printer busy' message. I paused the queue and restarted the printer and I was able to change the setting.

Update 4/4/2013: I bought a TN-450 2,600 page extended cartridge, and after 200 pages I have a toner light. I wonder if it was a counterfeit -- though I did buy it through Amazon direct. Alternatively it or the printer may be defective. Alas, it's 4 days past the return period so I'm stuck.  Looking at the 1 star reviews I'm not the only user to run into this. 

Update 5/21/2013: These aren't counterfeit cartridges, we really are only getting 600 or so pages per $55 cartridge. See: Gordon's Tech: Is Brother running a scam with its HL-2270DW printer and TN-450 toner cartridge?

Saturday, February 02, 2013

Sharing a credential store for a family (or small business) - and the end of the personal database

I have managed family credentials and web site data in a FileMaker database since 1996 [1]. That database now holds over 1600 records, of which at least 1200 are historical (dead accounts, obsolete urls). The historical items don't get in the way as much as one might think, and I kind of like this mini-history of the net.

This repository has survived many technology transitions. Over the past 10 years it has more-or-less synchronized with credential stores running on PalmOS and iOS. At some times I've used FileMaker sharing (web publishing and the IP based fmnet: protocol) to allow my wife to access and edit both of our credentials too. (I am too boring to have any secrets, and it's critically important that she have credential access if I'm unavailable -- as in dead.)

Alas, all things must come to an end. FileMaker and its half-baked replacement "Bento" are both dying. Filemaker Go, which I'd like to use on my iPhone, doesn't support true synchronization. Bento is supposed to have some synchronization capabilities with iOS clients, and there's some form of encryption on the desktop version, but I don't know of anyone using this product. Bento feels stillborn.

So while I could probably cobble together a solution using some combination of FileMaker licenses ($180+ each) and or FM 12 web sharing (Mountain Lion broke FM 11 web sharing [2]) -- but then I'm investing in a dead technology niche.

There are other issues with the way I use this credential store. I've been maintaining it in FileMaker and exporting to 1Password for iPhone access, but that means my iPhone repository is read-only. The update loop is slow; sometimes I don't have credentials on hand.

It's time to move.

After consultation with appnetizens there appear to be 3 solutions:

  1. Use 1Password with Dropbox. (I'm still on iPassword 2, but it works similarly. I don't mind upgrading anyway.)
  2. Use some other Cloud solution, such as shared Google Doc or Spreadsheet, Simplenote, etc.
  3. Use 1Password on her iPhone only (see below). [1Password 2 only - see update.]

The first two require the Cloud. I don't like that - for several reasons [3].

That leaves option 3, and that only works because 1Password runs constantly on my always-active account and 1Password on the desktop can support multiple iOS clients. It's not perfect as it gives her credential access only via her iPhone; but since we do use Google Sync that's less of a problem than it might be [4].

I think we'll try option 3, which implies I'm going to treat 1Passwords as the "source of truth" and slowly archive my last FileMaker database -- and then FileMaker itself.

RIP old friend.

Update:

Well, that's interesting. Technology transitions are rarely smooth. It's not clear that there is a non-Dropbox solution any more. I think I'll keep Filemaker on life-support a bit longer.
 
Incidentally, There's something disturbing about how 1Password syncs. Watching the behavior it's almost as though it replays every transaction that's every occurred on 1Password desktop; implying it still keeps fully deleted credentials. When I updated the cache 10,000 records were processed for 1,500 credentials. Looking at my backup history the agilebits keychain file is growing at a rate consistent with retention of deleted credentials. I think the only way really clean it out is to delete the keychain and start anew.

--

[1] FileMaker is a survivor from the pre-net era of software, when geeks used personal databases for address books, recipes, PDF based document management, record and CD collections,  archives of thousands of stapled paper medical journal article archive indexing with pre-web "Grateful Med" MEDLINE metadata (Grateful Med), club memberships, and much more. With a relatively large user base FileMaker was then quite affordable.

I will now interrupt this post to admit that review of those web 1.0 pages, and particularly the data store transformations of the past 8 years, has given me a dose of old-guy future shock. When I started researching this topic I didn't realize I was also writing an obituary for the personal database!

It has been one hell of a ride from my 1990s FileMaker article file to Google and the Strata conference I'll be attending in Santa Clara this month.

[2] I tried reenabling Apache on ML using a freeware pref pane. Apache ran, but FM web sharing still didn't work. Looking at the FM 12 writeup I think it barely works even there.

[3] Example: An encrypted database store is forgotten in the cloud. Six years from now, when Dropbox's assets have been acquired by the Russian mafia, 2012 encryption is trivially broken. I don't want this store public even in 2019. Not to mention the LastPass breach.

[4] So why do I allow that form of Cloud use? Good question.

Saturday, January 12, 2013

Removing 3500+ iCloud Contacts with no content (No Name) and Contacts.app use of 25GB of virtual memory

I suspect many people's iOS Contacts have been trashed by iCloud, but only a small percentage realize how bad things are. It may explain some of the mysterious data use and battery life problems some see.

I can't prove this of course; I'm not about to study a random sample of 100 devices. I can only go by my own disastrous experience and Emily's recent malfunction.

In my case the bug had to do with a change between MobileMe and iCloud in the handling of line endings resulting in large numbers of 'Groups' metastasizing between my devices. It was very hard to diagnose and fix that bug; since then things have seemed to go well with my use.

Emily's malfunctions seemed simple at first. I noticed she had both addresses on her device and in iCloud. I didn't expect that, but I figured it would be easy to clean up. [1]. Then I discovered about 3,500 entries in Contacts that showed up only as 'No Name'. This isn't a new problem and I don't know how long Emily had it. She never noticed, I suspect most non-geeks ignore that sort of thing. I wonder if it explains some battery life issues she's had, and even Spotlight and disk access issues on some of our older machines as these malformed entries propagated. (As noted below, during the removal process the OS X Contacts app used 25 GB of virtual memory and locked up my 27" iMac.)

Removing these malformed null-value entries was much harder than expected. Something about processing the deletes using Mountain Lion Contacts.app consumed vast amounts of virtual memory -- almost completely paralyzing my 2 yo iMac with the usual Spinning Beach Ball of Death (SPOD, SBOD, spinning wait cursor). (See: Getting control of a Mac when rogue software consumes virtual memory.)

In retrospect I think I wonder if I should have disabled my network connection, made the changes, and then reconnected and let iCloud process the cached change list. In any event, this is more or less what I did to delete them; in our home we have 1 iOS device (Emily iPhone) and two OS X devices (Mountain Lion Contacts.app and older Lion-only machine running Address Book.app) that synchronize to her iCloud account.

  1. Work from our primary Mountain Lion machine, and create a local backup using the Contacts Archive function.
  2. Turn off iCloud synchronization from the other iOS and OS X devices and delete all local contacts. On iOS the OS will offer the delete option, on OS X you have to delete them manually. (With iCloud sync off delete is easy.)
  3. Open Safari to view changes there.
  4. Because of the problems I ran into I suggest turning off Time Machine and, if you know how to do it, disable Spotlight disk indexing. At least uncheck Contacts from the Spotlight Search Results in the Pref Pane. You'll add them back in after you are done.
  5. Close all other apps and log out all other users.
  6. Do this incrementally - delete a few hundred, then quit and restart Contacts. When I skipped some of the above steps and tried to delete all 3,500 at once my machine locked me out. After some geek gymnastics I found Contacts.app was using 25GB of Virtual Memory -- enough to store about 1,000,000 contacts. Very little CPU was being used, but that much VM use is fatal.
  7. When you're done run Contacts Cleaner if you own it.
  8. Restart OS X, reenable Time Machine, Spotlight, etc.
  9. On iOS (iPhone for example) reenable iCloud account. (In Emily's case this brought up the merge with local dialog -- the original small problem. See [1] for the easy fix for that, it does mean letting changes sync back to desktop as well).
  10. Once that's settled down, reenable iCloud accounts on other devices. Let Spotlight catch up indexing them. You can disable then reenable Contacts search if you like
It will be interesting to see if her battery lifespan and Spotlight behaviors improve.

[1] If that's your only problem here's the way to clean it up:

  1. Remove iCloud account from phone (you'll put it back).
  2. Make sure only local device Contact remain.
  3. Add iCloud back -- you'll get offer to merge them. Merge.
  4. Clean up duplicates, easiest to do that on a desktop account using Contacts Cleaner (with my latest version I didn't disable network connectivity when doing this) or the dangerous fully automatic built-in Mac desktop duplicate remove.

See also:

More on the No Name bug:

Getting control of a Mac when rogue software consumes virtual memory.

Adam Fields claimed something is deeply broken in OS X memory management. He says it's been partly fixed in Mountain Lion, but .. "the VM subsystem still falls over if you push it too hard".

That fits my experience when a bug with iCloud and Mountain Lion Contacts used Contacts.app to consume 25GB of virtual memory on my 8GB iMac. My system ground to a halt, even though CPU use was light. My system was locked up by disk bound memory paging.

Actually, it wasn't quite locked up. It was just excruciating slow to respond; a mouse click took minutes to process while I watched the OS X Spinning pinwheel.

I wanted to figure out what was happening, so I didn't simply power down. I was able to get control, which is how I identified the iCloud/Contacts bug and was able to gracefully quit Contacts, shut down normally, and then deal with the Contacts bug [1]

To get control I needed to get Terminal.app running. Terminal will work where most other things are frozen, and from Terminal I could control some of the usual bad actors. Note that I run with a non-admin account and I don't have root enabled.

At first I tried to ssh from a remote machine. To do that I needed to know the current IP address of my iMac; it didn't help to know the afp:// network path [2]. The only way I know to do that is to via this Mac OS X Hint. Note the hint has an error in the string, the comment versions are correct. From my remote machine I connected to iMac from the Finder and then used one of these Terminal commands (first is what I did and it works fine):

  • netstat -na | grep 548
  • netstat -f inet -n | awk '$5 ~ /\.548$/ {sub(/\.[^.]+$/, ""); print $5}'

Alas, that didn't help. I hadn't enabled Sharing:Remote Login for my iMac (I've since fixed that). So I had to return to my iMac and, carefully and slowly, get Launchpad [3] up and then Terminal. [5]

Since the account was non-admin I had to get Admin privileges so I could run sudo (root, superuser) commands:

  •  su Admin
Spotlight is always a problem with system resources are low, so my first action was to get it out of the way. This is hard to do; I've read that the process restarts when killed but I tried anyway so I'd get a clean version restarting:
  • sudo killall mds
Then I used renice to set spotlight priority to a lower value [4], first finding the process ID (it was 43702 at the time) then renicing it:
  • ps -acx | grep mds
  • sudo renice -n 19 -p 43702
That bought me more responsiveness. I was then able to slowly quit all running apps and gradually free up system resources until I could run Activity Monitor and figure out what was wrong with Contacts.app

[1] So there are at least two Mac bugs involved in this. One is a longstanding problem with OS X memory management that, I suspect, goes back to OS X 10.0. The other is one of several newer bugs with iCloud and Contacts. [2] That can be found by using Finder "Connect to Server" dialog from a remote machine. [3] LaunchBar and Finder weren't responding but Doc and Launchpad were slow but did respond. [4] It's annoying that renice isn't accessible from Activity Monitor -- in Windows there's a GUI for this control.  [5] I now put Terminal in the dock of my primary non-admin account for faster access next time and I enabled remote login in Sharing options: To connect use ssh username@ip_address, example "ssh molly@10.0.1.4".  If you have enabled root I think you can use "ssh root@10.0.1.4", but in my testing with ML where I have not enabled root that doesn't work.

See also:

Monday, December 31, 2012

iPhoto bug: black or all white photos when editing on MacBook Duo

I haven't used iPhoto on our white MacBook running Lion for years. I did so recently and I discovered I was afflicted by the Photo turns black while editing bug.

At first if I clicked the 'enhance' button the image vanished to be replaced by an all white icon. If I click Shadow or some other edit controls it went entirely black.

I upgraded to the very latest version of iPhoto (9.4.2) and repaired permissions/repaired database (cmd-opt-start). Now I have exactly this behavior:

iPhoto editing shadows makes displayed...: Apple Support Communities

... I am using a MB core 2 Duo at 2.16 MHz with 4 MB RAM with MacOS 10.7.4 and iPhoto '11 (ver 9.3.2) and have a problem with editing photos when I change the 'shadow or highlights'. The photo turns to black when I move the slider off the zero mark and the photo returns when I move the slider back to zero.

I had the same problem with the previous version iPhoto ('09 I think) and bought the new version '11 to try to fix the problem. The problem seemed to go away for a while but has returned. I have tried starting iPhoto using command/option and repairing the permissions, then rebuilding the thumbnails, then repair the database and then finally rebuilding the database. Each time I tried a repair, I tested the shadow/highlight edits and got the same black result. All other adjustments work fine. It is just the shadow/highlight adjustment that turns everything black ...

I moved the problematic library to my primary machine and, unsurprisingly, there were no problems at all.  I think there's a bug in iPhoto that renders it incompatible with my old MacBook's video processor. Apple is very unlikely to fix this.

Update: Aperture on the same old MacBook has no problems. If I adjust shadow in Aperture however, the image is all black in iPhoto.

This is a bug that affects rendering of at least some JPEGs following shadow/highlight edits on MacBook Duo with integrated Intel GPU.