Removing 3500+ iCloud Contacts with no content (No Name) and Contacts.app use of 25GB of virtual memory

I suspect many people's iOS Contacts have been trashed by iCloud, but only a small percentage realize how bad things are. It may explain some of the mysterious data use and battery life problems some see.

I can't prove this of course; I'm not about to study a random sample of 100 devices. I can only go by my own disastrous experience and Emily's recent malfunction.

In my case the bug had to do with a change between MobileMe and iCloud in the handling of line endings resulting in large numbers of 'Groups' metastasizing between my devices. It was very hard to diagnose and fix that bug; since then things have seemed to go well with my use.

Emily's malfunctions seemed simple at first. I noticed she had both addresses on her device and in iCloud. I didn't expect that, but I figured it would be easy to clean up. [1]. Then I discovered about 3,500 entries in Contacts that showed up only as 'No Name'. This isn't a new problem and I don't know how long Emily had it. She never noticed, I suspect most non-geeks ignore that sort of thing. I wonder if it explains some battery life issues she's had, and even Spotlight and disk access issues on some of our older machines as these malformed entries propagated. (As noted below, during the removal process the OS X Contacts app used 25 GB of virtual memory and locked up my 27" iMac.)

Removing these malformed null-value entries was much harder than expected. Something about processing the deletes using Mountain Lion Contacts.app consumed vast amounts of virtual memory -- almost completely paralyzing my 2 yo iMac with the usual Spinning Beach Ball of Death (SPOD, SBOD, spinning wait cursor). (See: Getting control of a Mac when rogue software consumes virtual memory.)

In retrospect I think I wonder if I should have disabled my network connection, made the changes, and then reconnected and let iCloud process the cached change list. In any event, this is more or less what I did to delete them; in our home we have 1 iOS device (Emily iPhone) and two OS X devices (Mountain Lion Contacts.app and older Lion-only machine running Address Book.app) that synchronize to her iCloud account.

1. Work from our primary Mountain Lion machine, and create a local backup using the Contacts Archive function.
2. Turn off iCloud synchronization from the other iOS and OS X devices and delete all local contacts. On iOS the OS will offer the delete option, on OS X you have to delete them manually. (With iCloud sync off delete is easy.)
3. Open Safari to view changes there.
4. Because of the problems I ran into I suggest turning off Time Machine and, if you know how to do it, disable Spotlight disk indexing. At least uncheck Contacts from the Spotlight Search Results in the Pref Pane. You'll add them back in after you are done.
5. Close all other apps and log out all other users.
6. Do this incrementally - delete a few hundred, then quit and restart Contacts. When I skipped some of the above steps and tried to delete all 3,500 at once my machine locked me out. After some geek gymnastics I found Contacts.app was using 25GB of Virtual Memory -- enough to store about 1,000,000 contacts. Very little CPU was being used, but that much VM use is fatal.
7. When you're done run Contacts Cleaner if you own it.
8. Restart OS X, reenable Time Machine, Spotlight, etc.
9. On iOS (iPhone for example) reenable iCloud account. (In Emily's case this brought up the merge with local dialog -- the original small problem. See [1] for the easy fix for that, it does mean letting changes sync back to desktop as well).
10. Once that's settled down, reenable iCloud accounts on other devices. Let Spotlight catch up indexing them. You can disable then reenable Contacts search if you like

It will be interesting to see if her battery lifespan and Spotlight behaviors improve.

[1] If that's your only problem here's the way to clean it up:

1. Remove iCloud account from phone (you'll put it back).
2. Make sure only local device Contact remain.
3. Add iCloud back -- you'll get offer to merge them. Merge.
4. Clean up duplicates, easiest to do that on a desktop account using Contacts Cleaner (with my latest version I didn't disable network connectivity when doing this) or the dangerous fully automatic built-in Mac desktop duplicate remove.

See also:

• iCloud transition went as expected - disastrously 6/2012
• iCloud group replication resolved by AppleScript 7/2012
• An AppleScript to prevent the iCloud CRLF bug - normalize line endings 7/2012
• Synchronization is Hell 10/2008. Technologists underestimate this problem so commonly it has to be a universal wetware malfunction.
• Spanning Sync contacts cleaner - a quick review 1/11 (it's been upgraded since then, seems even better)
• Gordon's Tech: Getting control of a Mac when rogue software consumes virtual memory 1/12/2013

More on the No Name bug:

• Contacts with "No Name" need a mass...: Apple Support Communities
• Multiple 'No Name' contacts showing up in Address Book | MacFixIt - CNET Reviews

Getting control of a Mac when rogue software consumes virtual memory.

Adam Fields claimed something is deeply broken in OS X memory management. He says it's been partly fixed in Mountain Lion, but .. "the VM subsystem still falls over if you push it too hard".

That fits my experience when a bug with iCloud and Mountain Lion Contacts used Contacts.app to consume 25GB of virtual memory on my 8GB iMac. My system ground to a halt, even though CPU use was light. My system was locked up by disk bound memory paging.

Actually, it wasn't quite locked up. It was just excruciating slow to respond; a mouse click took minutes to process while I watched the OS X Spinning pinwheel.

I wanted to figure out what was happening, so I didn't simply power down. I was able to get control, which is how I identified the iCloud/Contacts bug and was able to gracefully quit Contacts, shut down normally, and then deal with the Contacts bug [1]

To get control I needed to get Terminal.app running. Terminal will work where most other things are frozen, and from Terminal I could control some of the usual bad actors. Note that I run with a non-admin account and I don't have root enabled.

At first I tried to ssh from a remote machine. To do that I needed to know the current IP address of my iMac; it didn't help to know the afp:// network path [2]. The only way I know to do that is to via this Mac OS X Hint. Note the hint has an error in the string, the comment versions are correct. From my remote machine I connected to iMac from the Finder and then used one of these Terminal commands (first is what I did and it works fine):

• netstat -na | grep 548
• netstat -f inet -n | awk '$5 ~ /\.548$/ {sub(/\.[^.]+$/, ""); print $5}'

Alas, that didn't help. I hadn't enabled Sharing:Remote Login for my iMac (I've since fixed that). So I had to return to my iMac and, carefully and slowly, get Launchpad [3] up and then Terminal. [5]

Since the account was non-admin I had to get Admin privileges so I could run sudo (root, superuser) commands:

•  su Admin

Spotlight is always a problem with system resources are low, so my first action was to get it out of the way. This is hard to do; I've read that the process restarts when killed but I tried anyway so I'd get a clean version restarting:

• sudo killall mds

Then I used renice to set spotlight priority to a lower value [4], first finding the process ID (it was 43702 at the time) then renicing it:

• ps -acx | grep mds
• sudo renice -n 19 -p 43702

That bought me more responsiveness. I was then able to slowly quit all running apps and gradually free up system resources until I could run Activity Monitor and figure out what was wrong with Contacts.app

[1] So there are at least two Mac bugs involved in this. One is a longstanding problem with OS X memory management that, I suspect, goes back to OS X 10.0. The other is one of several newer bugs with iCloud and Contacts. [2] That can be found by using Finder "Connect to Server" dialog from a remote machine. [3] LaunchBar and Finder weren't responding but Doc and Launchpad were slow but did respond. [4] It's annoying that renice isn't accessible from Activity Monitor -- in Windows there's a GUI for this control.  [5] I now put Terminal in the dock of my primary non-admin account for faster access next time and I enabled remote login in Sharing options: To connect use ssh username@ip_address, example "ssh molly@10.0.1.4".  If you have enabled root I think you can use "ssh root@10.0.1.4", but in my testing with ML where I have not enabled root that doesn't work.

See also:

• View connected file sharing clients' IP numbers - Mac OS X Hints
• Tutorial: Troubleshooting Spotlight | MacFixIt - CNET Reviews: 2006! Old, but sadly still relevant.
• Gordon's Tech: Spotlight indexer (mdworker) priority is too high in Lion/Mountain Lion 9/2012 - permanently setting Spotlight to lower priority. Excluding Aperture did help me.

iPhoto bug: black or all white photos when editing on MacBook Duo

I haven't used iPhoto on our white MacBook running Lion for years. I did so recently and I discovered I was afflicted by the Photo turns black while editing bug.

At first if I clicked the 'enhance' button the image vanished to be replaced by an all white icon. If I click Shadow or some other edit controls it went entirely black.

I upgraded to the very latest version of iPhoto (9.4.2) and repaired permissions/repaired database (cmd-opt-start). Now I have exactly this behavior:

iPhoto editing shadows makes displayed...: Apple Support Communities

... I am using a MB core 2 Duo at 2.16 MHz with 4 MB RAM with MacOS 10.7.4 and iPhoto '11 (ver 9.3.2) and have a problem with editing photos when I change the 'shadow or highlights'. The photo turns to black when I move the slider off the zero mark and the photo returns when I move the slider back to zero.

I had the same problem with the previous version iPhoto ('09 I think) and bought the new version '11 to try to fix the problem. The problem seemed to go away for a while but has returned. I have tried starting iPhoto using command/option and repairing the permissions, then rebuilding the thumbnails, then repair the database and then finally rebuilding the database. Each time I tried a repair, I tested the shadow/highlight edits and got the same black result. All other adjustments work fine. It is just the shadow/highlight adjustment that turns everything black ...

I moved the problematic library to my primary machine and, unsurprisingly, there were no problems at all.  I think there's a bug in iPhoto that renders it incompatible with my old MacBook's video processor. Apple is very unlikely to fix this.

Update: Aperture on the same old MacBook has no problems. If I adjust shadow in Aperture however, the image is all black in iPhoto.

This is a bug that affects rendering of at least some JPEGs following shadow/highlight edits on MacBook Duo with integrated Intel GPU.

Making Siri Useful

I'm getting more value from Siri lately. I think it's partly the enhanced noise canceling of the iPhone 5, though that may have other problems.

The trick, of course, is that I'm getting trained to use Siri. Even in my post-50 state of degeneracy, I'm a quicker study that Siri (though she does seem to be learning my voice).

I mostly use Siri to create reminders, notes, appointments or text messages while driving. This means I can't look at the screen or touch the screen; in fact the screen is usually locked. In this use case this is what helps:

• Assign relationship names (wife, etc) in Contacts. Siri cannot recognize my wife's name even with phonetic additions and multiple experiments, but she does recognize "Wife". You can do this in Contacts on OS X or even iOS, or you can tell Google directly (ex: "Steven F___ is my brother")
• Tell Siri which Contact is "You". This is very important.
• Train Siri at home (make test requests and correct errors)
• Think of what I want to say (plan it out).
• Turn off the fan (and radio of course) in my van before speaking a command.
• Press Home button and raise phone from flat horizontal towards my face then sharp pivot to my right ear (seems to help activate Siri on a locked phone). If wearing a headset, any headset, press and hold the 'call' button.
• Wait for the initial tone.
• Say what I want ("Remind me in 1 hour", "Take a note", "Make an Appointment at 9pm", "iMessage My Wife".)
• Wait for the tone.
• Speak in a measured steady pace with a brief "pause" between words (these pauses are perceptual, not usually real). 
• Use punctuation. It really helps to say 'period' when I'm done.
• After I speak, wait quietly while Siri decides I'm done.
• Use the critical words: Change and Cancel as needed. (Example: Change, or Change the ___ to...)

Incidentally, the iOS Google.app voice recognition is phenomenal. Siri is nowhere near as good, but of course the Google app isn't integrated with my iPhone. I assume it will eventually be integrated with Google's web services, but it will likely never work when an iOS screen is locked.

See also:

• Six tips for mastering Siri | Macworld - best of the tips pages - prior to this one of course.
• Gordon's Tech: Fixing Siri - When Really Sorry isn't good enough (if Siri is flailing, reset network settings. You will need to retrain.)
• Gordon's Tech: What I learned from reading the iPhone 5 manual.
• iMore published an excellent tutorial series on Siri. This link finds them as of 12/12.

Blogger's comment infrastructure - the end of anonymous

Google's Blogger has fought the good fight against spam comments for years, but it's failing now in several different ways. In the spam wars, failing to keep up is equivalent to surrender.

I assume that Blogger, like Google Reader itself, is slowly going the way of Google Reader Shares. Blogger is legacy GoogleMinus, not a good fit for the post-2011 Google. A small but dedicated team doesn't have the resources to keep it healthy.

I have looked at Disqus but $1,200 a year for SSO is too much for my budget. Other suggestions are welcome; I'd like a commenting system for Blogger that:

• Gives me control over what identity-authentication systems to enable, or, better yet, lets me define comment-rules on the basis of identity-authentication.
• Lets me blacklist authenticated users.
• Costs about $100 a year.

Anonymous comments would be nice to have, but they require spam filtering and CAPTCHA doesn't work any more. I'm afraid they are toast.

I suspect I can't get the commenting solution I want for Blogger, so I may have to switch to authentication via Google or turn them off altogether pending a future WordPress migration.

• Gordon's Notes: CAPTCHA has failed, and so anonymous comments may go too.
• Gordon's Tech: Spammers have found a hole in Blogger's comment system

Update: I've revised the title of this post, because when I actually, you know, looked at the current Blogger options they include OpenID as well as Google's authentication. Blogger comments support is still dated by current standards, but I'll switch to Google/OpenID, reenable notification for comments by email when > 2 weeks old, and leave out CAPTCHA. No more anonymous comments sadly.

Voice call audio clipping and dropouts on iPhone 5: hardware failure, bug, or network issue?

Modern voice quality is highly variable at the best of times, and I don't do all that many business calls on my mobile, so it took me a while to realize there was something very wrong with the voice quality on my AT&T iPhone 5 phone calls over the MSP LTE/3G network.

This post in a 27 page Apple Discussion thread matches my experience (emphases mine), except that many reports are from Germany as well as AT&T.  Some have problems only at particular locations. The thread describes many different issues, some of which seem to be Verizon specific or defective headphones and the like -- but this is the one I get.

https://discussions.apple.com/message/20440496#20440496 (sk1sk1)

... when on a phone call on iPhone 5, the sound from the distant party cuts in and out, drops entire words and phrases, the beginning and end of utterances, and background and ambient sound from the distant party. I can hear everything on iPhone 4, but much of the sound is dropped by iPhone 5. I’ve found a way to reliably reproduce the problem, perhaps useful for showing Apple the problem we’re having. Use Apple’s headset. Call from a very quiet location. Place a call to a call center’s greeting recording, ideally one that’s long and recorded at a varying volume, not too loud. I used Hertz On Demand here in the U.S. Set the volume to the lowest setting (important). Turn on mute. On my iPhone 5, the sound cuts in and out on 100% of the calls on every iPhone 5 I’ve tried, whereas everything is clear (though quiet) on my iPhone 4. The Genius bar had the same experience with their iPhone 5. I think the guy at this link has the best graphic demonstration of the problem: http://www.iphone5callquality.com/measurements/. You can clearly see what the phone is doing – it is discarding quiet signals. Unforunately, that includes entire words and phrases, especially if you've got the volume set low. I want to hear everything, don’t you?

One report says it is fixed in 6.1. Other fixes are dropping LTE or enabling hearing aid mode (that doesn’t work with headset though).

In my testing I switched back and forth between an iPhone 4 and iPhone 5 on a single conference call. The effect was pretty dramatic. I think it gets more pronounced after the first 6-15 seconds of a call. It feels like some kind of audio compression or background noise cancellation gone bad. Curiously, I think I can hear better when I'm on a call in my noisy van with lots of background noise at my end.

Several reports claim disabling LTE helps, though my understanding is that on AT&T's network voice always uses 3G, LTE is data only. Others say 'hearing aid mode' clears the problem up for non-headset use, but has no effect on headset. One developer reported that 6.1beta fixes the issue, another than a full factory default reset helped. Others claim improvements with a replacement phone, and many have no trouble at all. WiFi on/off doesn't make any difference. Several report problems are site specific. iPhone 5 to 5 connections are  the worst.

On the other hand, there really aren't that many complaints overall. Most users seem to be unaffected.

I suspect it's a combination of certain phone serial numbers/chipsets with certain networks. It might be fixable by a software update or a true fix might require a phone replacement. I've scheduled an Apple Store appointment and I'll share what I learn here. If the store doesn't know of an issue I'll hold off on a repair/replacement.

I've tried the 'LTE off' "fix", but since AT&T uses 3G for audio I doubt that will work. I have also tried the 'Hearing Aid' fix -- I can imagine that would help with non-headset calls. If it does I'll avoid the headset for a while.

See also: iPhone 5 Call Quality problem.

Update: from the Bouwmeester site: "HUGE NEWS from Apple Ireland: 'Your #iphone5 call quality issue is real, the next iOS version will solve it.'  but then:  "Apple: 'There MIGHT be a solution for the #iphone5 call quality issue in the next iOS but it is not guaranteed."...

Update 12/11/12: I had tolerable results both at my office and during the drive home using my Apple earbuds. Compared to my home the primary differences are different cell towers and more background noise. I don't think LTE settings make any difference for me. I do have 'hearing aid enabled' set but I doubt it makes any difference and, in any case, it's supposed to have no effect when a headset is used. I did try Settings:Reset:Resest Network Settings (you will need to reenter your wifi pw).

FWIW, I submitted a report to AT&T via email.

Update 12/12/12: I visited the local apple store, and, unsurprisingly, they hadn't run into this problem. The staffer I worked with felt it probably wasn't a hardware issue, he was wondering about a problem related to the noise cancellation function. He thought the very quiet setting of my home calls was a more likely factor than my local cell tower. He also suggested a network setting reset - which I'd done. He also wondered if my Speck candy shell was influencing the behavior of the microphone used for noise cancellation -- I'll try a call without it.

The other thing I'll try is some background mood music we bought years ago for our then toddlers -- be interesting to see if that reduces voice clipping on the incoming calls.

Lastly, I'll try lying the phone flat during some calls so the rear noise canceling microphone is either face down on a sound absorbent surface or face up.

ssspluto had a good summary in a Discussion Group Post dated 12/12/2012 - 8:19am

Update 12/14/12: I think I've traced this about as far as I can, and there is a workaround pending an Apple fix

I've ruled out a number of factors contributing to the problems I experience. For the record, I am iPhone 5, AT&T, St Paul area and I use a wired headset (Apple and 3rd party). My problem is audio clipping on my incoming audio.

I don't think my problem is a hardware issue, though in modern complex devices there is a thin boundary between software and hardware issues.

There is really only one circumstance I run into SERIOUS problems, though there are subtle issues elsewhere: a conference call on our corporate Global Conference system when I'm at home. I can connect to the same conference using an iPhone 4S and I get the normal (mediocre) sound quality of a VOIP based conferencing system.

Here are the things that don't seem to matter:

• LTE on or off
• WiFi on or off
• Case on or off
• Position of phone
• Covering or uncovering the rear background-noise-canceling headphone
• Resetting my network settings
• Model of wired earset/headset
• Restarting phone
• Background noise where I am

Here is what works

• unplug my headset when I've enabled 'hearing aid mode' and listen to the phone directly or use speaker phone.

I think there are multiple issues in this thread, but, as noted by a few contributors, this looks like an issue with the new-with-iPhone 5 noise cancellation software/hardware Apple is using and its interaction with AT&T's 3G compression in some cell towers.

The fix is 'hearing aid mode', which probably changes or disables noise cancellation when not using a headset. It is possible that a hardware problem on some iPhones is causing the noise cancellation product to misbehave, but since this is new software, and since for me the problem is so much worse with this one audio source and possibly even with selected cell towers, I'm somewhat optimistic it will be fixed with next iOS update.

I'm pretty sure Apple's engineers know what's wrong.

Update 1/9/13

The bug continues to annoy, but with the hearing aid setting I can do the problematic home-based Global Crossing conference calls on speaker phone or by old-fashioned handheld phone. In the meantime I read reports that the carriers know there's a problem, and one indefatigable Apple customer has a report from Amsterdam....

Yesterday I visited the Apple Store in Amsterdam to demo the call quality problem of my iPhone 5. Two employees from the Apple Store (an Apple Genius and a person from the Business department) made test calls with my iPhone 5 and had to admit that there clearly is a problem with the call quality. When you are on a call the incoming voice drops in and out, the audio sounds digitized and choppy.

We also tested my iPhone 4 with the same version of iOS (6.0.2). The call quality of the iPhone 4 was perfect. The Apple Genius decided to place more test calls with two of their own iPhone 5’s and experienced the same call quality issues. All test calls were made using the standard Apple EarPods.

To rule out that the problem could be caused by settings or installed apps, the person from the Genius Bar activated a brand new iPhone 5 for me. When we made test calls with this new-out-of-the-box iPhone 5 the result was the same: choppy sound on every call. The exact same effect that can be heard in my YouTube video.

The person from the Genius Bar concluded that this is a real problem that they cannot solve. He told me that he would do everything for me to find a solution for the problem and that I would hear from Apple soon. I hope that the Apple Store Amsterdam is more helpful than Apple Ireland.

 I was pretty sure Apple understood the bug and was working on a fix, but I am beginning to wonder ... 

Update 1/31/13

I think 6.1 fixed the problem I had, though I have yet to test from my home cell tower. I did a Global Crossing VOIP conference call from my car and it went well, I've seen other reports of success. I hope to test from my quiet home setting and home cell tower next week.

Spotlight indexer (mdworker) priority is too high in Lion/Mountain Lion.

Spotlight was badly behaved in Leopard, but by late Snow Leopard Apple seemed to have most of the bugs worked out.

Not so in Lion/Mountain Lion. It's hogging resources and seems ever more prone to corruption. I think, among other things, Apple set the process priority too high -- perhaps because they're now coding for SSD storage. Spotlight can paralyze my 2 yo 27" iMac; sometimes a forced shutdown is my only option (which seems to corrupt the index -- so I pay back later).

Windows Process Explorer allows adjustment of process priority, but Mac Activity Monitor doesn't. There are ways to adjust mdworker priority by editing /com.apple.metadata.mds.plist attributes of LowPriorityIO and Nice, but these seem like risky system edits. On my currently too-slow MacBook running Lion I, will, however, experiment with the recommendations on this site:

Shortcuts I use to set up a new OSX user account
Set Spotlight indexer (mds, mdworker) to a lower processor priority:
sudo defaults write /System/Library/LaunchDaemons/com.apple.metadata.mds LowPriorityIO -bool true
sudo defaults write /System/Library/LaunchDaemons/com.apple.metadata.mds Nice -int 20
(there are several speedups on this page I'll try on the MacBook)

I think there's more than one bug with Spotlight in Lion/ML. For example, I think performance deteriorates over time. I need to restart my ML system much more frequently that Snow Leopard.

Apple needs to give lots of attention to Spotlight in its ML maintenance updates.

See also:

• Shortcuts I use to set up a new OSX user account (Gnarlodious)
• MDS and mdworker processes gone wild: Apple Support Communities
• Is mdworker Jeopardizing Your Resources? Run It at a Lower Priority | The Grey Blog
• Tutorial: Troubleshooting Spotlight | MacFixIt - CNET Reviews: A good collection of tips and fixes. I'm going to exclude my Aperture/Image folders from Spotlight indexing; I don't think Aperture's internal search uses Spotlight and I never need to find photos using Spotlight.