Debugging network account lockouts: issues with Microsoft Active directory authentication

I recently experienced a personally new and novel set of computer network related issues. I'll have more to say on Gordon's Notes about my take on the implications and lessons of this experience, but on this blog I'll stick to measures end-users might take.

If you're reading this I'll presume you are a user on a corporate network and you are now unable to get at network services. If you request a shared drive or other network resource you are asked to provide your credentials (username and password). You may be unable to login to your workstation while you are connected to the network, though if you pull the network cable or disable wireless access you can login locally.

In this case it is likely that your network authentication is failing. Your credentials are not valid, you've been locked out.

There are legitimate reasons to be locked out of course, but most of the time this is an error. A Microsoft Active Directory group policy setting in your organization specifies an allowed number of failed authentication attempts in a certain time interval and "you" have passed that limit.

By "you" of course I mean whatever is trying to login with your username -- but not your current password. The problem, you see, is that many things may be doing that. Some may be on your machine, some may be on other machines you've used or use, and some may be in places you can't imagine. One of these things may be you, of course, entering your password incorrectly more than, say, five times, in a certain interval.

Ahh, but you say you only made one mistake? Well, maybe something else was trying four times in the key interval. Your one mistake was the last straw.

This is a big problem. You'll find many hits on the topic if you start looking. It's a Cloud problem (new tag today!). It's what happens when authentication starts to diffuse, and when you don't have a robust system for distributing authentication privileges. It's what happens when credentials are cached or distributed, and there aren't robust tools in place to monitor and track -- or when organizational structures block recognition.

Microsoft has tools for diagnosing active directory account lockout issues, but they are not accessible to end-users like you ...

• http://technet.microsoft.com/en-us/library/cc738772(WS.10).aspx
• http://www.microsoft.com/downloads/details.aspx?FamilyId=7AF2E69C-91F3-4E63-8629-B999ADDE0B9E&displaylang=en
• best overview (private blog).

As an end-user victim these are some things you may investigate once your help desk has unlocked you. Good luck ...

• Change your network password, that may fix some caching issues.
• OS level drive encryption software, bolted onto a decrepit XP infrastructure, can be a problem. These typically synchronize credentials with Active Directory -- and we all know synchronization is Hell. Look into any associated logs that might show how synchronization is preceding. See if you can change your password using the UI controlled by the encryption package and watch that propagate to the server.
• Group policy updates may be failing, resulting in passwords failing to comply with standards and leading to rejection. Research use of the gpudate.exe /force command to update local copies of corporate policies.
• Eliminate all drive letter mappings on all machines. I know longer do drive letter mapping on corporate networks. These can have cached credentials that fail to update.
• If you use Remote Desktop, log in to every RD machine you use and make sure you are fully logged out again. You may need to apply all fixes and patches there as well.
• Try shutting down your main workstation when you are not at working -- or disconnect it from the network. If you're locked out then you may suspect the problems are from other sources.
• Do not use Windows Search to index mounted drives.
• remove all IE stored information - cookies, passwords, etc. Used Delete All from the IE General/Delete settings tab. Note this is the ONLY option if you want to be sure to remove any stored credentials from IE Add-Ons.
• Consider uninstalling any applications that authenticate with Active Directory, such as Office Communicator.
• Evaluate all applications that might interact with Microsoft Sharepoint, because these require Active Directory authentication. This may include:
  • Windows Live Writer: Posting from WLW to a SP blog implies an authentication event
  • Lotus Connections: If you use Lotus Connections web-based feed reader against a Sharepoint feed there's an implied authentication event. (In my testing these subscriptions appear to fail, but does LC attempt to authenticate with its internal credentials? What about if the user IDs match between LC and SP but the passwords differ?)
  • Outlook 2007: Outlook 2007 is able to subscribe to SP Calendars and other Sharepoint Lists. All of these imply authentication. Prior to SP1 Outlook 2007's subscription/feed support was extremely buggy.
  • Any feed reader that works against Sharepoint authenticated feeds

Personally, this is the nastiest problem I've come up against in 25 years "behind the mast". I'll have more to say in my opinion-oriented blog about how this has changed my approach to personal and cloud computing and to the new approaches I'm taking to risk mitigation going forwards.

Update 6/2/09: Focus is now on a combination of a Sharepoint List synchronization that could not be removed from Outlook 2007, a possible configuration error on Sharepoint, Outlook configured to send only on manual send/receive, send/receive configured (by default) to include the unremovable SharePoint list, and Outlook offline caching of credentials.

Update 6/4/09: Microsoft Wireless PEAP always caches credentials. Could be a contributor in some situations.

Update 6/12/09:The saga continues. To remove the long stuck Sharepoint list in Outlook 2007 I had to remove reference to it in the Outlook send/receive group. In fact, I removed most things from that group. The NTLM account lock problem went away -- but I then had to manually authenticate the first time I used Outlook to access Exchange server. In other words Outlook was no longer able to deliver my credentials automatically. (The advanced security settings for Outlook did not have "manual credential" checked.)

After a week of this my laptop was refreshed. Using a brand new image I was again locked out. (I did have to install Retrospect Pro to restore data, but I think the first lockout was before I restored anything.)

This went on for a few days, then I did into an obscure option in Outlook 2003 (and 2007) properties and set Outlook to always require manual credential entry. The account lockouts stopped.

I'm going to study this for a few days, and see if I can get locked out by turning off manual credential entry. If I can confirm this does the trick, I'll try to bring very specific fix suggestions to our puzzled help desk and security services. I need to better understand the NTLM/Exchange/Outlook authentication procedure.

Update 6/12/09b: I've asked this question on serverfault.

Update 6/27/09: I post an answer to my own serverfault question:

... I've not been locked out for over a week even after turning re-enabling Outlook pass-through authentication, so even though there was no definitive cure I can report where I left things.

As a reminder, the last time I was locked out I'd just received a brand new laptop with a fresh corporate image.

The very last things I did were:

1. I found the brand new corporate image included two drive mappings. Sigh. (Sound of head hitting wall.) I'd removed them from my old laptop long ago, but they were back. I removed them again. It wasn't the only problem in the corporate image.

2. I experimented with switching Outlook 2003 authentication between "automatic" (default), Kerberos only (modern) and NTLM only (legacy). Switching to Kerberos only seemed to resolve problems, but I think that was a red herring. Switching back to the default didn't restore the lockout problem.

3. I use Retrospect Professional (EMC Retrospect) for Windows to backup my workstation to an external drive. (Corporate backup isn't bad, but restore takes about a week.) That software has an autolaunch feature. I'd set it to auto-launch using the logged-in credentials rather than the treacherous feature of providing credentials. I wonder though about an intersection between the mapped drives and the auto-launch. I turned off Retrospect Pro auto-launch for now.

I very much appreciate the link Neobyte provided to Microsoft's June 2008 troubleshooting page - Troubleshooting Account Lockout

http://technet.microsoft.com/en-us/library/cc773155%28WS.10).aspx

I'm left with some psychic scars. Given the astounding variety of problems associated with Microsoft's authentication services and their pile of legacy hacks, and the intersection with distributed authentication and post-hoc security features like authentication lockouts, I'm now deeply conservative about my use of any new or novel corporate network or "cloud" initiatives. They need to be built on a far more robust infrastructure than what Microsoft provides, and they require both IT funding and IT reorganization to implement.

7/21/09: I found yet another potential contributor -- one I'd long forgotten about. I'd once set up my iPhone to connect to the corporate WLAN. To do this I had to enter my Active Directory login credentials. The iPhone connects automatically when the WLAN is in range. So what happens when my network credentials change and the iPhone tries to connect? I'm not sure. Maybe it fails once and doesn't try again -- generating only one lockout hit. Maybe it tries repeatedly. Who knows. The point is, we're screwed.

We need better ways to manage user authentication and privilege control, and we need them desperately.

As for the iPhone, there's no way to have it remember network credentials yet not automatically connect when WiFi is enabled. So I deleted my corporate WLAN credentials from my iPhone.

Federated authentication - Gmail and Facebook

Facebook now supports "linked" Gmail accounts, by which they mean OpenID 2.0 authentication.

I linked my FB and Gmail accounts, so now as long as I'm logged in to Gmail I can use FB without additional authentication. The link process also grants FB access to my Gmail address book -- for better or worse. Facebook will also accept an OpenID URL.

I swear I saw evidence of a Gmail specific OpenID URL recently, but I can't recreate it. Google has not yet officially released an official OpenID URL for Gmail accounts they are available, oddly, via Blogger.

This can't come fast enough. This kind of authentication means I can use a robust password with Google and not have to maintain a large number of complex passwords. It also means I can integrate account information without having to (unthinkable) share my Gmail/Google account password.

Now if Google would only accept more robust forms of authentication than mere passwords ...

Update 8/31/09: This only worked for a few weeks, then it stopped working. I also experienced a possibly unrelated increase in the need to reauthenticate. I'm not surprised this sort of thing doesn't work at first -- the cooperation requirements are very steep.

OS X accessibility - radio shortcuts, shortcut cleanup, voice over and magnify

Continuing in the theme of OS X accessibility (see also VisiKey and magnify toolbars), I've several additional recommended modifications to add to my old OS X accessibility configuration document (one day):

1. Keyboard shortcuts: OS X has numerous kb shortcuts. For many elders or persons with visual impairment they can be fumble finger traps waiting to confuse with unexpected behaviors. I turned off almost all of them on my mother's machine.
2. Voice Over: The Leopard "Alex" voice is a good improvement, and Voice Over itself is one of the rare true improvements in 10.5 over 10.4. I configured it to use the new voice and the Caption Panel. I mapped Voice Over toggle to the F13 key on my mothers VisiKey kb (underneath the key reads something like Print Screen, but to OS X this is the F13 key). She has kb stickers for visually impaired persons, but they're not needed for the VisiKey kb. Instead I pasted the V letter on the F13 (for voice over). She can read that. She doesn't like using voice over, but I'm hoping she'll get accustomed to it.
3. Shrink/magnify: I map these to F14 and F15, and pasted the - and + stickers on them. I set Zoom to a shade below the 2, it's easy to hold the key and zoom up. Minimum zoom is 0. She runs on a 19" CRT because they do far better than LCDs at displaying 1024x768 over a large surface. Obviously true scalable UIs would be a great benefit.
4. Radio shortcuts: more below.

I'm very pleased with the radio shortcut. I rediscovered this myself, then found this explanation afterwards ...

How to create a radio shortcut using iTunes | sync :: the tech & gadgets blog

... while the built-in radio streamer isn’t much of a surprise to those who spent time navigating around iTunes, but what you might not be aware of a way to place an icon on your desktop that links you to your favourite station – and with added functionality...

1. Open iTunes and click on the radio tab on the left-hand side of the screen and select a station with the kind of music you like...

2. Once you have a station you like ... drag it onto your computer’s desktop (or copy and paste) and you’ll see an icon (shortcut) that immediately begins the audio stream when you double-click on it.

This is great, because iTunes, though improving now, is still hard for my mother to navigate. The drag and drop for stations (not, alas, for albums or tunes) creates .webloc files, which open in iTunes. I can mix these with shortcuts to BBC iPlayer stations (like BBC 3 and BBC 4 - there are some quirks there though) and with shortcuts pointing to the physical iTunes albums on her hard drive.

These are easy to navigate in the folder paradigm she's accustomed to.

Update: The radio shortcut has an odd side-effect. Each time you click on one of these it launches iTunes, but it also downloads a playlist file to the desktop from the source station. Kind of messy!

Managing a failing Canadian videotron cable connection

Many of the posts in this blog are of interest to very few people.

That's not an accident. There are some who subscribe to this blog, but it's really intended to be a set of references that work with Google. My most appreciated posts are often my most exotic. It's a big world now.

This post is very exotic. It will be of interest only to foreigners supporting a Canadian, well, maybe Quebecois, Videotron customer.

The background is that my mother, who lives in Quebec and is quite disabled, has an archaic Videotron modem. It was old when they installed it -- as a minimal-charge ($30/month for cable internet access) customer she may have been given a recycled model.

Her cable modem is now well beyond its service life; it's dropping connections every 1-2 weeks. The connection can be restored by power cycling, but it is very hard for her to get to the the power strip. More importantly, this is a typical way for a router/modem to fail. The connection drops will increase over time until the modem fails completely.

The device needs to be replaced. I thought I could just buy a new one during one of my periodic check-in visits. Wrong. This is what I learned ...

• You cannot buy a replacement for a failing Videotron cable modem. Actually, I did buy one at Future Shop, but that was a bad mistake. What I bought appears to have been forgotten inventory. Happily Future Shop did accept the return. Videotron should contact their past resellers and ask them to return their inventory. (Amazon US, by contrast, sells DOCSIS-compliant cable modems that are reported to work with many American ISPs.)
• Videotron has two sorts of retail outlets in Quebec. One sells movies and the like, the other sells services to new customers. Neither variety provides support, neither variety will accept an old device to exchange for a new one. I think if you discontinue Videotron service that it might be possible to return an old device to some of these outlets.
• Videotron "rents" devices. I'm not quite sure what that means. There's some complexity about a $99 fee that might be charged if one leaves Videotron, but maybe that's not charged if you return the device.
• Videotron's support model is entirely on their installers and onsite visits. You can do small things with their reasonably well staffed support people, but device problems require a visit. The usual routine is to call on one day, the service call is the next day. So someone has to be home. They will typically phone a brief time before a service call. I have a hard time imagining how people can arrange to be home like this.
• Videotron has a well staffed support line but many of the staffers are very new. Even the managers are fairly new; they were all flummoxed by the Future Shop device I bought -- that was before their time. (Just to make things harder on Videotron's support staff, I am effectively unilingual English. Quebec is a French province/nation with a slowly shrinking English minority. All of the service people are speaking to me in an alien tongue.)

Update: When the Videotron service guy arrived, he confirmed all was well outside. He seemed at first mildly skeptical about replacing the modem -- until he saw it. He claimed it was 15 years old, which I think is impossible. Maybe 8. He put the tiny new one in place and started to leave -- until I showed it didn't work. Yes, dead out of the box. So we pulled another toy out, and that one works.

Can't select Jabber or Google Talk for iChat? Here's one reason.

I really felt like crying when I ran into this latest bit of Apple tragi-comedy.

I tried configuring iChat on my mother's managed account (protect the Dock from accidental deletions), but I couldn't use her Gmail credentials (Google Talk option). Jabber and Google Talk were grayed out.

Why?

Google tells us ..

Apple - Support - Discussions - Can't add jabber or google talk, ...

... Jabber [and Google Talk] and in Fact Bonjour over iChat are excluded in Leopard when Parental Controls are activated...

It doesn't matter that the Parental Controls have no restrictions on iChat or the web. If you enable parental controls, even if all you're doing is protecting the Dock from changes, then iChat can't use Google Talk.

Why not? Why this senseless, irrational, bit of blithering madness that's persisted, without documentation, through 7 point updates to 10.5.7?

Because Apple hates us.

There's a comparable mysterious "Gray out" in iTunes related to iPhone parental controls, but at least that makes a kind of sense.

I hate you too Apple.

Update 4/17/2010: An Apple Discussion post describes enabling Adium functionality even with Parental Controls:

http://*.*.live.com
https://*.*.live.com
http://messenger.live.com

By adding the above we were able to Adium to work while still having parental controls turned on.

See also: OS X Parental Controls: The https bug and our family Google Apps services.

Accessibility in 10.5.7 - the magnify toolbars and VisiKey

(See accessibility posts for prior tips.)

I've upgraded my mother's Mini from 10.4 to 10.5.7 and installed a VisiKey keyboard.

I made the move to 10.5 because 10.4 is nearing end of life, and I thought 10.5 was becoming reasonably well baked (wrong, wrong). I also wanted the option of using 10.5's mediocre iChat in place of Google's elder unfriendly Google Video Chat. Lastly, since I no longer have a 10.4 machine at home and I use LogMeIn to manage her machine I wanted her on the same OS as our family.

I installed the VisiKey because her macular degeneration has progressed far enough that the need to see the keys has overcome her fondness for the cool look of her Apple keyboard (she's not a geek, she really does like "coolness").

The VisiKey's not bad, but there's a bug in the driver installer. In a multi-user machine you have to manually add the VisiKey driver to each user's LogIn list. Without the driver most of the kb features work, but not the "Internet", Email, and Search buttons.

Although I'm no great fan of 10.5 (and believe me, I'm going to take my time with 10.6!), there are a few accessibility improvements. In several app toolbars (Safari 4beta, Mail.app 3.0) there are Smaller/Bigger buttons like these (grayed out here so very murky):

Ok, so I lied. They're not automatically there, you have to customize the toolbars to get them (right click  on toolbar then choose customize). You have to modify the toolbars in Mail.app for browsing, new message, reply, etc. In some cases, like "New message" you can add these controls but they don't seem to do anything [1]. In reading mode, however, they do work [1].

It helps.

I also found the album view in iTunes 8.1.1 isn't bad for low vision use.

So a few accessibility improvements, though so far they don't outweigh the misery of 10.4 to 10.5 migration for me.

Update 5/31/09: There are bugs here. In some modes they enlarge all text, in other modes you have to select the text first. Looks like the responsible dev teams weren't always on speaking terms.

OS X Printer driver problems with 10.5 (Leopard) - the HP 1012

I'm having just so much fun upgrading my mother's vanilla Mac Mini to 10.5. First it was the buggy Mail.app update, now I find her Apple's 10.5.7 (Leopard) HP 1012 printer driver doesn't work. The printer worked perfectly with 10.4 (Tiger) - of course.

Print jobs pause for a time, then there's a printer response, then they hang, then they just ... stop.

Of course I have lots of company:

• Apple - Support - Discussions - HP 1012 printer & Leopard ...
• Can't print to HP 1012 with Leopard

I've tried a few fixes, such as resetting the print system (right click on printer in print and fax preference display, choose reset). I also found that the printer was shared by default but that there was a "Printer Sharing is turned off" message; I turned off sharing.

I seem to have fixed the problem for the Administrator account, but not for a regular user account. I may try promoting the user to Administrator, seeing if I can fix it, then trying them again as a regular user.

Power cycling the printer, or clicking on hold/resume a few times, will restart printing. Neither is a good option for my mother of course.

This 10.5 update has helped me think differently about 10.6. I'll take a look at that one in 2011. Of course that means I'll need to buy my new machine while they're still shipping with 10.5 ...

I'll update this post if I'm able to fix the problem ...

Update 5/29/09: At the moment things are working. It is a weird situation, however.

To recap, I was able to print successfully from my admin account using the installed printer drivers, perhaps because (from my Software Update Preference Pane Installed Update history) Apple released an HP Printer driver update in September 2008. I couldn't, however, print from my primary non-admin account.

Here's where it gets tricky. In order to print from my admin account I'd had to reset the printing system.

Even after I did that though, I couldn't print from my mother's non-admin account on the same machine. My hunch is that with the 1.1.1 update if I'd also reset the printing system from my mother's account it might have worked.

Yes, reset from the separate accounts.

Why do I think that might have worked?

Because, instead of doing that I installed the Gutenprint drivers (per Apple). Hint: Don't waste time trying to figure out the install directions, reading the manual, etc. The current version has an installer that does all the work for you, and, for you Gimp veterans, there's no longer any need for Ghostscript, web configuration, etc. All the directions and tips you need are in the installer documentation folder (DO read the readme file).

After installing the Admin account worked fine -- but the user account still didn't. I reset the admin account (again), but still only the Admin account worked.

That's when I did a reset from the user account as well as the admin account. Then I could print from the user account and the admin account. (Interestingly other accounts I created on the machine were also able to print, without a reset).

I'll update this post again after I've had some more experience -- and to see if it still works post reboot.

To recap -- before you try the Gutenprint be sure you have the September 2008 Apple update and try resetting the printer on EVERY account that has trouble -- which means adding back the printer definitions multiple times.

If you still can't get the Gutenprint drivers to work, this post may help though it didn't apply to me.

My hunch is gunk in the queue -- like maybe permissions gunk.

PS. Early in this process I even "repaired permissions". As usual it did nothing but suck time. I think the OS X "repair permissions" utility is some sort of sick Apple joke.

Update 5/30/09: It's not really fixed, after a day or so I got only the infamous "PCL: Unsupported Personality" atop all printed pages. This time adding and removing printer while using Gutenprint had no effect. So I tried it with the Sept 2008 Apple HP drivers and they worked. I also turned on printer sharing, just for kicks.

Clearly we need a new printer. This won't work for long.

I'll probably buy the Brother HL 2140 for my mother. Neither Canon nor HP can produce OS X device drivers to save their shriveled little souls.

Update 9/3/09: Unsurprisingly, it stopped printing a week or so after I left my mothers. I replaced it with the Brother. Weirdly, this printer is showing as supported in 10.6 (CUPS)! I don't believe it, but if you have 10.6 and test it out please let me know in comments. I ended up buying the Brother HL-2170W for my mother -- the 2140 Amazon comments weren't that encouraging.