Debugging network account lockouts: issues with Microsoft Active directory authentication

I recently experienced a personally new and novel set of computer network related issues. I'll have more to say on Gordon's Notes about my take on the implications and lessons of this experience, but on this blog I'll stick to measures end-users might take.

If you're reading this I'll presume you are a user on a corporate network and you are now unable to get at network services. If you request a shared drive or other network resource you are asked to provide your credentials (username and password). You may be unable to login to your workstation while you are connected to the network, though if you pull the network cable or disable wireless access you can login locally.

In this case it is likely that your network authentication is failing. Your credentials are not valid, you've been locked out.

There are legitimate reasons to be locked out of course, but most of the time this is an error. A Microsoft Active Directory group policy setting in your organization specifies an allowed number of failed authentication attempts in a certain time interval and "you" have passed that limit.

By "you" of course I mean whatever is trying to login with your username -- but not your current password. The problem, you see, is that many things may be doing that. Some may be on your machine, some may be on other machines you've used or use, and some may be in places you can't imagine. One of these things may be you, of course, entering your password incorrectly more than, say, five times, in a certain interval.

Ahh, but you say you only made one mistake? Well, maybe something else was trying four times in the key interval. Your one mistake was the last straw.

This is a big problem. You'll find many hits on the topic if you start looking. It's a Cloud problem (new tag today!). It's what happens when authentication starts to diffuse, and when you don't have a robust system for distributing authentication privileges. It's what happens when credentials are cached or distributed, and there aren't robust tools in place to monitor and track -- or when organizational structures block recognition.

Microsoft has tools for diagnosing active directory account lockout issues, but they are not accessible to end-users like you ...

• http://technet.microsoft.com/en-us/library/cc738772(WS.10).aspx
• http://www.microsoft.com/downloads/details.aspx?FamilyId=7AF2E69C-91F3-4E63-8629-B999ADDE0B9E&displaylang=en
• best overview (private blog).

As an end-user victim these are some things you may investigate once your help desk has unlocked you. Good luck ...

• Change your network password, that may fix some caching issues.
• OS level drive encryption software, bolted onto a decrepit XP infrastructure, can be a problem. These typically synchronize credentials with Active Directory -- and we all know synchronization is Hell. Look into any associated logs that might show how synchronization is preceding. See if you can change your password using the UI controlled by the encryption package and watch that propagate to the server.
• Group policy updates may be failing, resulting in passwords failing to comply with standards and leading to rejection. Research use of the gpudate.exe /force command to update local copies of corporate policies.
• Eliminate all drive letter mappings on all machines. I know longer do drive letter mapping on corporate networks. These can have cached credentials that fail to update.
• If you use Remote Desktop, log in to every RD machine you use and make sure you are fully logged out again. You may need to apply all fixes and patches there as well.
• Try shutting down your main workstation when you are not at working -- or disconnect it from the network. If you're locked out then you may suspect the problems are from other sources.
• Do not use Windows Search to index mounted drives.
• remove all IE stored information - cookies, passwords, etc. Used Delete All from the IE General/Delete settings tab. Note this is the ONLY option if you want to be sure to remove any stored credentials from IE Add-Ons.
• Consider uninstalling any applications that authenticate with Active Directory, such as Office Communicator.
• Evaluate all applications that might interact with Microsoft Sharepoint, because these require Active Directory authentication. This may include:
  • Windows Live Writer: Posting from WLW to a SP blog implies an authentication event
  • Lotus Connections: If you use Lotus Connections web-based feed reader against a Sharepoint feed there's an implied authentication event. (In my testing these subscriptions appear to fail, but does LC attempt to authenticate with its internal credentials? What about if the user IDs match between LC and SP but the passwords differ?)
  • Outlook 2007: Outlook 2007 is able to subscribe to SP Calendars and other Sharepoint Lists. All of these imply authentication. Prior to SP1 Outlook 2007's subscription/feed support was extremely buggy.
  • Any feed reader that works against Sharepoint authenticated feeds

Personally, this is the nastiest problem I've come up against in 25 years "behind the mast". I'll have more to say in my opinion-oriented blog about how this has changed my approach to personal and cloud computing and to the new approaches I'm taking to risk mitigation going forwards.

Update 6/2/09: Focus is now on a combination of a Sharepoint List synchronization that could not be removed from Outlook 2007, a possible configuration error on Sharepoint, Outlook configured to send only on manual send/receive, send/receive configured (by default) to include the unremovable SharePoint list, and Outlook offline caching of credentials.

Update 6/4/09: Microsoft Wireless PEAP always caches credentials. Could be a contributor in some situations.

Update 6/12/09:The saga continues. To remove the long stuck Sharepoint list in Outlook 2007 I had to remove reference to it in the Outlook send/receive group. In fact, I removed most things from that group. The NTLM account lock problem went away -- but I then had to manually authenticate the first time I used Outlook to access Exchange server. In other words Outlook was no longer able to deliver my credentials automatically. (The advanced security settings for Outlook did not have "manual credential" checked.)

After a week of this my laptop was refreshed. Using a brand new image I was again locked out. (I did have to install Retrospect Pro to restore data, but I think the first lockout was before I restored anything.)

This went on for a few days, then I did into an obscure option in Outlook 2003 (and 2007) properties and set Outlook to always require manual credential entry. The account lockouts stopped.

I'm going to study this for a few days, and see if I can get locked out by turning off manual credential entry. If I can confirm this does the trick, I'll try to bring very specific fix suggestions to our puzzled help desk and security services. I need to better understand the NTLM/Exchange/Outlook authentication procedure.

Update 6/12/09b: I've asked this question on serverfault.

Update 6/27/09: I post an answer to my own serverfault question:

... I've not been locked out for over a week even after turning re-enabling Outlook pass-through authentication, so even though there was no definitive cure I can report where I left things.

As a reminder, the last time I was locked out I'd just received a brand new laptop with a fresh corporate image.

The very last things I did were:

1. I found the brand new corporate image included two drive mappings. Sigh. (Sound of head hitting wall.) I'd removed them from my old laptop long ago, but they were back. I removed them again. It wasn't the only problem in the corporate image.

2. I experimented with switching Outlook 2003 authentication between "automatic" (default), Kerberos only (modern) and NTLM only (legacy). Switching to Kerberos only seemed to resolve problems, but I think that was a red herring. Switching back to the default didn't restore the lockout problem.

3. I use Retrospect Professional (EMC Retrospect) for Windows to backup my workstation to an external drive. (Corporate backup isn't bad, but restore takes about a week.) That software has an autolaunch feature. I'd set it to auto-launch using the logged-in credentials rather than the treacherous feature of providing credentials. I wonder though about an intersection between the mapped drives and the auto-launch. I turned off Retrospect Pro auto-launch for now.

I very much appreciate the link Neobyte provided to Microsoft's June 2008 troubleshooting page - Troubleshooting Account Lockout

http://technet.microsoft.com/en-us/library/cc773155%28WS.10).aspx

I'm left with some psychic scars. Given the astounding variety of problems associated with Microsoft's authentication services and their pile of legacy hacks, and the intersection with distributed authentication and post-hoc security features like authentication lockouts, I'm now deeply conservative about my use of any new or novel corporate network or "cloud" initiatives. They need to be built on a far more robust infrastructure than what Microsoft provides, and they require both IT funding and IT reorganization to implement.

7/21/09: I found yet another potential contributor -- one I'd long forgotten about. I'd once set up my iPhone to connect to the corporate WLAN. To do this I had to enter my Active Directory login credentials. The iPhone connects automatically when the WLAN is in range. So what happens when my network credentials change and the iPhone tries to connect? I'm not sure. Maybe it fails once and doesn't try again -- generating only one lockout hit. Maybe it tries repeatedly. Who knows. The point is, we're screwed.

We need better ways to manage user authentication and privilege control, and we need them desperately.

As for the iPhone, there's no way to have it remember network credentials yet not automatically connect when WiFi is enabled. So I deleted my corporate WLAN credentials from my iPhone.