Saturday, October 04, 2008

Reconsidering Google: life without customer service

We have a lot of our personal data invested in Google.

Gigabytes of email. Gigabytes of photos. Thousands of blog postings. Six Google Apps domains, including our family domain and Minnesota Special Hockey. Maps. Googe group posts. Google Notebook items. The Family Calendar. Contacts.

A lot of stuff. Stuff tied to a single user name and password.

With no customer service ..
Digital Domain - Can’t Open Your E-Mailbox? Good Luck - NYTimes.com

... If you’re a Gmail user, what you’ll want to do after a few more unsuccessful, increasingly frantic attempts is to speak with a Google customer support representative, post haste. But that’s not an option. Google doesn’t offer a toll-free number and a live person to resolve the ordinary user’s problems.

Discussion forums abound with tales of woe from Gmail customers who have found themselves locked out of their account for days or even weeks. They were innocent victims of security measures, which automatically suspend access if someone tries unsuccessfully to log on repeatedly to an account. The customers express frustration that they can’t speak with anyone at Google after filling out the company’s online forms and waiting in vain for Google to restore access to their accounts.

Tom Lynch, a software entrepreneur who lives near Austin, Tex., discovered early last month that he had been locked out of both Gmail accounts he used; he had no idea why. He received boilerplate instructions for recovering his accounts that did not apply to his particular circumstances, which included his failing to maintain a non-Gmail e-mail account as a back-up. He said it took him four weeks, including the use of a business directory and talking with anyone he could find at Google, before he succeeded in having service restored....

... Google does provide phone support to Gmail customers who subscribe to Google Apps Premier Edition, which costs $50 annually and includes larger storage quotas and other benefits. Customers who use the advertising-supported version of Gmail, however, must rely solely on what Google calls “self-service online support.”...

... Last month, with cases like Mr. Lynch’s in mind, I contacted Google to see what the company had to say about my suggestion that it add phone support for its customers with account-related problems. The company returned with a debate team of three to argue the negative position: Matthew Glotzbach, who works with Google’s business customers; Roy Gilbert, who handles consumers; and Greg Badros, who is an engineering director.

Mr. Glotzbach began by saying that “one-to-one support isn’t always the best answer” because it would take Google too long to collect lots of data about a problem that is affecting many users simultaneously.

For systemic problems, data collection is important. But not for other categories. Account recovery could be slow for a locked-out customer who doesn’t have a backup e-mail account, and who declined to provide a security question and answer because of concerns that someone else could use it to get in (which is what someone did to Gov. Sarah Palin’s Yahoo Mail account).

Mr. Badros argued that Google asks so little personal information of a new Gmail customer that it’s hard to determine identity when the genuine user and the impostor both present themselves to claim the account, and neither can produce the verification. He said more information could be asked of users when they sign up, but the inconvenience would dissuade them from trying the service.

Mr. Gilbert added that proving identity with only minimal information is a problem, whatever form of communication is used to reach customer support. He said, “Even if they were standing right in front of us, it wouldn’t help.”

THIS makes sorting out competing claims seem permanently hopeless, when, of course, this is not the case; it simply means that standard security questions will not suffice. But if Google were to use real people to sort out identity problems over the phone, the only remaining consideration would be the one that Google’s panel of experts didn’t mention in our talk: cost.

Google says it has “tens of millions” of Gmail customers. (It declines to be more specific.) If it’s willing to consider phone support for account-access emergencies, it can take heart in the example of Netflix, which last year adopted phone support with enthusiasm, replacing online support completely. For all customers. For all problems. And without resorting to an offshore call center.

It turns out that a staff of 375 customer service representatives are enough to handle calls from Netflix’s 8.4 million customers, answering most calls within a minute. Netflix says with justifiable pride that it has received the top ratings in online retail customer satisfaction by both Nielsen Online and ForeSee Results....
I pay Google for extra storage for my Gmail and Picasa Web Album accounts, but that still doesn't get me any customer service.

As noted above there is customer service associated with upgraded Google App accounts, but the price is $50/user/year. So $200 for our family. The commercial Google Apps accounts are really aimed at corporations; they aren't a reasonable solution for families.

We do get free Google Apps Educational/Non-Profit service for our Minnesota Special Hockey site (free upgrade for non-profits). I can confirm the email response is very fast. You get corporate-grade support.

I don't believe the line in the article about "no falsely recovered accounts". The world doesn't work that way. There are no perfect tests. If Google really hasn't had any "falsely recovered accounts" that means they have shut out thousands of legitimate account owners.

On the other hand, kudos to this journalist for noting that anyone who fears losing their account won't use Google's obscenely inane security question, but if you don't answer the question then you have no hope of account recovery. (I've gone to my Gmail account and answered the question with a password-like string I now store with in my backed-up password database.)

Google should offer a support service with enhanced user authentication procedures for a fee of $25 a year, and bundle it with an extra 5-10GB of storage.

If they don't do that, I'm going to have to reevaluate my Google relationship.

Update: I reviewed https://www.google.com/accounts/EditUserInfo after reading this story.

Google has added a lot of new features to the Google Account information since I created my account years ago. I changed the security question to "what is your secondary google password?" and gave it a 50 character grc.com generated random hex string.

I then added additional email addresses that I control through my DreamHost domain and completed all the "optional" identity related questions. These email addresses are distinct from whatever extra email addresses may have been defined in Gmail settings. These addresses are associated directly with the Google account. They may act like a kind of merged identity.

I use unique passwords on the two external services. One is outside of Google completely, the other is in a distinct Google Apps account. So for now I feel a bit better. I appreciate the warning!

Symmetry: Apple and Microsoft kb and the wonders of Google Custom Search

When addressing technical OS problems, the Apple and (especially) Microsoft knowledge bases are often the best place to start.

Unfortunately Google searches for topics covered in Microsoft KB articles are often obscured by lots of ad funded pseudo-splog "tech sites". It takes some digging to find the good stuff.

That's the beauty of my favorite Google custom search engine. I simply add the URLs for those knowledge bases and Google boosts those results above the noise.

Wondrous, really. The custom search engine Searches my Google Reader feeds and 13 other sites including: support.apple.com/kb/, http://support.microsoft.com/kb/, www.sciam.com, bestyoucanbe.blogspot.com, tech.kateva.org ...

Even the Encyclopedia Britannica (since we still subscribe) gets boosted and included in our personal version of Google.

Which brings me to the title of this post. On entering the URLs I noticed a funny symmetry ...
support.apple.com/kb/
support.microsoft.com/kb/
Chance probably, but given all the ways in which Microsoft and Apple are trading innovations (admittedly more from Apple to Microsoft) this caught my fancy.

Friday, October 03, 2008

An iGoogle Gadget that can display your Google Apps Calendar

I've been looking for months for an iGoogle gadget that would display our family google apps calendar.

I finally found one by searching on "google apps calendar".

Dang, but it's hard to find this stuff. Definitely an unsolved problem. Here's the link, with my review ...
Google Apps Calendar

... I only found this by searching on Google Apps Calendar. I'll promote it on my blog. Standard Google Calendar apps all assume they're displaying the calendar associated with one's Gmail account. I want to display our family domain calendar; I have access privileges from my Gmail account. This does the trick....

Windows Server 2003 – read this if you abruptly lose network connectivity on a restart

I rebooted our corporate Windows Server 2003 today. I was moving it to a UPS. No problem – except when I restarted I had no network connectivity.

First I saw a “service didn’t start, check the event viewer” message. The event viewer just told me I couldn’t register with the domain. I couldn’t do that because I didn’t have network access. I got the usual “may have limited connection” error.

I did all the usual things (ipconfig, repair connection, swap cables, switch accounts, login as local user, test everything, etc etc) but they all passed. The big breakthrough was when I investigated the advanced boot options on restart. Windows 2003 includes a “safe start with network” option. When I did that I had a network connection.

There was a lot more work to do before I found that disabling IPSEC service, then rebooting after disabling it, fixed everything.

I easily blew 6-8 hours of work today.

Lesson 1: Run Safe Boot/Safe Start with networking first.

Then you work your way through this Microsoft kb article. I’ll excerpt some key points, then pass on a trick, then I’ve got to go home and finish up the work I couldn’t do today …

How to troubleshoot startup problems in Windows Server 2003

How to Start the Computer in Safe Mode
When you start the computer in Safe mode, Windows loads only the drivers and computer services that you need. You can use Safe mode when you have to identify and resolve problems that are caused by faulty drivers, programs, or services that start automatically.
If the computer starts successfully in Safe mode but it does not start in normal mode, the computer may have a conflict with the hardware settings or the resources. There may be incompatibilities with programs, services, or drivers, or there may be registry damage. In Safe mode, you can disable or remove a program, service, or device driver that may prevent the computer from starting….
How to Use System Configuration Utility

System Configuration Utility (Msconfig.exe) automates the routine troubleshooting steps that Microsoft Product Support Services technicians use when they diagnose Windows configuration issues…

… Click the General tab, and then click Selective Startup.

…Note You might be able to determine more quickly which service is causing the problem by testing the services in groups. Divide the services into two groups--select the check boxes of the first group, and clear the check boxes of the second group. Restart your computer, and then test for the problem. If the problem occurs, the faulty service is in the group with the selected check boxes. If the problem does not occur, the faulty service is in the group with the cleared check boxes. Repeat this process on the faulty group until you have isolated the faulty service.

It took hours.

Here’s the trick. Boot in Safe Mode first. Then run msconfig.exe and look at the services. Assuming things work in safe mode, the ones that are running (sort by that column) are good. Now uncheck all services, check the ones that are currently running, apply, restart.

When you restart you’re in the equivalent of Safe Mode, but you can use msconfig.exe to add services in blocks.

The UI of this app is dismal. I sorted alphabetically, then did screen captures to a Word document to get a complete alpha sorted list. I printed that to guide my tedious enabling of sets. (In theory you can do the binary sort approach faster. Long story, can’t explain.)

One thing to watch for.

When you enable “Error Reporting Service” you start getting … error reports! Wow. So if gets enabled with a bunch of other items, you might think you’ve found a problem. Wrong. It’s just that now you’re getting the error reports.

IPSEC.

So now I have to figure out what the #$!#% happened. I don’t think we’ve done any software installs on that box or tweaked any services. Did some antiviral update trigger a problem?

Update: This experts exchange article may be related, but the responses are not accessible. A clue:

Description: The IPSec driver has entered Block mode. IPSec will discard all inbound and outbound TCP/

IP network traffic that is not permitted by boot-time IPSec Policy exemptions. User Action: To restore full unsecured TCP/IP connectivity, disable the IPSec services, and the restart the computer. For detailed troubleshooting information, review the events in the Security event log
This suggests an interaction between Group or Local security policy, IPSEC block mode, and loss of network access. I wonder if a corruption or misconfiguration of a local policy setting could cause this.

Update: This article connects group policy file corruption to IPSEC problems and loss of network access, and points out there are definite bugs with group policy editing. I didn't touch local or group policy on our server, but perhaps another admin might have. I now see there have been nasty unfixed bugs.

Update: I'll take a look at these when I get back to work on Monday, then update this post. I think we're narrowing things down to a corruption of misconfiguration of a group policy file that activated IPSEC and disabled, without any meaningful entry in the event monitor, all network TCP/IP traffic.
  • http://support.microsoft.com/kb/870910: looks like a pretty pertinent kb article
  • http://support.microsoft.com/kb/914962: IPSEC bugs fixed in SP2. So did some later upgrade break them again? Clearly I need to check windows update for the server.
  • http://support.microsoft.com/kb/898060: After SP1 a security update broke IPSEC. Should be ok in SP2, but did it get broken again?
  • http://marc.info/?l=patchmanagement&m=121632162501913&w=2: A fairly recent DNS spoof prevention security update from Microsoft has broken IPSEC on some machines.
  • http://support.microsoft.com/default.aspx?scid=kb;en-us;816579: In place upgrades when WS 2003 is truly hosed. I don't think this applies, but nice to know.
Lots of evidence that the Windows 2003 IPSEC architecture and TCP/IP stack are pretty fragile. No wonder Microsoft famously redid the network stack in Vista. They weren't reacting to XP, they were reacting to Windows Server.

So Monday I'll look at windows update and try opening, reviewing and savng the IPSEC and Group Policy files. If they're corrupted they may cause other problems.

Update 12/14/08: I'm grateful to an anonymous visitor for finding the underlying issue. S/he references two Microsoft kb articles, I've added a less important but related third article.
A botched security update 953230 (MS08-037) causes a variety of Windows 2003 failures due to a UDP port conflict. Essentially Microsoft switched to random port assignments, which is good, but they forgot some ports might be in use (bad). Depending on what gets randomly whacked, you may lose a service.

The latter references the problem I had:
Event Type: Error
Event Source: IPSec
Event Category: None
Event ID: 4292
Date: Date
Time: Time
User: N/A
Computer: Server_name
Description: The IPSec driver has entered Block mode. IPSec will discard all incoming and outgoing TCP/IP network traffic that is not permitted by boot-time IPSec Policy exemptions.
User Action: To restore full unsecured TCP/IP connectivity, disable the IPSec services, and then restart the computer. For detailed troubleshooting information, review the events in the Security event log.
Update 12/31/08: Nope, it didn't work.

I finally got around to applying Microsoft's fix and it didn't work!

So even after I reserved these ports:
3343-3343
1645-1646
1812-1813
2883-2883
4500-4500
I still got the service failure notice on restart and lost my network connections. Guess I'll have to wait for a service pack. I removed the registry changes I'd made (why ask for trouble?) and again disabled IPSEC services.

Thursday, October 02, 2008

Google Apps calendar on the iPhone - the top secret web display

It's not the OTA blackberry-like iPhone gCal sync we want, but it's something ...
The official update feed from the Google Apps team: New Google Calendar features for the iPhone

... Google Calendar users in the US can now add new events, invite attendees, and see daily and monthly views of their agendas from the iPhone. This release also includes speed improvements for the iPhone interface....
Until now the webapp was read only.

In an essay on the darkness of Apple's App Store policies Gruber mentions a few more calendaring options ..
... Apple doesn’t seem to have any problem allowing Calendar competitors into the App Store. Notes Calendar is a $3 Lotus Notes calendaring client. iExchange Remote Calendar is a $10 calendaring client for Exchange. It can’t even be explained by some sort of anti-Google bias at Apple, because they’ve also accepted SaiSuke, a $10 dedicated Google Calendar client. If these are OK, why not a dedicated Gmail email client? The only explanation is that Mail is deemed untouchable and Calendar is not...
Update 11/10/08: The iPhone Google App didn't show me the new calendar. I had to use the URL: http://www.google.com/m/a/faughnanlagace.com (our family domain) to see the new calendar. You can also use: http://calendar.google.com/a/faughnanlagace.com/m. There's still no calendar search -- an odd omission.

iPhone audio recording: Plum Record, Audio Recorder, others?

Eons ago I used to record conversations on my third generation iPod (not to be confused with iPhone 3G. I can't recall if I was able to transfer the audio files to the desktop.

It worked quite well -- until Apple obsoleted the hardware connection!

It finally occurred to me that I could do that again. The Monster adapter I bought for my BOSE headphones works just fine as a standalone microphone - I just have to unplug it if I want to hear playback without headphones. (I wouldn't mind finding a direct cable connector for analog to analog transfer though. I don't need stereo recording ... yet ....)

Plum Record will record files that can be transferred to the OS X desktop and translated there. Audio Recorder transfers by email but uses native iPhone audio formats (.caf, uncompressed). (Yes, we all want access to that damned USB cable.)

The Audio Recorder FAQ says the iPhone will work with a standard mini-jack mike.

I purchased Voice Recorder a while back but took it off my phone. I'll have to see if they offer any audio-transfer options.

One reason we don't see gSync for the iPhone

We all want Blackberry-like Google Sync to Google Calendar.

Here's one reason why we're waiting ...
A touch of Cocoa: inside the iPhone SDK: Page 2

... Apple also provides access to some system-wide data in the form of the address book, with both model and view classes exposed to developers. The equivalent classes for the calendar data, which only recently appeared in the desktop OS, are missing from the iPhone. Here's hoping we don't have to wait for 3.0 for them to appear...
Since Google Sync would be competing with MobileMe there may be other obstacles to App Store distribution, but first we need an iPhone API for Calendar.app.

See also SyncML.