Mass archiving of thousands of Gmail messages - working around Gmail's new bugs

A family member had accumulated over 6000 messages in her Gmail inbox. No problem thought I, I'd just use the old "Select All" trick and be done with it.

Not so fast. Turns out this is broken in Gmail 1/2014. Actually, it's worse than it was in Nov 2013, I added an additional correction to the Stack Exchange thread [1]. 

Google now returns only 45 messages at a time, regardless of search critera. This worked:

1. Turn off special boxes, turn off conversation mode
2. Do search: before:2014/1/1
3. Use Select All box
4. NOW, only now, I see "Select all messages that match this search". Still see only 25 results. Click archive button. See "Loading...."

The Google still needs us, but increasingly it despises the humans it once serves. We know how this ends.

Seriously [2], The Google isn't into its old web apps any more. It is an AI/robotics company now, leaving the limitations of a merely digital existence behind.

- fn -

[1] In Jan 2014 only personal blogs, Stack Exchange, and Apple Discussions are healthy information sources.

[2] Ok, I was serious on the first one too. It's not personal, it's just corporate evolution in action.

Persistent performance issues on a friend's iMac: eternally stuck spotlight indexing

A buddy of mine is more tolerant of computer problems than I am, but he's been dropping hints that he'd like me to take a look at his Mac Mini. He's an AS/400 guy, not a Mac geek.

So I dropped by and did the usual hygiene on his Snow Leopard Mini (SL is a great OS, so no rush to change that). Turned off Spotlight (make boot disk "Private"), restarted holding down D key to run hardware test, restarted holding shift key to run Apple's Safe Mode (startup volume check), ran software update, checked Flash versions, ran Onyx.

After that it seemed fast and smooth. So I restarted Spotlight and saw "Estimating Indexing Time" in the Spotlight drop down. Not just for seconds -- for minutes. Spotlight was stuck. 

One solution, is to move all 3rd Party Spotlight Importers out of /Library/Spotlight username/Library/Spotlight. He had 3: iWork, Office, and Parallels. I moved them all out and Spotlight quickly estimated remaining time and completed indexing.

I think his months/years of performance issues came from Spotlight constantly reindexing/getting stuck. Naughty Apple -- there should be a better way to deal with flawed 3rd Party products. (Maybe there is in Mavericks.) If I had to bet I'd wonder if the versions of Office docs created by Open Office somehow cause problems for Microsoft Office's mdimporter.

That's quite a bit of suffering for a bad 3rd party Spotlight indexing tool he didn't know existed. It's also illustrative of how hard it is to maintain a modern computer; what was a minor cleanup for me had stalled him for months. One reason iOS is so popular -- and so closed.

Cleaning up a Twitter account may no longer be possible

Eons ago I wrote using my TrueName. In those days the sun was brighter, the snow softer, Google was good, and unicorns danced on rainbows.

That was a long time ago, in the Second Age. (In the First Age, there was no spam.)

Now I write as John Gordon [2], and my TrueName net content is the essence of bland. Except for a forgotten TrueName Twitter account I'd used for a few months after the fall of Google Reader Social [1]. That one is a bit spicy, and fully available to curious customers and employers.

Be nice to be able to clean that up and make it a part of my corporate persona. Once upon a time I think it was possible to do that, using things like Twitwipe, delteallmytweets, tweetdelete and so on. These days, however, those Twitter App sites are infested with spam and adware. Delete All MY Tweets seemed the least bad so I tried it.

It didn't work. I don't pay much attention to Twitter, but I do know as they turned to the Dark Side they did limit use of their API. I suspect none of these services work any more - I suspect they were sold and turned to the not-good-side. It may be possible to write an AppleScript or Python script to sequentially delete tweets using the twitter web app, but even there I suspect there are limits.

The best I could do was to Protect all of the Tweets on that account. That means anyone who wants to follow it has to be granted permission. Alas, because this is Twitter, current Followers are a problem. You can't simply return them to a non-follower state, you have to Block them. I didn't mind blocking the half that were spambots, but there were a few there who followed me from our Google Social days. Most don't seem to be active on Twitter any more, so I hope they aren't offended. 

One last bit of the old net facade fades away...

[1] My John Gordon Twitter account is active, though it's largely an echo of my much appreciated app.net account.

[2] Yep, G+. I have a few G+ accounts, but my primary TrueName gmail account is G-.

Switching my 2009 iMac to an SSD - six months later

What do you call a $1,500 computer that contains a fragile and unreliable component in the core of an unserviceable case? What do call the guy who designed it?

The computer is an iMac, and the guy, Johnny Ive, is often called a genius. Not by me though.

My iMac's first drive died after 2 years of use; I paid for an (badly executed) Apple store replacement and was later reimbursed as part of a recall program. Two years later that drive died.

Yeah, not happy. I hate spending the money, but the hassle of working with Apple service is worse. Backups, hauling the 17" iMac around, waiting, the restore, spending money for a drive that's 4 years obsolete (Apple only replaces 'like-with-like')... Ugh.

So I decided to explore my options. I ran off an external enclosure for a while; I was surprised how well that worked, though the still living internal drive seemed to trigger startup glitches. That gave me time to think; I decided I wasn't willing to throw the 2009 iMac out. On the other hand, if I was going to put more money into a 4yo machine I wanted more reliability, and I wanted performance that would carry me through 2017.

So I bought a 1TB Samsung 840 EVO SSD for (then) $650. I ran it in an enclosure for about a month -- if it failed I wanted a hassle free return. Once it passed the high mortality startup phase I paid First Tech Minneapolis around $140 or so to do the swap (See Ives: not a genius. Among other things, FirstTech says they install a temperature sensor that prevents the amok fan problem).

It was perfect. Well, actually, no. I started having kernel panics. Which had nothing to do with the SSD, one of my firewire cables was coming apart. It took an inordinate amount of time to figure that out. That wasn't the only problem associated with my migration, the process of swapping drives, backup, restore, and Mountain Lion upgrade uncovered lurking problems like...

Gordon's Tech: OS X Mountain Lion cannot delete sparsebundles containing over 262,144 bands (2TB+)

... after cloning the primary drive to an external firewire drive, I noticed an old Permissions bug I've probably ignored for years. Sophos Antivirus didn't seem to like the clone, and I decided it wasn't worth the bother anyway, which led me to Sophos uninstall heck (mostly missing documentation). Next building a Mountain Lion Install SD card for my new SSD exposed weird behaviors of the ML Installer...

... I decided it was a good time to move my iTunes Library to an external firewire drive. That meant I needed to make some room. No problem, I could just delete a 2.4TB Carbon Copy Cloner sparsebundle...

Happily the last big problem was four months ago. So I've had a chance to see how things worked.

Which is very well. Aperture was getting painfully slow, now it flies. My VMWare Fusion XP used to thrash my hard drive, now it zooms. Overall this is the biggest performance improvement since I went from an 8086 to an 80386.

I can't speak for reliability -- the main reason I went SSD. For what it's worth, Samsung claims ...

Samsung's 840 EVO SSD uses TLC memory, yet because of the sophistication of the controller chip and its software, it will outlast any other component of the laptop or desktop it's in, according to Chris Geiser, senior product manager of Samsung's Memory and Storage Division.

"If I'm writing 10GB a day to a 120GB SSD, it will last over 10 years," Geiser said.

Ten years would be fine. Less biasedsources vary.

I think I made the right call. I spent about $800, but I got a high performing machine that should be good for 4-5 years -- not less because the heat output is diminished. A new iMac with faster CPU but overall equivalent performance (and no DVD!) costs $2,800.

There's only one obvious defect -- my iMac's 2009 hardware test now fails because a fan is not detected. Bummer.

Some bullet points from my installation are below...

• I did a fresh Mountain Lion install on the new drive including an admin account with a distinct name (you don't want to cause a name collision with the restore).
• Since I was running off an external drive I used Migration Assistant to move data over the SSD after I did my fresh install.
• I unwittingly upgraded to iTunes 11, but I've finally gotten used to it. Fortunately by the time I upgraded many of the worst bugs were fixed.
• I put my 340GB iTunes media on an external 2TB drive. I don't need SSD performance for that, external works fine, I have backups and I avoid using costly SSD space and I reduce write traffic on my SSD (writes shorten SSD lifespan)
• As with all drive migrations I had to delete my Google Drive data, reinstall Google Drive and let Google restore my files.
• I started out with an encrypted drive but I ran into problems with startup accounts. I suspect this was partly related to my fraying firewire cable kernel panic problem, but I ended up removing the encryption. I may try it again in a few months.
• I researched the various TRIM debates and decided not to use the Trim Enabler hack. If performance lags in a year or so I'll create a fresh image then reformat. I hate messing with core system functions.
• After migrating to the SSD Time Machine let me continue against my prior backup. That was a pleasant surprise.

How to clean up the Samsung Smart TV you shouldn't have bought

You shouldn't have bought that Samsung Smart TV [1].

You should have bought a dumb TV with decent speakers, a simple remote, and simple HDMI switch box so every user can effortlessly switch inputs [7]. Pair it with an Apple TV and, if you insist [4], something to stream Amazon video [2]. If you want to record over-the-air TV please report to the local police station [3].

Alas, you did buy the spyware-by-design [5] Smart TV. You gaze in horror at the crapware infested screen, remembering pre-iPhone mobile and HP winboxes. Set aside an hour or so, because you have cleanup to do.

Short of hacking the TV your cleanup options are limited to:

• Delete: Only a few of the apps can be deleted. I assume they didn't pay enough to Samsung.
• Move: You can create a folder to move some app icons too. Moves are slow.
• Lock: Almost everything can be 'locked', even things that can't be moved. You can't lock the bundled IE browser however.
• IE Browser only: If you have the patience, you can find a 'restrict' option in the browser settings. Set a passcode and enable nothing.

For the standard Lock and the IE Browser restriction you are asked to enter a passcode -- even though you never set one [6]. The default is always 0000, then you can change it.

 

The basic cleanup pattern is then:

• Use the Tools button (on remote) to create a folder
• Select items on screen with remote, click Tools, and see your options (Delete, Move, Lock). Delete when you can, Move when you can, Lock all you can't Delete.

At the end of the day you will be stuck with a number of apps on screen, but they will be locked and thus confusion is limited.

 

You really shouldn't have bought that Smart TV :-).

[1] I've had four hours of experience with post-1994 TV. Isn't amateurism wonderful?

[2] As of today neither Google nor Apple set top boxes will stream Amazon video. Of the options listed here the Roku has a good reputation. The XBOX 360 is abysmal, the original Wii did a good job. I assume Amazon wants to do their own hardware solution. Apple TV does Netflix well. For now we've installed Samsung's Amazon streaming app -- it's slow to start but has worked for a couple of days.

[3] Few now remember the short time when it was easy and inexpensive to record over-the-air (OTA) TV; VCRs dropped their advanced scheduling abilities in the early 90s. The Tivo era died at the hand of Cable and content owners, now there's a crowdfunded effort for OTA DVR with minimal results. For a brief time Samsung SmartTV supported recording to USB stick or drive, but this interfered with their revenue model and has been quietly dropped.

[4] Amazon Prime streamed video library is a very mixed bag. Movie selections are abysmal. Television is variable, but they do offer BBC and thus Dr Who. Apple TV has PBS - with some Amazon carve out exceptions. The media landscape today makes the Netflix DVD era seem a golden dream.

[5] During my cleanup process I ran into at least 4 EULAs; I assume they all grant Samsung the right to monitor everything we do.

[6] Samsung copies Apple extensively, but they need to copy more.

[7] Samsung's comparable dumb TV is more expensive than their Smart TV. I assume that's partly better components, but it's possible that the Smart TV cost is subsidized by the bundled video options.

See also:

• xkcd: HDTV
• xkcd: I Don't Own a TV
• New LED TV: Family photos look ghastly. Apple T… on App.net Apple TV shares blame here, but, of course, modern TVs are nowhere near as capable as modern computer displays.

Update 5/17/2014 - restoring the missing PVR (recording) function: The SamyGO TV firmware hacking project enables “Video Recording”. The download site is now at http://download.samygo.tv/. We have a Samsung UN40EH5300, per samygo.tv I think this is an “EH5300” model:

<TYPE><REGION><SIZE><YEAR><SERIES><VARIANT>[misc details]

where

U: LED

N: North America

Size: 40”

E: 2012

H: ? H series? EH series? (Wiki is confused here)

5300: variant 5300

Alas, the wiki SamyGo wiki makes it clear that hacking a 2013 Samsung is no trivial task. The relevant wiki page is a work in progress - as of 12/2012. Nonetheless, on the forum there are discussions on hacking it …

1 - One For All remote URC-7320 or similar

2 - Press magic key 3 sec then enter 0812 Press magic key 3 sec then enter 994 + magic key + 00020 + red key ( now red key is 3SPEED ) Press magic key 3 sec then enter 994 + magic key + 00027 + green key ( now green key is FACTORY )

with tv turned on press info in tv original remote and green key in ofa remote and thats all

The URC-7320 is no longer sold, but Amazon does sell a URC-8820. Elsewhere a blog post says using the IR remote hack is particularly treacherous…

I found a discussion on enabling EH5300 PVR, but I think this is a generic direction, I can’t see it’s been used on a 5300.

1.- Download ruSamsungTVCommunicator

2.- Connect your TV and your PC to your local network.

3.- Open ruSamsungTVCommunicator and wait until it finds your TV (If it doesn't find it, you'll need to manually enter its ip address)

4.- A prompt will be shown in your tv, accept it.

5.- Turn off your TV and ruSamsungTVCommunicator then press {INFO} {MENU} {MUTE} {POWER} on the remote and re-open ruSamsungTVCommunicator.

6.- Press FACTORY and after this 3SPEED buttons on ruSamsungTVCommunicator.

7.- From Service Menu select Control, than Sub-option; Find PVR and change it to ON and PVR Num =1

8.- Save settings by pressing POWER on remote controller (TV turns off)

9.- Enjoy your new PVR!!!

I actually tried running ruSamsungTVCommunicator on a no-file-system-access XP VM running on OS X. I set a VM snapshot, then reverted post experimentation. There were quite a few unsettling pauses in the process, which might have been the XP antivirus keeling over. I didn’t get far with the experiment. I think I’ll have to give up on my hacking the Samsung — at least until there’s a less worrisome toolset.

iOS 7.0.3: Large data charges, newly enabled iCloud movie downloads, and missing cellular data controls

Things got quite weird as I and my app.net correspondents dug into this problem. See the updates below for the full story, which seems to have been a change in AT&T services that triggered a glitch in iOS cellular data controls just at the start of a billing cycle. A truly 21st century experience.

---

The messages started to appear around 3pm today ....

AT&T Free Msg: The following mobile ... has used 100% of the additional 200 MB of data ... We've added another 200 MB ... $15 will be charged to your account...

One after the other. They stopped at $136. Which AT&T, to their credit, forgave when I begged for mercy.

So what happened?

Well, to start with, our family is on legacy AT&T family plan contract that offered a low cost 200MB/month data plan option (as in this TidBITS article). It is a relatively cheap way to buy a subsidized phone -- $10/month for the voice, $15 for the iMessage-only data plan, low total cost of ownership. We did it 2 years ago before we switched #1 and #2 to H2O Wireless ($40/year, no data). That contract is up now and #3 is going to join her brothers in ultra-low-cost paygo land. Her 4S will need WiFi for data.

So it's easy to run through that plan. Except it shouldn't have happened so quickly. Take a look at the cellular settings for data in iOS 7 (see Update, wish I'd been more careful to include lower range in my screenshot):

Notice there's nothing there for Video or Movie? That's because iOS 7 isn't supposed to download movies from iCloud over a cellular connection. I tried on my 5s -- nope, not allowed.

But when I ran out to look at her phone I found she'd downloaded a 1.5 GB movie via cellular data. That shouldn't be possible, but there was a bug like this in iOS 6. (Note before iOS 7 this couldn't happen, a movie was either on the phone or unavailable.)

I think she started downloading the movie over WiFi, from the car in our driveway, at the very limit of our household WiFi. As she drove away her phone switched to LTE ... but the download continued.

It's just a theory -- but I'm betting that's what bit us...

See also

• Serious iOS 7 Flaw with iTunes Match and Cellular Data - similar bug, but we don't do Match.

Update 11/29/2013: After writing the post below I found I could start streaming an iCloud movie on WiFi, then walk away from home until the WiFi died, and iOS 7 will continue streaming it over LTE. Which it's not supposed to allow. That's how my daughter ran up her bill.

Update 11/30/2013: This is getting weirder. In an app.net discussion @sirshannon sent me a screenshot with Videos as a cellular data option. I looked at my phone and there it was. I figured I was just demented, but @wickedgood sent a screenshot without the option and Emily doesn't have it on her phone ... yet ...

 

It's possible that AT&T is making network changes that enabled both my daughters download and, belatedly, this option. Or perhaps they've changed carrier settings -- though I thought that required a user action.

In the modern world, sometimes we're crazy and sometimes it's just reality flux.

Update 11/30/2013 b: Sure enough, this morning my daughter's phone also has the Cellular data control for Video.app (controls grayed out here as her cellular data is currently disabled).

She didn't have it last night, which was easy to tell as she has so few data using apps. I've since turned all but Mail.app off.

Update 11/30/2013c: Later in the day I was able to inspect the 3rd iPhone on our family plan. Emily's phone still does not have the Videos.app cellular data setting. Studying her phone I realized Apple had put some of its Cellular data controls in one settings menu (Cellular) and others in the iTunes & App Store menu (and some in both). At that point I had to laugh. These screen shots are worth a look...

First - no Videos.app control. I tried a few tricks  that might "bring it on", including a power cycle, but nothing worked. I suspect it will appear tomorrow though.

Notice in her iTunes & App Store settings there's a special message. Her iPhone didn't, for some reason, see Cloud media, it's not fully associated with iTunes. I turned on automatic download for iBooks and, as you'll see below this message disappeared.

Take a look at these two screenshots from the iTunes and App Store config. There's an option to show or hide Videos that are in the cloud. That makes sense, but why the heck did Apple put a Cellular Data setting here?!

I tried downloading a movie from after disabling wifi (so this isn't yesterday's test, where I started the download on wifi then walked out of range) - 

As expected. I'll check again tomorrow and report when her phone gains the Videos.app cellular data control.

What a mess.

 

 

 

Configuring an EnGenius Technologies Long Range 11n 2.4GHz Wireless Bridge/Access Point (ENH202): Computers & Accessories

A buddy of mine decided he wanted a bridged connection to my home network. With a bit of input from me he settled on the EnGenius Technologies Long Range 11n 2.4GHz Wireless Bridge/Access Point (ENH202) ($90/each for two devices). It took a couple of tries, but we were able to configure a point-to-point bridge connection.

The reason this took a couple of tries is that the EnGenius comes with the worst documentation of any device I've ever seen. It's not the usual problem of limited documentation -- there's lots of documentation. Problem is, it's all incomplete and/or contradictory. Unsurprisingly, so is the firmware for these devices. This feels like its made in China 2013 for China 2013. It doesn't feel cheap though, it feels pretty solid -- made to survive outdoors. Go figure.

Oh! And the first set we ordered both shipped without the inline adapter for the Power Over Ethernet connection. We reordered and got the adapters. Yeah, a mess.

The documentation and hardware confusion has to be at least partly related to feature creep. This device is supposed to connect offices or remote billboards over 1 km point-to-point connection, but it does several different things:

• Access Point connected to Router/Net
• Client Bridge remote (IP Based)
• WDS Access Point
• WDS Bridge Mode MAC Based Network

We device ships with a 1 page setup page that gives some documentation clues, but most of the documentation is on a mini-CD that requires an old-fashioned spindle drive. We ended up configuring the devices as a WDS MAC based bridge (connection is based on MAC address, not IP address) relying on two documents:

• ENH202 User Manual Version 2.0. I don't believe this is available online, it came on that mini-CD. version 1 is available on the Engenius web site. 

• Learn How to Configure EnGenius Wi-Fi Products for Popular Applications (PDF version of a PPT) WDS part is age 25, they forgot to tell you to specify WDS mode.

I suspect WDS MAC mode is not the 'safest' way to configure these devices. Not only is it almost undocumented (despite all the documentation), but the link quality lights don't work in WDS mode and the UI is rough and inconsistent. For example, sometimes saving a change restarts the ENH202, other times you have to find the Save/Restart menu item and explicitly save changes, and for password changes, it's not quite clear when and how it saves a change.

If you decide to try MAC based WDS bridge (can include up to 4 devices connecting to each other) the sequence is somethign like this:

1. Starting from the default IP address, manually assign each device an IP address that will work with their network destination (when configured as Bridge both will we on the same network, for us we used 192.168.0.100 and 192.168.101 with subnet mask 255.255.255.0. (Usual procedures with changing IP of configuring computer, etc).
2. Name devices so you can tell which is which.
3. Learn MAC (ethernet) address of each device. Disregard what's printed on a sticker -- it was completely wrong. Use the web config UI.
4. In the WDS Link Setting Screen enter MAC address of remote device (on each device).
5. Optional: We set devices to 802.11n only, specify matching channel (band within n range), etc.
6. Test link.
7. Add encryption.
8. Add device password

Given that the link quality lights work in Client Bridge, but not in WDS Bridge mode, and given the poor quality of the WDS Bridge documentation, I suspect this device works better in Client Bridge mode (IP based connection)

Once we had it working the  WDS Link Status screen gave us a receiving signal strength of RSSI dBm -74. Wikipedia put that in context for units of dBm:

• 80: FM radio 50km range (output)
• 33: Maximum output from a UMTS/3G mobile phone (Power class 1)
• 30: Typical RF leakage from a microwave oven
• 31: Maximum output from a UMTS/3G mobile phone (Power class 4)
• 15: Wireless LAN transmission power in laptops.
• 0: Bluetooth standard 1m range (output)
• -10:  maximum received signal power (−10 to −30 dBm) of wireless network
• -80 dBM: typical range of wfi received signal power (range is -70 to -90)
• -127: Typical received signal power from a GPS satellite
• - 140; Received signal for LTE phone
• - 192: thermal noise floor 1Hz bandwidth space

So for a WiFi received signal we're not too badly off (our homes are close). We could have boosted power further but so far throughput is limited by my 8-10 mpbps DSL connection.