Interesting OS X observation from Firefox development

OS X has memory issues, and APIs that don't do anything ...

Firefox 3 Memory Usage pavlov.net:

... On Mac, If you look at Activity Monitor it will look like we’re using more memory than we actually are. Mac OS X has a similar, but different, problem to Windows XP. After extensive testing and confirmation from Apple employees we realized that there was no way for an allocator to give unused pages of memory back while keeping the address range reserved.. (You can unmap them and remap them, but that causes some race conditions and isn’t as performant.) There are APIs that claim to do it (both madvise() and msync()) but they don’t actually do anything. It does appear that pages mapped in that haven’t been written to won’t be accounted for in memory stats, but you’ve written to them they’re going to show as taking up space until you unmap them. Since allocators will reuse space, you generally won’t have that many pages mapped in that haven’t been written to. Our application can and will reuse the free pages, so you should see Firefox hit a peak number and generally not grow a lot higher than that....

My sense is that OS X does a lot to test the patience of application developers.

I've been using Firefox 3 beta 5 on OS X, and I feel that it's faster and much less likely to peg my CPU.

WiTopia personalVPN - need custom DNS configuration

I've been paying for the Witopia personalVPN service ...

Gordon's Tech: WiTopia personalVPN 

...PPTP VPN: This is built into OS X, though in 10.4 it works through the peculiar 'Internet Connect' application rather than the network preferences (where I looked for it). Easy to use, requires no additional software. This style of VPN is disdained by experts for some security issues, but of course it only has to be better than nothing -- which is what everyone else at the Hotspot is using. It's the old 'park next to the better bicycle' theory...

Problem is, it really slows down my browsing. Long delays for every transaction. I'll send them a note and see if they have any ideas, but I'm inclined to disable renewal of the service and try another vendor ...

Update 5/1/08: Both Witopia and a helpful commenter tell me that it's essential to use the Witopia DNS servers to get good performance. I think that's the problem, so I'll change and test. If it fixes the problem I'll amend my post title to "Use the DNS servers!". (The recommendation is down in the Witopia FAQ, I'll look over their site again but I think they could do with a bit of rework on setup documentation.)


Update 5/31/08: When I was running 10.4.11 I didn't see a way to specify a DNS address for the PPTP connection alone, and I didn't want to change my usual DNS address. With 10.5.3 it's fairly obvious how to change the DNS address for the PPTP connection alone (advanced button).

I added the DNS address from the Witopia FAQ: 38.119.98.220 Performance seems quicker, but, oddly enough, the Witopia Wiki won't open. Other sites seem fine, so perhaps the wiki is coincidentally offline. I'll test this out for a while.

Why was my OS X mouse behaving so badly?

It had been bugging me for a while, but now I was going over the edge.

OS X 10.4.11 wasn't responding to my mouse clicks. I was double, triple and quadruple clicking where one should have done.

What had gone wrong? Some recent update? Something stealing the mouse clicks?

I keep my system very plain, so I decided to check into a new mouse driver. It had been a while since I'd updated my Microsoft IntelliPoint driver -- years, come to think of it.

I installed the new IntelliPoint and IntelliType Pro driver from Apple's site. The reboot took a long time, the mouse install seemed to require some major rebuilding.

The new driver had loads of new options, but my mouse clicks were still being missed. Now, however, the driver told me my battery was low.

New batteries, but the problem remained. The driver had one more diagnostic though ... my signal was oddly weak.

Yes! The small wireless receiver (this is pre-Bluetooth) had fallen off my desk, and was now buried between the back wall and the back of the desk. Restoring the receiver restored the signal -- and my mouse clicks.

Of course the real problem was that I'd lost my receiver, but it was odd that I didn't get the battery and signal warnings until I installed the new driver. My IntelliMouse driver was ancient, probably inherited from a 10.3 build. I wonder if some recent update didn't knock out the battery warnings it used to give; Apple can't possibly test against ancient drivers.

Really, these computer systems are a wee bit complex.

I do like the new mouse options however, especially Microsoft's acceleration support. I feel as though I got a new mouse out of the deal.

Google Docs: still no working clipboard

I've been using Google Docs for ages, and there's been no progress in providing any kind of reasonable clipboard functionality with any OS X browser. We're told to use ctrl-c to work with the system clipboard instead of using the native Google Docs clipboard. Of course that shortcut works for Firefox on OS X, but not Safari.

Honestly, Google Apps is still a semi-useful toy compared to a desktop application. Progress is very slow.

Firefox noscript add-on - time to start using it

A recent large scale hack of Microsoft's IIS web server means that lots of reputable web sites may be hosting exploits more commonly seen on the shady side of the net.

So it's probably time to start using the Firefox NoScript add-on ...

Hundreds of Thousands of Microsoft Web Servers Hacked - Security Fix

There is a great add-on for Firefox called "noscript," which blocks these kinds of Javascript exploits from running automatically if a user happens to visit a hacked site. Currently, there is no such protection for IE users, and disallowing Javascript entirely isn't really an option on today's World Wide Web. True, you can fiddle with multiple settings in IE to add certain sites to your "Trusted Zone," but that option has never struck me as very practical or scalable.

I've been using it for a few days. I whitelisted a number of the Google sites I use (if they're hacked we're all doomed) and so far it's been easy to enable JavaScript when needed by clicking on the S icon.

Video editing - lord, this is ugly

I'm pretty disgusted with iMovie '08, so I decided to read a review of Final Cut Express 4.

Yuck.

MacWorld desperately tries to say nice things about Apple, but despite the "4 mouse" rating the review reveals a right mess. Different behaviors on different processors, inability to import some formats iMovie '08 handles, etc, etc.

Video editing was always pretty complex, but I think the profusion of codecs and file formats has driven it over the cliff.

Apple's offerings are pretty sad at the moment (iMovie HD was the closest they got to a good solution, and they abandoned it). Unfortunately I think Apple is the only consumer video option for the Mac.

Digital-TV converters: choices for the Saint Paul area

A local paper's tech blog reviews Digital-TV converters available locally. His preference is the "Digital Stream DTX9900 sold by RadioShack". I need to spend my vouchers before they expire, so I'll probably follow his lead here.

From what I've read elsewhere broadcast digital may require a fairly expensive antenna investment. So the converters are only a part of the cost. We have very cheap antennae; I'll report on what we find in our real world testing. All we need is for one sports channel to come in so my son can spend his tv-time credits.

Update 5/4/08: Bloody 'ell, it worked.

QuickTime Player: saving the configuration files

QuickTime Player Pro AppleScript driven batch translation from FLIP Video 3ivx to Apple Intermediate codec requires a QuickTime configuration (.set) file.

You can't save a configuration file from QuickTime Player Pro. (How hard would that have been?)

You need to open video file, set up the export settings, then run an AppleScript like this one:

tell application "QuickTime Player"
tell first document
save export settings for QuickTime movie to file "BootDrive:Users:jfaughnan:Documents:aic2.set"
end tell
end tell

You'll need to edit the "BootDrive ..." stuff to match your hard drive (Volume) name and user name.

What a nuisance!

Apple stripped AppleScript functionality from QuickTime Player - in a fairly rude way

Once upon a time you could use AppleScript to make QuickTime Player do many of the tricks that QuickTime Player Pro is supposed to do. In those days people wrote scripts to do things like batch file translation:

Gordon's Tech: FLIP Video Ultra camcorder: iMovie HD works, iMovie '08 doesn't

Update 3/30/2008: I found an AppleScript that should do what I need. It tells QuickTime Player to open multiple AVI files, process them to DV stream, then delete the originals. Unfortunately, it's giving me cryptic error messages.

Update 3/31/2008: I'm beginning to think Apple quietly disabled AppleScript driven conversion in QT Player. Nobody seems to know anything about it.

As you can probably guess my theory of 3/31 seems to be correct.

When I ran those scripts with QuickTime Player, they simply didn't work. No error messages, but the AppleScript APIs didn't seem to do anything.

I paid for QuickTime Player Pro and now the original scripts work.

This seems kind of obnoxious. At the very least QuickTime Player should have produced an error message like "AppleScript functionality requires QuickTime Player Pro". Apple wasted a fair bit of my time.

Video codecs: iPhoto, iMovie HD, iMovie '08 and FLIP Video

As a part of my ongoing struggle with FLIP Video, I'm delving ever deeper into the broken world of Apple video.

iMovie '08, for example, recognizes some video formats when it's importing directly, but a smaller set when the video is stored in iPhoto. I even have an sneaking suspicion that my test results differ between my Intel MacBook and my PowerPC iMac.

The FLIP camcorder uses the 3ivx toolkit form MPEG-4 compression and playback, but it's a completely proprietary implementation. It can only be read with a 3ivx decode, so it's not what I want to keep video in.

So what format makes sense? I'd like something that

• is fairly standard
• is fast to edit
• doesn't use tons of disk space
• doesn't lose immense quality when it goes through edit cycles
• is recognized by iMovie '08 even when the video is stored in iPhoto
  

One option is Apple's Intermediate Codec (used by Final Cut, emphases mine):

Final Cut Pro 5: About HDV and the Apple Intermediate Codec:

... The Apple Intermediate Codec is a high-quality video codec that Apple developed for use as an alternative to native MPEG-2 HDV editing in an HDV workflow. Instead of editing the MPEG-2 HDV data directly, you can capture video from the tape source and then transcode it with the Apple Intermediate Codec to optimize the video data for playback performance and quality.

Working with the Apple Intermediate Codec is less processor-intensive than working with native HDV. Unlike MPEG-2 HDV, the Apple Intermediate Codec does not use temporal compression, so every frame can be decoded and displayed immediately without first decoding other frames. The drawback of this codec is that it requires three to four times as much bandwidth and hard drive storage space as MPEG-2 HDV.

Data rates for the Apple Intermediate Codec are variable; the data rates and storage needed may vary slightly, depending on the complexity of your footage. Images with a lot of detail have a higher data rate, while images with less detail have a lower data rate.

I did some experiments converting FLIP Video to MPEG-2 and Apple Intermediate Codec (AIC). On my MacBook iMovie '08 recognized the AIC encoded .mov files within iPhoto (you need to restart iMovie to get it to recognize new iPhoto additions):

• Original (3ivx): 3.6MB
• Apple Intermediate Codec with AAC encoding for audio: 14MB
• MPEG-2: highest quality, AAC audio - 15MB
  MPEG-2 "high quality" - 7 MB

So in my testing the MPEG highest quality was about the same size as the AIC file, but the MPEG-2 "high quality" was half the size. Both were fairly quick on a MacBook.

On my iMac, using QuickTime Pro, I wasn't able to find MPEG-2 as an option for export. I did find Apple Intermediate Codec and even on the old PowerPC machine the 3ivx to AIC conversion was pretty quick. The resulting file was "seen" by iMovie '08 even when it was stored in an iPhoto library.

So I guess for the moment I'm going to be transcoding to Apple Intermediate Codec. Problem is that QuickTime Pro doesn't do batch file export. More on that later ...

Update 12/25/08: This all largely obsolete when Apple finally updated iMovie HD so it will work with FLIP camcorders after you install the 3ivx files (at least on Intel machines).

In any case, I did come across more explanations of 3ivx and other video codecs in a well done Gizmodo review:

Okay, so all that stuff up there are industry-wide standard video codecs. On top of all of those, various entities love putting out their own spin on those standards. As we mentioned before, DivX (proprietary) and XviD (open source), for instance, use MPEG-4 Part 2 (more specifically, MPEG-4 ASP) compression, meaning stuff that'll natively play back MPEG-4 ASP will also play back DivX. Like the Xbox 360, for instance. There are a ton of MPEG-4 ASP-based codecs, actually, like FFmpeg, 3ivx and others, but DivX and XviD are the most common.

So 3ivx is a sibling of DivX. I wrote a later update on this topic.

Camino not working with Gmail

Camino has stopped working with Gmail. I assume the bug is well understood -- it's not obscure! The screen redraws constantly, never progressing beyond the first 20 pixels or so.

I tried upgrading to the new 1.6, but it has the same problem. I'm back to Firefox beta 3 for now. (Safari has never worked well with Google products, and it still doesn't.)

I'm surprised there's no fix yet.

Update 4/28/08: I followed the advice in comments; emptying the cache fixed the problem. It's funny, I used to empty the cache the first time I had browser problems, but it's been years since that did anything. I just lost the habit ...

Devon Technologies free OS X services - including WordService

I'm very surprised I haven't been using this free service app. DT lists several others worth looking at. All free. Thanks CT comments!

Needful Things: Services

WordService 2.7

This service provides 34 functions to convert, format, or speak the currently selected text, as well as insert data or show statistics of the selection within all Cocoa applications (such as TextEdit, Mail, iChat, Safari, XCode, or our commercial applications) and Carbon applications supporting services.

Features:

Reformat, Remove line attachments/endings/links/multiple spaces/multiple feeds/quotes, Trim line beginnings/line endings/lines, Sort lines ascending/descending, Shift left/right, Initial caps of words/sentences, All caps and lowercase, Mac/Windows/Unix line endings, Rotate 13, Straight/Smart Quotes, Encode/Decode tabs, Insert date/date and time/time/contents of path, Speak native/German text, Statistics.

Has Google heard of the Macintosh?

From a Google blog post on malware:

Official Google Blog: Working together to fight malware

... Use anti-virus software. Most anti-virus software is specifically designed to find and remove harmful software on your computer. Be sure you have anti-virus software installed on your computer (you can get a free trial through Google Pack if you don't), keep it current, and use it to run frequent full-system checks...

I don't know any OS X user running antivirus software; in any case it would cause far more problems than viruses have to date.

Clearly, someone needs to bring a Macintosh to Google's office. That's the problem with these big, slow, corporations, they're stuck in the 20th century.

Then Google could write something like this ...

Avoid Windows XP. We recommend Vista with anti-virus software if you want to invest in a brand new quad core 4GB 64 bit system with all new hardware, Desktop Linux, or any Macintosh.

OS X support notes - recent updates

Several interesting Apple Support articles have come out recently:

• Mac OS X 10.4: Keychain Access asks for keychain "login" after changing login password. The keychain password and the user account password are stored separately. It's possible to change one and not the other. This tells how to manually change the keychain password to match the login password.
• Browsing for PCs on a network requires that the PCs have NetBIOS enabled. (Wow, that's really old and creaky stuff.)
• How to recover when a server won't let you connect with a different account
• Single user and verbose startup (verbose is new to me)

Someone else has noticed a one year lifespan for a modern XP hard drive

I have two corporate XP machines, and between then I have to replace a drive every six months. My home XP and OS X drives last for years.

So I now have two completely unrelated corporate backup systems that run nightly.

I haven't seen anyone else comment on the lifespans of corporate hard drives, so I liked this post (emphasis mine)...

DadHacker » Blog Archive » Thoughts while rebooting

On the IT-ridden machines I regularly have to swab out twenty megabyte log files, logs from things that I didn’t even know were running on the machine, and when I find something like “ArScnr38″ running I have no idea if it’s spyware or something that an IT monkey stuck on my laptop to scan my Excel spreadsheets.

It’s hilarious when four different scanners are fighting for disk access. No wonder our drives are dying after like a year in service. I don’t work late, so I can only imagine what the buildings sound like at 3AM when Windows Update goes into its happy dance and reboots every single workstation.

“Shhhh… wait for it.”

“What, Dad? I’m sleepy.”

“Any second now…”

clikclickClickClikCLIKCLICKCCLLIICCKK-CLICK-***KA-CHUNGGGKGKGKG!!!!!***

“Wow! Do the lights flicker like that in every time zone?”

“That was nothing. Wait until they all ask the DHCP servers for an address!”

...The IT philosophy of bloat appears to be: “Screw the user, we own the machines, and if they can’t get work done with them then they can’t do any damage. More scanners! And loggers! And Java-based enterprisey things with *****up XML configuration schemas! If there’s CPU or disk space left we’re not doing our jobs; we have to pay for that call center expansion somehow!”

That's what I see. Between my heavy duty database work, the antivirus scanner, the corporate HP monitoring systems, Windows Search indexing, and the two nightly backups the hard drivers are being worked to death.

I really need to switch to an in-office NAS with a hot-swappable RAID array -- so I can rotate out bad drives without the hassle of a restore.

Aperture's SQLite Database

I'm going to fire-up the SQLite Database browser and follow these directions: The Deep Dark Depths of the Database (and some fingerprinting) - Inside Aperture. I wonder what iPhoto is doing these days -- be interesting to look for SQLite files in other Apple applications.

I recall that SQLite was thought to have threading issues with 10.4. I wonder if that's better with 10.5.

A last lesson from our Samsung i500

I like my iPod. I liked the Palm Vx. I even mostly like my MacBook.

So I ain't impossible to please. Just not easy.

So it was hard so say the long good-bye to our Samsung i500s and all their accesories!

They were great tech for their day and today. Palm Classic goodness -- fast, super reliable, elegant. Graffiti One. Compact clamshells with good sound quality and an excellent form factor. Line 'em up against the iPhone and they do pretty well. Sure -- no mobile web. On the other hand, way more reliable and they have tasks too!

Alas, they gradually died. When my wife's failed we switched to AT&T for her BlackBerry and to line-up for my iPhone.

There was one major flaw of the i500 though -- and one related lesson. All the connectors were proprietary - and transient. No USB goodness (BlackBerry, Nokia 6555), and no evil-but-ubiquitous connector like the iPod -- just another one off from Samsung.

So, when a "free for anyone" listing on Craigslist got no hits, I had to toss everything out. Connectors, chargers, accessory cables, -- the whole bit.

That's the last lesson, though most of us have figured it out by now. Don't buy devices that use proprietary connectors -- unless they've made it to evil-but-universal status.

It's a great way to narrow your purchasing options ...

Introduction to the relational database

ATPM 14.04 - FileMaking: Getting Relational turns out to be a fairly comprehensive and quite readable introduction to relational database design. I didn't see anything that's FileMaker specific, it applies equally well to MySQL, Microsoft Access, Oracle, etc. (The promised f/u articles will be FileMaker specific.)

It's a handy reference to keep around for colleagues, students, etc.

Charles Ross covers a range of topics typically taught over the course of several lectures, so don't be surprised if it takes a while to get through the entire article. If you need to understand the topic, this is worth printing out and studying.

More iPhoto humiliation: the vanishing edited video

iMovie '08 is so abominable that it makes iPhoto look good ... by comparison.

Still, it's not hard to humiliate iPhoto, even if we forget that after 8 releases it still can't import a Library.

We know that iPhoto has a nasty video export bug that can result in total data loss. That's bad, but this is at least comparable:

1. Import 3ivx MPEG 4 encoded video with AVI wrapper into iPhoto.
2. Right click on the Edit button and open in QuickTime Pro.
3. Trim the video and save. QT Pro offers to save in .MOV format. Choose that.
4. Return to iPhoto. The video you see is the previous version.

Basically the "Edit" option in iPhoto doesn't work for Video. It should be "grayed out" and unavailable for selection.

I bet QA found this one in early testing of iPhoto '08, but product management decided it wasn't worth fixing.

They should be ashamed.

Firefox 3 beta: they didn't tackle the tab UI problem

I've been using Firefox 3 beta 5 with OS X. I've noticed a few problems -- windows initially appeared almost off screen, one window would abruptly close, but overall it looks like a good incremental improvement. It hasn't pegged my CPU yet, so that's encouraging. (Firefox 2 would do that without any help from Flash.)

Alas, they didn't do anything to tackle the multi-window tab UI problem. I was looking for a view that would render windows and tabs in a browseable outline view with tab names and mouse-over window views, but there's no sign of this.

Lesson from FLIP Video: your video is doomed

Petabyte-equivalents of home video have vanished over the past 40 years. Odd sized films, film degraded and glued together, beta tapes, odd VHS variants, digital tapes, digital disks, abandoned digital file formats ... there are many routes to the graveyard.

Things are not about get get better. I don't have any data, but I'm blogging -- I don't need any friggin' data.I'm guessing most video now is coming from cell phones, digital cameras, and new packagings like FLIP Video. So how well will this video do over the next 40 years.

Digital cameras commonly use "Motion JPEG" [1]. Mobile phones use all kinds of formats, but they're converging on a wrapper called "3GP", behind that wrapper are all kinds of semi-standard data formats. The FLIP Video camera the kids and I are using on our Wisconsin Dells holiday uses 3ivx, of which I recently wrote:

Gordon's Tech: FLIP Video Ultra camcorder: iMovie HD works, iMovie '08 doesn't

...Videos are in "AVI" format -- that's a metadata wrapper around a codec. In this case coded is 3ivx MPEG-4... 3ivx Technologies is hoping you'll buy the full version from them...

I ended up writing an extended post with lots of updates; I learned quite a bit about iMovie '08 (disgraceful, Apple shipped at least one year too soon [3]), QuickTime and QuickTime Pro, MPEG Streamclip, video formats, video codecs, editing software, etc.

I haven't quite figured out the optimal strategy for editing and storing FLIP Video 3ivx encoded video, but I think there's a very good chance the 3ivx files will be completely unreadable within 15 years. It's a completely proprietary format, with no particular reason to expect it to become a lasting "standard".

So what will last? Well, I'm betting reasonably compliant still image JPEG will be readable a hundred years from now [2], so I think Motion-JPEG video might persist. Motion-JPEG's simplicity makes it easy to edit too, assuming one doesn't try to convert from the highly JPEG compressed images to any lossy format. I'm not so sure about the sound formats though. DVD-Video without copy-protection might also be expected to last, but that simple name hides a lot of complexity and variation with sound and video compression standards and metadata wrappers.

So called DV-stream is one name given the streamed version of 'digital video' standard (actually is a standard), but a casual glance at Wikipedia reveals lots of room for incompatible variations on digital media. It does lend itself well to editing (no intra-frame compression) but it makes for huge data files. It's dying off as MPEG-4 and HDV "standards" take over.

Hmm. Sounds like a real mess. I've read lots of discussions about archival image formats, so I'm sure this Google archival video search will yield lots of great advice.

Well, as of April 2008, not so much great advice. Basically, no advice at all.

Those petabytes of lost home 8mm home film recording are about to joined by peta-giga-tera bytes of every conceivable unreadable combination of video, audio and metadata formats.

Don't get to attached to those precious video moments ....

-- Footnotes --

[1] I'm still figuring out what this corresponds to in QuickTime Pro's export menu. I think it's the JPEG export option that shows up in the video export list, but I've reason to suspect things are even more confusing than they appear.

[2] Assuming our non-human inheritors are curious about their precursors. JPEG is what I store my photos in, I assume the original RAW files will be absolutely unreadable within 10 years.

[3] Seriously. The more I play with it the worse it gets. For example: even the one or two video formats it can import aren't recognized when the same files are stored in an iPhoto repository. If anyone ever has the delusion that Apple has some special magical interest in their customers, they need to review the iMovie '08 story. I use a lot of Apple products, but they're not marvelous. They're only better than the alternatives.

HP 15C emulator for the iPhone

Found this via Cosmic Variance: hpcalc-iphone - Google Code. It's a GPL project, includes the beloved HP-15C. I'll want this one, it will be out with the official SDK.