Video: The state of compression is not good. Blame it on Congress.

The more I study analog to digital video conversion and export compression the more I see I've left civilization.

This is a domain of arcane knowledge and of unknown unknowns -- "experts" who really don't know what they're talking about. I'm sure there are real experts, but they're bored silly by the level of net discussion -- including Apple's own technical notes. It reminds me of my 1981 medical school lectures in renal physiology [1], or my @2002 research into color profiles.

In the absence of publicly accessible expertise I hope I'll have time to do my own experiments, such as creating a range of QuickTime 10.x exports with a span of export parameters. I'm afraid, however, that I don't have that much time.

Which means I probably need to give up on my idea of using .mp4 files the way I use .JPEG -- as an archival storage format [2].

So that means I need to buy more storage -- 2TB should do it. From that point on, given the amount of video we capture, I will just need to upgrade every 2-3 years (4, 8, 16, 32....). And I need to invest in either firewire 800 or thunderbolt.

So now I'm looking at the 1995 era ".dv" format [3]. Should be straightforward. Right?

Yeah, that's what I thought until I read this wikipedia article, which, as is usual for wikipedia, reads like a mixture of a true expert and some confusing amateurs [5] (but is still better than the alternative...):

DV - Wikipedia, the free encyclopedia

... The DV codec was launched in 1995 with joint efforts of leading producers of video camcorders...

... DV video employs interlaced video scanning with the luminance sampling frequency of 13.5 MH...

... One video frame is formed from either 10 or 12 such sequences, depending on scanning rate, which results in a data rate of about 25 Mbit/s for video [4] and an additional 1.5 Mbit/s for audio. This results in a compression rate of 5 to 1...

... All DV variants except for DVCPRO Progressive are recorded to tape within interlaced video stream...

... When video is captured onto a computer it is stored in a container file, which can be either raw DV stream, AVI, WMV or Quicktime. Whichever container is used, the video itself is not re-encoded and represents a complete digital copy of what has been recorded onto tape. If needed, the video can be recorded back to tape to obtain full and lossless copy of the original footage....

So even .DV covers a multitude of sins -- and is a significant compression from the original stream.

Even so, a mixture of .DV and Apple's proprietary professional formats seems the only way to go for now. I hope within a decade we'll have the standards and software to do better.

Or perhaps not. It's worth asking why progress has been so slow over the past decade -- and why the "standards" for still and video images date from the early 1990s. The answer seems to be something Congress and the USPTO did in 1996 ...

USPTO issued Final Computer Related Examination Guidelines stating that "A practical application of a computer-related invention is statutory subject matter. This requirement can be discerned from the variously phrased prohibitions against the patenting of abstract ideas, laws of nature or natural phenomena".

We may need legal reform before we have an archival video format. This is not a technology problem.

[1] After weeks of confident and contradictory statements from researchers and textbooks, I wondered whether they knew they were faking it, or if they'd fooled themselves. I never did find out.
[2] JPEG itself is a non-specific marketing term that hides a very wide range of codec tweaks and metadata "standards". Still, it's a recognized library of congress standard (TIFF is even less standard.). There isn't an image format today that has a comparable archival lifespan, so I use 99% (minimally compressed) JPEG and delete the RAW images -- holding my nose the whole time. Yes, I wish JPEG-2000 had succeeded. Our IP (patent) framework is intensely dysfunctional. 
[4] Now I know what the data rate setting in iMovie 11's QuickTime export parameter means.
[5] In particular there's confusion about DV handling during capture vs. how it's represented during editing/playback and confusion about what containers are. Of course, as a non-expert, I corrected that section.