Digitizing home video (once again)

I have made (2000) several (2004) tries (2012) at digitizing our family video but my lost best effort only got about 10% done. I figured I’d just hire a local service and then I let it slide.

Apple’s recent codec retirement announcements prompted me to check what was available locally. I found ancient web sites that were internally inconsistent, no noise reduction prior to compression, unclear codec choices … none of it gave me much confidence. (But see [1,2])

So I’m back at it again. This time I might have an accomplice — someone who needs money and would benefit from learning the tech. So maybe we’ll make a better go of it.

The delays may have let to some data loss, but on the other hand the tech is a bit better. My first attempt would have been with a 400MHz Celeron. Yes, that’s an ‘M’. In those days hard drives were measured in tens of GBs. Now the cheapest hard drive I can buy would hold all of our video.

The tech is a bit better, but choice of codec is still an issue. In 2004 I favored H.264/AAC. I ran into an amazing number of headaches with the Apple software I was using.

For the modern era I found three good references:

• DIY VHS Preservation: Planning for Video Digitization at the American University Library
• Audio Visual Digitization Hardware and Software - Marriott Library - The University of Utah
• Building a Video Preservation Rack for In-House Digitization AV CLUB | Issue 1 | Cardinal Tales

They give me a feeling of how tricky it is to do analog video capture well. Time Base Corrector?  BNC terminations? Waveform monitors? CRT monitors?! Yikes.

I did like the sounds of the BlackMagic Intensity digitizer ($240 for T3, $200 for USB 3) used at AUL (Amazon reviews are not great however). It can save output as a lossless file. I want to capture the video as “uncompressed 10-bit 4:2:2” then denoise it, then export as ProRes. Since my accomplice is a student I’ll probably buy the Pro Apps Bundle.

Ideally the process would be automated - capture uncompressed, denoise and incorporate metadata, save as ProRes.

What would I do with this material once it’s digitized? The tech isn’t here yet, but eventually I’d like to incorporate brief (silent) video fragments into my screensaver library. So between showing 10,000 images, show a 30sec of video from our family @ 1995. One day?

- fn- 

[1] From my 2000 page I see Walmart and Target were do video to DVD-R conversions for $35 a tape with YesVideo. They are still around! The price is now $26 for one 2hr tape.  

[2] Pogue years ago recommended Southtree (his screenshot of a VHS tape on a modern laptop screen is remarkable — 333x480 pixels). Their site is impressive; at the moment they’re advertising $57 for up to 3 tapes on 1 thumb drive. I contacted Southtree to ask about denoise/ProRes/etc but they kindly responded that they are consumer-only, so just mp4 compressed.

1Password Logins Notes field blank? Maybe it's the CR.

When I moved to Mojave I had to give up on my ancient FileMaker Pro database. The modern versions of FileMaker are far too expensive; consumer databases have passed into history.

For lack of a better alternative I moved over to 1Password. I liked them when they worked with a local password store, I’m not keen on their current cloud solution. I just don’t trust their tech.

I exported as CSV as I’d done many times before. This time, though, I ran into a problem that was probably always there. The first time I went to look up my ‘secret question answers’ they were nowhere to be seen. The Login Notes field where I’d imported them was empty on both iOS and mac OS.

Later I realized the data was there — but only in edit mode. If the default read-only view the notes field showed as empty.

It took a bit of playing around to realize what’s wrong.

Text fields in my version of FileMaker, probably from 2014 or earlier, uses the old classic Mac OS line separator - the single “CR” code. It doesn’t use the OS X/macOS/Unix standard LF or the old DOS CR/LF. 

When I exported as CSV the output used CR as a separator. When the CSV was imported into 1Password the CR separators went along. They work fine in edit mode but not in view mode.

The proof of the problem was to edit in 1Password, deleting the line feeds and adding new ones. After saving the note displayed correctly.

Update: Per 1Password tech support — turning off markdown formatting causes the notes to appear!

Converting my old video formats to something that might persist

Apple is killing QuickTime 7 (download 22MB Snow Leopard version here)[1]. The announcement has some useful references.

So I’m back to thinking about one of the worst topics in the geek world — video file formats and codecs. Almost all of which are encumbered by walls of vicious patents (AV1 is the great hope).

It’s been about 4 years since I last made stab at this topic. I reinstalled QT Pro then too!

Conversion from old formats is a PITA. In 2015 I converted an old WMV file (I think I used Flip4Mac [2]) to (I think) lossless uncompressed AVI. The original was 23MB, the AVI output was 311MB. Today I used QuickTime Player 10.5 to open the AVI and it created a 52MB (lossy) .MOV file (Info says it’s H.264 for video, AAC for audio). There’s no control on the conversion compression.

To get this done I’m going to have locate my video files (some in iMovie projects, some in the file system, some in Aperture) and apply some kind of batch conversion where needed. Spotlight’s ability to search for codecs might help, I suspect some useful utilities will show up now that this is getting some attention.

What should I convert to? Ideally I’d choose something designed for video editing. The Smithsonian recommends "Motion JPEG 2000, MOV, AVI”; I don’t understand how MOV gets in there, I thought it was just a container. Also MOTION JPEG 2000 is on Apple’s kill list.

Ok, so things are bad. But we knew that.

Apple’s ProRes is one example of what’s known as an “intermediate codec”. Apple seems to be dedicated to it and I gather it’s widely used in the video industry. So let’s see what other choices there are ...

Fifty. That link lists 50 intermediate codecs. Ugh. Maybe ProRes 442 HQ isn’t the worst option.

I’ll probably have to play around and study some more. Some things I’ll look at ...

1. Wondershare is $40 for 1 year. There’s a free demo that converts the first 1/3 of the video. It converted an old .wmv file to an HEVC (H.265, AAC) without blinking an eye. Judging by the 1/3 converted the 23MB WMV file would convert to an HEVC file of similar size.
2. Apple Compressor: $50, or have my college son buy the whole ProApps bundle (Compressor, FinalCut, etc) for eduction for $200.
3. QuickTime Player’s built-in conversion.

See also:

• “Swiss Army knife for QuickTime” Perian turning off the lights 2012
• Why I hate video: Format, codecs, DRM and m4v vs mp4 2011. Still hate ‘em.
• MPEG Streamclip video converter as Compressor alternative 2012. This was last update for Mountain Lion!
• AVI video files in 2014 on the Mac -- you should probably convert them while you can.
• FLIP Video Ultra camcorder: iMovie HD works, iMovie '08 might 2008. FLIP used 3ivx MPEG-4, iMovie ’08 couldn’t use it, but iMovie HD could if the 3ivx coded was installed.
• Video codecs: iPhoto, iMovie HD, iMovie '08 and FLIP Video (2008 again!)
•  . Fun stuff.

- fn -

[1] Yes, that’s MB. Wow. It does indeed run on Mojave! Not only that, but I dimly remembered that I had a registration code for QT 7 Pro. Being a total nerd I still have it on hand. I entered the code and clicked “register”, but the registration server is long gone. It still registered though, I quit and restarted and I have the Pro features back.

[2] Replaced by Switch. For $10 you can convert wmv to mp4, for $200 you can to more output formats. If you search wmv to mov conversion you get a lot of junk, so this is worth paying attention to. I might buy it.

Replacing Sierra's "All My Files" in Mojave

OS X  / macOS had, for years, a Finder Favorite that listed all files by date modified. It was removed in Sierra for unknown reasons.

The closest replacement I’ve found is to save a “This Mac” spotlight search on ‘.’ That is, create a Spotlight search for your Mac and enter a period as search criteria.

This works on Windows 7 too (I don’t have Win 10).

Recovering from a photo crop error when you need to print 4x6

I needed to print our team hockey photo at Walgreens, but somehow I’d saved a cropped version of the team photo instead of the original — and these days Walgreens doesn’t let one print an image without cropping [1]. Part of the image was cut.

My first thought was to custom print from Aperture to PDF then export a JPEG from that (it’s complicated) — but something funny happens with page orientation in Mojave/PDF.

The real answer was to use my old copy of Acorn (still works in Mojave). I resized the canvas to fit the horizontal image and fill in the right 3:2 horizontal:vertical orientation. Then I threw in a title for photo. Looked like it was deliberate.

[1] Once upon a time I think that was an option — just printed with white borders.

Mojave has brought back one of my least favorite macOS behaviors - screen saver bug.

When I used Sierra my screen saver was based on Aperture albums stored in an external drive. When I traveled the screen saver switched the default, but when I reconnected it went back to Aperture.

Mojave can’t use Aperture/iPhoto albums. So Screen Saver points to a share drive. When I travel it switches to default, but when I return it *doesn’t* switch back to my share.

Anyone have a workaround? I wonder about creating some kind of local alias for the remote image folder ...

Update: I created a symlink for the share folder that has my slideshow images using the SymbolicLinker service. Then I pointed ScreenSaver slide show to the symlink. When disconnected ScreenSaver shows black screen. When connected shows my images.

Migrating from Blogger to WordPress ... again ...

I’ve been contemplating migration to WordPress for almost a decade, but Google kept Blogger good enough to keep that headache at bay.

Alas, the days of good enough are ending. Google is removing their photo management API without recourse. They do support posts with images, but only by using their web interface. It’s a concrete and undeniable sign that Blogger is either dead or going to a bad place.

I though I’d migrate first to wordpress.com then to my Dreamhost open source wp install, but via Twitter Daniel Jalkut tells me he got better results using the open source importer directly.

I’ll do a dry run on one of my big blogs first. The URL won’t change but I’m sure feed subscriptions will have to be redone (ugh).

Update 2019/04/06 - results of the pipdig import process

I tested the Dreamhost free version of the pipdig importer from a Dreamhost wordpress (open source) blog. The results can be seen here for the moment, I’ll eventually delete them. I found:

• It doesn't remap internal links. This is a big disappointment. Links continue to direct to blogger, once that account is gone they will be invalid
• There’s no option to migrate images that I can see.
• It missed at least 4 posts from the source blog — specifically from early on. No idea why and it suggests more are missing.
• It does copy drafts over.
• It requires a LOT of access to your Google account! If you use this utility I suggest creating a new google account, give it access to your blog, then after the import destroy it. 
• The paragraph breaks are missing - line feeds vs <p>. This is an ancient Blogger problem with MarsEdit; a legacy of the original sin of English language text formatting end-of-line standards. I think Blogger is mostly to blame.
• Images were not relocated locally, they remain at their original locations.

Pipdig is better than nothing, but I’m going to try wordpress.com’s import tool next. I wonder if a better solution wouldn’t be a static site that I could archive on my personal web server, then do a web server redirect to handle the links. For now I’m still on Blogger. The porting experience reminds me of the impossibility of leaving Apple’s defunct Aperture photo management app.

(As I write this the wordpress import is processing - result should eventually show up at gordontest.tech.blog temporarily, but we’ll see if it works. It’s taking a long time.)

Update 2019/04/06b

Well, that wordpress migration didn’t go so well:

Your site has been suspended from WordPress.com for violating the Terms of Service. If you believe this was done in error, please contact us as soon as possible to have the suspension reviewed….

I sent a contact inquiry, nothing yet.

Synology NAS and security risks of enabling notifications

I use a Synology NAS to backup our two MacBooks. I’ve been ignoring it for 4 years, but a recent hardware failure made me look into it.

I found a number of packages installed and/or running that I’d not updated and mostly didn’t need. So I removed all those and I created a reminder to check the NAS quarterly. I also realized I hadn’t gotten monthly status reports for a long time — for years really (if ever)

To enable Synology email status reports you have to configure Notifications. Old-school SMTP is rarely available now, so I experimented with the Gmail option. I got this:

Oookaaay … that’s an interesting range of permissions. Synology is a Chinese corporation, so this effectively gives Xi the ability to harvest my email. Instead I created a synology user on one of my domain based Google Suites and enabled access there then forwarded to my email.

Interestingly my old settings suggested I had gone down the Gmail road at one point. I wonder what I was thinking, in my 2015 post I commented “Synology is a very Chinese product — including off-key English syntax. I wouldn’t install it in a US government facility.” Maybe I started the setup and then stopped?

MacBook Air shutdowns - it was the battery

My 2015 Air shut down suddenly two weeks ago. The battery was at about 80%. When I got it home and plugged it in it showed classic bad SMC behavior — the power diode didn’t light. 

I did an SMC reset and it worked, but a week later it did the same thing. I did an SMC reset again, but without checking if it was necessary.

It happened yet again. This time it worked fine as a soon as I plugged it in. That gave me hope that it was a battery issue, even though system info showed only 80 or so cycles. It’s an old battery.

After doing the usual 3 backups-to-current-state-prior-to-repair (one update to my Carbon Copy non-bootable clone backup, one fresh full bootable clone, and one Time Machine backup) I brought it in. It failed the diagnostic test with a big red dead battery note.

So $140 when the part comes in, which is a nice relief. If it had been the motherboard that would be $340 and I’d have a machine with a 4yo battery and a 4yo SSD. Might be better to just buy new.

Simplenote web (beta) version has much improved notes export

The beta version of Simplenote’s web client will export notes as text files and will use the note title as the name of the text file. When I’d previously exported from my Simplenote library the text files names were all GUIDs. 

This is obviously much better. It means, as long as you don’t mine plaintext, you can get your data out of Simplenote in a portable and useable form.

To use the beta version of the Simplenote web client use the URL simplenote.com/new.

I’ve been using nvALT to enable data freedom for Simplenote [1], but it’s good to have a second option. (nvALT works on Mojave, but had issues with High Sierra. I’m planning to skip from Sierra to Mojave.)

PS. Recently Simplenote added plaintext note import, though needs Electron version for Mac.

- fn-

[1] It took years longer than it should have, but Automattic fixed search in the Mac client.

See also

• Progress of sorts - integrating my simplenote memory extensions with gDrive 11/2017
• Escape from Outlook Notes - export as text files, import into NvAlt or other 3/2017
• Tinderbox, Simplenote, MindNode and data freedom10/2011. "All software dies. Ask Lotus Agenda users. Or the users of, and this is far from a complete list, Ecco Professional, InfoCentral, PackRat, Sidekick, Arrange (Common Knowledge - MacOS), InfoDepot, In Control, Cross-Ties, Palm Notes, Commence, Ascend, Arcadia (OS/2!), GrandView and MORE, Inspiration, Outlook Notes, and so many, many, many more.”

Extend Aperture's lifespan with Adobe DNG Converter for Mac

Aperture on Sierra doesn’t support RAW files from my Canon EOS SL2. I don’t know if there would be any support if I upgraded to High Sierra or Mojave, but I think not. I tried using Canon’s RAW to JPG converter but it was achingly slow and it defeats the purpose of shooting RAW in the first place.

So I’ve been shooting JPEG [1]. Today, through Facebook’s Aperture User Group, I learned that Adobe DNG Converter output can be treated by Aperture in Sierra as a type of RAW format. So I downloaded the app and tried it on a CR2 file from my SL2. It was extremely fast and produced a DNG a few MBs smaller than the CR2 file. Aperture opened it a bit more slowly than I remember it processing my older Canon RAW files, but there no real issues.

Adobe DNG Converter has a truly ugly Mac UI, but I have no problems with that. I suspect DNG is only a minimal archival improvement on CR2 so I’ll mostly continue to shoot JPEG (because everything sucks [1]), but now I have the option to do CR2 when I want better results. Since I already use Image Capture to bring images off my devices rather than Aperture the extra conversion step is a modest cost.

- fn -

[1] Twenty ago I was sure we’d get one of many better lossy image formats, of which JPEG2000 is the only one I can remember now. We never did, partly due to patents and partly for reasons I don’t understand. I think cameras have gotten better at making the best use of JPEG, which itself has iterated over time. In 2018 some SLRs shoot DNG (not Nikon or Canon of course), Apple’s cameras shoot patent-encumbered HEVC (not HEIF, that’s the container damnit), and there’s lots of proprietary RAW. I suppose HEVC is an improvement over RAW, but only by a bit. HEVC is likely to lose out in the long run to AV1 and disappear — with no comment from Apple when it converts. In terms of a practical archival image format we basically have PNG and JPEG with no metadata standard and perhaps some flavor of TIFF. Basically everything sucks, which is very 2018.

Incidentally, the image formats Preview can export to in Sierra when you use the Option key trick (apple doc) — prior to Mountain Lion they were all shown. OpenEXR was from Industrial Light and Magic but it’s as dead as old JPEG-2000. (As near as I can tell Mojave has the same list and it still doesn’t include HEIF/HEVC, which seems a vote of some sort.)

Did you follow Apple's two-factor authentication advice to provide a friend's phone number for SMS authentication?

I bet you didn’t do this [emphases mine]:

…You should also consider verifying an additional phone number you can access, such as a home phone, or a number used by a family member or close friend. You can use this number if you temporarily can’t access your primary number or your own devices….

… To use two-factor authentication, you need at least one trusted phone number on file where you can receive verification codes. Consider verifying an additional trusted phone number other than your own phone number. If your iPhone is your only trusted device and it is missing or damaged, you will be unable to receive verification codes required to access your account…

… If you're signing in and don’t have a trusted device handy that can display verification codes, you can have a code sent to your trusted phone number via text message or an automated phone call instead. Click Didn’t Get a Code on the sign in screen and choose to send a code to your trusted phone number…

Apple 2FA implementation has a high risk of account access loss (Google has better 2FA recovery options). Apple’s recommended mitigation is to use multiple SMS verification numbers, not just the one mandatory number. Since SMS is an essential part of Apple’s 2FA, and SMS is a poor way to do 2FA, Apple’s 2FA is fundamentally insecure (mac bloggers seem universally unaware of this incidentally).

Damned if you do and damned if you don’t. On balance, if you use 2FA, you should have at least two SMS numbers numbers associated with your (insecure) Apple ID.

PS. To Apple’s credit, you need both a password and SMS to access your iCloud account, and you can’t reset the password just using SMS. In the absence of a trusted device the password reset process is mysterious and takes a few days.

PPS. You can use a Google Voice number as a trusted number. That way you can use a web browser to retrieve the authentication code.

Remember to remove a device from your Apple ID account list if it is sold or wiped

Apple associates devices with your Apple ID. If you are using two factor authentication they get authentication requests. If you wipe or sell or dispose of a device you really want to remove it from this list.

Go to https://appleid.apple.com/account/manage, select the device, and remove it.

I don’t think the reset/wipe phone setting does this automatically.

Printer configuration is a still a pain (AirPrint edition)

I replaced an old Brother HL-L6200DW with a new Brother HL-L6200DW. Isn’t stasis wonderful? Scanners and printers don’t change much any more.

I sort of remembered how to test these up. I got it working via ethernet, then got it on our WiFi network. That worked for my Mac, but AirPrint wasn’t happy. Same printer, same name, different Mac ID. iOS said it would let me update to the new printer, but then it failed.

I had to restart my router, rename the printer in the web GUI (I’m sure this isn’t in the manual), restart printer, restart phone. Then it found it.

I’m sure there’s an easier way. Nobody prints though, so maybe not.

PS. The printer has all kinds of web services enabled. It’s basically waving a hack-me flag. It’s a bother to figure out what I can turn off.

An almost useful Siri Shortcut: "Voice memo record"

Apple forgot to add a Siri command that would launch Voice Memo.app and start recording. You can launch it with Siri, but it won’t start recording.

They did add a Siri Shortcut though - “Record a new voice memo”. So I finally found a use for that feature, I added “Record voice memo” and “Voice memo record” as Siri shortcuts. I think the 2nd one works better.

“Record a memo” doesn’t work because Siri uses that to open Voice Memo.app without launching the recording (iOS doesn’t warn about this during shortcut creation). The behavior is also a bit buggy, sometimes the phrase launched Voice Memo.app but didn’t start recording.

I tried creating a springboard Shortcut, but even though “Record a new voice memo” is in Siri & Search settings shortcuts, it’s not available in Shortcuts.app. (btw, deleting a Shortcuts.app shortcut uses an insanely weird UI).