Power monitoring generator, well

We have a new generator, since we live in a third world country where PG&E is unable to provide reliable electricity. It works pretty well, a fancy Generac standby unit with load shedding. Even has a decent website monitoring the generator via WiFi, also there’s hacker software for reading the generator’s state from a serial diagnostic port.

Anyway, when we’re on generator power the UPS units sometimes complain, switch to battery power briefly. I think I’ve also noticed the lights flickering. That got me worried the generator was putting out bad power so I hooked up my IoTaWatt to monitor the AC power.

The IoTaWatt’s main function is to monitor power draw through individual circuits. It directly measures current. But it also monitors voltage and frequency via an AC transformer sensor. Near as I can tell it samples that data many times a second but the most detailed data I can get from the status API is once a second. The most detailed graphs are every 5 seconds.

Generator load

Voltage and Frequency. Generator switches on at 12:06

As the above graph shows, the voltage and power doesn’t really change much when we switch over to generator power. Both PG&E and the generator are providing about 122V. The generator’s frequency isn’t quite as stable as PG&E was but a variance of ±0.1Hz isn’t that bad. (PG&E sometimes approaches that variance too.)

Well-related power blip

The graph above shows what happened as 12:17:30, one of the times my UPS kicked in. Turns out just then the well pump turned on (the red line), adding about 1.5kW load (or about double total power, the blue line) in that instant. The generator didn’t quite react fast enough. The power draw of some other loads (yellow and purple, basically room lights) dips briefly. More telling: the frequency of power, the black line, drops significantly, by nearly 0.15Hz. That’s the sign the generator is overloaded, it starts running more slowly and the frequency drops. It recovers quickly enough, but it’s enough to make a sensitive UPS unhappy and cause a visible flicker in the lights. Note the voltage (green line) doesn’t notably drop; I imagine the generator has a voltage regulator that smooths this out.

So that’s why the lights flicker and the UPS complains. I think that’s just a limitation of a consumer generator. It’s plenty of power to make everything run, but it’s not quite as clean as when you have utility power.


Looking at this reminded me to take another look at what the well pump was doing.

It’s still coming on every 30 minutes or so even when there’s no water demand at all, which suggests a leak somewhere. I know it’s not outside, because I turned that valve off. I’m pretty sure it’s not inside either, I’d have noticed. Maybe the well is leaking back into itself, against the check valve? Dunno. Like I said before it’s wasting about $0.50 of electricity a day, but otherwise seems mostly harmless.

Above is a cumulative graph of well power usage over 24 hours (blue) with transitory power (green). As you can see we use a lot of water overnight, when the irrigation is running, about 5kWH or $2 worth. The well doesn’t stay on continuously for much of this, other than one period about 6:15am. Our water system has no buffer of any kind; no tanks or reservoirs.

Ubuntu 19.10 to 20.04 upgrade failure

I seem to be cursed when it comes to Ubuntu updates. I just tried upgrading Ubuntu 19.10 to 20.04 (ie: eoan to focal) on a ordinary home server. Once again do-release-upgrade failed right as systemd was being updated. In the past that’s because the systemd upgrade broke my network configuration, because of course I’m doing the upgrade via ssh. (In a screen session though!) This time around it was worse, because the system wouldn’t boot:

- - -[ end Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(0,0) ]- - -z

Well that’s exciting.

Turns out this is a well known problem with an easy solution. Basically if something interrupts an apt upgrade of the kernel at the wrong time, the kernel will be installed and chosen as the default but the matching initrd ramdisk to make it bootable won’t be properly installed. (ramdisk creation seems to be deferred to the very end of the apt process). The result is the system won’t boot with the new kernel. UEFI may be part of what makes this a challenge.

Fortunately there are usually other older kernels also installed that still work on the hard drive, so all you have to do is convince GRUB or whatever boot loader you’re using to boot an older kernel. Then you can go in and fix things.

The command that seems most clearly to fix things is
dpkg --triggers-only linux-image-5.4.0-29-generic
Of course you need to substitute the right package name there, for the latest kernel that’s not booting. You can find it this way by listing all packages installed with “triggers pending”:
dpkg -l | grep ^it
Another bit of advice on that Stack Overflow page is to run
dpkg --configure -a
This will force any packages that were installed but not configured to configure themselves. My aborted Ubuntu upgrade seems to have left me with ~40 unconfigured package, this fixed them all in a jiffy.

I sure wish I knew why upgrading systemd sometimes breaks networking on Ubuntu. It’s happened on two different systems of mine. And it’s an absolute disaster if you only have local access to the machine. Hilariously, do-release-upgrade still advises against running it if you’re ssh into the machine. How am I supposed to update the machine, fly to the datacenter and type at the keyboard?!

I also wish that apt or do-release-upgrade were a little more robust in leaving you with a bootable kernel.

The good thing here is that apt and dpkg are very robust themselves. This is the third time I’ve had do-release-upgrade fail on me now, but each time I can pretty much rely on apt and dpkg to fix whatever mess was created.

Bonus round: USB keyboards

This whole ordeal got a lot more complicated because I couldn’t find a keyboard to plug into the Intel NUC hardware to access the boot console. I use an old USB wired Apple Mac 101 key keyboard on my regular computer. For some bizarre reason that fails to work when plugged in to the NUC for accessing the BIOS. I ended up having to wait two days for Amazon to ship me a cheap $10 Chinese keyboard. That’ll teach me to not have a bunch of old computer parts lying around the house.

Making Amiibo for Animal Crossing

Nintendo has this system called Amiibo where you buy little figurines for a game and then scan them on your Switch console to do various in-game things. Breath of the Wild Amiibo unlock outfits, for instance. And Animal Crossing New Horizons Amiibo unlock new villagers to come live with you.

But Amiibo are expensive! New they are in the vicinity of $20 on Amazon. And there’s some collector / scarcity thing going on too so some of the rare ones are like $100. I’m not a big fan of paying real money just to unlock digital content. I mean sure I’ll pay for a major DLC, but I have no ethical problem creating my own unlock for a character.

Anyway, it turns out that Amiibo are just NFC 215 tags. And those are easy to write with pretty much any NFC-capable mobile phone, or with a $30 bit of NFC hardware. (Surprise! Your phone can write tags as well as read them!) And of course you can easily find the tag data online.

I ended up following this guide on Reddit. The FAQ is helpful too. In short:

  • Buy some blank NFC 215 tags on Amazon. They cost anywhere from $0.25 to $1.00 each depending on volume and shipping speed.
  • Get an NFC writer. I used TagMo on my Android Pixel 3 phone. It’s an unauthorized app, so you’re taking a small security risk installing it.
  • Get the special unfixed-info.bin and locked-secret.bin files installed on the tag writer. They’re only 80 bytes each. See the Reddit guides for where to find these.
  • Get the Animal Crossing tag data. This requires downloading files of dubious licensing status. A site called NFC Bank is a popular source.
  • Burn the tag. You can then scan it again to test it.
  • In game go to Isabelle and choose the Amiibo option for inviting visitors. Scan your new tag with the game controller.

The one small bummer is the cheap tags aren’t reusable. There’s alternate hardware which will spoof NFC tags for you. I think also some more expensive writable tags are re-writable.

Another option if this all seems too hard is you can apparently buy pirated Amiibo tags on Etsy, eBay, etc. Folks put a lot of effort into making nice cards or buttons for them too so they look good.

Firefox browser versions for Android

Summary: Firefox Browser is the stable consumer product. It’s about to be replaced with Firefox Preview, a major new release you may want to switch to early. There’s also a collection of betas and nightly builds.

One thing that’s nice about Android is that Firefox works nicely on it; there’s no iOS requirement that all browsers really just be reskinned versions of the vendor’s browser component. I’ve set up Firefox to be my default browser and that setting mostly sticks other than Google launching Chrome occasionally on me. I think Firefox is even used as an embedded browser in other things, but I’ve never been sure.

Anyway, the confusing thing is there’s so many things called Firefox on the Google Play store. Here’s a rundown of them all.

Firefox Browser (org.mozilla.firefox): this is the one you probably want, the mainline consumer app. This is currently version 68.8.0. This is the old Firefox that’s been around for awhile; this is not the new version announced in January 2020.

Firefox Preview (org.mozilla.fenix): this is the new hotness. Version 4.3.0 for me right now, but also says “GV: 76.0”. It’s a major update involving a port to GeckoView. See also these details of new features. Dark mode, more privacy, an option to put the toolbar on the bottom of the screen (near your thumb), a Private Browsing button right on the tab selector. Main change though is a full new rendering engine. Supports fewer addons, but does have uBlock Origin. This new stuff is slated to be released to the public as the Firefox Browser pretty soon.

Firefox for Android Beta (org.mozilla.firefox_beta): this looks like a slightly older version of Firefox Preview. Currently reports version 75.0.0-beta.6, also “GV: 75.0”. I would guess ordinarily this would be the cutting edge beta version of Firefox but right now during the transition to the new hotness, Firefox Preview is getting updates faster. Not positive.

Firefox Nightly for Developers (org.mozilla.fennec_aurora)
Firefox Preview Nightly for Developers (org.mozilla.fenix.nightly): I didn’t test these. I assume they’re just nightly builds of the two main development lines. FWIW the Preview Nightly currently reports “GV: 78.0a1” so it looks fairly far ahead of the main Firefox preview

Firefox Focus (org.mozilla.focus): a privacy-oriented experimental browser. Version 8.2.0 (75.0) right now. I think this was most relevant a couple of years ago and has taken a backseat to what is now Firefox Preview, which includes some of the same privacy features.

There’s also a bunch of stuff using the Firefox brand name that isn’t a browser at all. Firefox Private Network VPN (org.mozilla.firefox.vpn): is Mozilla’s new project selling VPN service to users. Firefox Lockwise (mozilla.lockwise) gives access to the Firefox password store. I think they’re trying to become a password agent like 1Password? Firefox Send (org.mozilla.firefoxsend) is a tool for sharing files. Last update was March 2019, not sure if the project is abandoned. Notes by Firefox is yet another text memo product. Firefox ScreenshotGo organizes screenshots with OCR and text search. And finally Mozilla Stumbler is a wardriving tool for building out Mozilla Location Service data. Last release Feb 2019.

Android Tablet: Samsung Galaxy Tab S5e

I just got a new Android tablet to replace my 2017-era iPad. Since moving to an Android phone last year (Pixel 3) I’ve come to realize I just like Android more. For ordinary stuff ordinary people do, Android is good, just as good as iOS. Sure there’s nits to pick with both systems, but they’re both great.

What I like about Android is that it also allows a level of more sophisticated stuff. There’s a brilliant capability for apps to interact with each other, for instance, setting things up like Unix pipelines. Android out of the box lets you customize the keyboards, the launcher, really most aspects of the system UI; no jailbreaking required. And a fair number of the Android apps seem more hacker friendly than their iOS equivalents.

Android’s ecosystem also does seem genuinely more open. Firefox integrates as a browser way better on Android than Apple ever allowed. Want to add more storage? There’s an SD card slot. (Why the F has Apple never allowed that?) Want to buy a book in the Kindle app? Of course you can; it’s not like iOS where Apple demands their 30% cut. Want to sideload an app from some random site? No problem, just disable the security check and you can. (A double edged sword, to be sure.)

The big downside to Android is that the vendors don’t distribute vanilla Android, they hack and munge and add crap. Worse they subtract value, particularly in how they sit on operating system versions for months, years, forcing folks to run out of date buggy versions. Google really needs to get more control of this problem.


Another problem with the Android ecosystem is the hardware quality varies. I dorked around with an Amazon Fire 7 briefly but the hardware is such garbage there was no joy in using it. (To be fair, it was super cheap.) And tablets are seen to be a losing market for some reason, particularly on Android, so there’s not a lot of options.

I read a bunch of guides online (this one was the best) and landed on the Samsung Galaxy Tab S5e. It has an excellent screen (2560×1600 AMOLED), is very thin and light, and at $350 is cheap enough to feel like an accessory / toy, not a whole laptop. The Galaxy Tab S6 is the new flagship, better in most every way, but at $650 it’s awfully pricy. The only thing it’s really missing is S Pen support but I had no desire to have that anyway. There’s an S6 Lite coming but it seems worse than the S5e for the same price, I don’t quite get it.

The S5e hardware seems great. Very solidly built. Nice camera I’ll never use, a working fingerprint reader for unlocking, wifi seems good. Screen is lovely although there’s some weird white balance software at work that’s not helping. Only complaint so far is I can’t find a magnetic cover just for the screen; all the cases are wraparound and cover the back too. Oh well.


So, what about the software? First, the big problem; it’s still only running Android 9. Android 10 came out eight months ago. Being on 9 puts me well ahead of a lot of crappier Android devices, but I hate being beholden to Samsung to get around to putting out an OS release.

Still it works nicely as a stock Android system, there’s none of the lockdown shenanigans that Amazon Fire does trying to keep you from customizing. The GBoard keyboard installs fine, including local voice recognition. Samsung’s launcher is OK but I switched to Nova Launcher mostly because it alphabetizes the apps tray view. Also it’s nicely customizable.

Samsung does install a bunch of its own crap, but it mostly stays out of the way. Bixby was the biggest annoyance but Nova Launcher mostly hides it. For some reason there’s two Clock apps and two Calendar apps and the like, one Google and one Samsung. Meh, I can live with it. It’s annoying Samsung won’t let me remove their preinstalled stuff.

But Android is hacker friendly, so you can install them! I followed this guide to uninstall a bunch of apps using adb to get a command line shell on the tablet. (Try that on an iPad, lol.) Another option is to use a guide like this one along with an app and config files which debloats the device for you. I wanted to control it myself, so I did it by hand.

I already had adb on my system thanks to having Android Studio. The adb.exe executable is in /Users/nelson/AppData/Local/Android/Sdk/platform-tools for me. Here’s a list of stuff I uninstalled by hand:

  • com.microsoft.skydrive
  • com.microsoft.office.officehubrow
  • flipboard.boxer.app
  • com.samsung.android.calendar
  • com.samsung.android.contacts
  • com.sec.android.app.popupcalculator
  • com.samsung.android.email.provider
  • com.sec.android.app.kidshome
  • com.sec.android.app.samsungapps
  • com.sec.android.app.samsungapps
  • com.sec.android.widgetapp.samsungapps
  • com.sec.android.app.sbrowser
  • com.sec.android.gallery3d
  • com.samsung.kidsplay
  • com.samsung.android.app.notes

Special GSAM bonus notes

I use the GSAM Battery Monitor to track battery usage for apps. It works fine by default, but it can work better if you grant extra permissions to the app to let it get at some debugging APIs. There’s multiple conflicting instructions online about how to do this, including a very nice tutorial in the app itself that works right until the final crucial command. I think this worked fine on my Pixel 3? But it didn’t on this tablet, what finally worked was at the bottom of this discussion:

adb -d shell pm grant com.gsamlabs.bbm android.permission.BATTERY_STATS

TCL Series 6 TV

Just got a new TV, a TCL 6 Series and couldn’t be happier. I bought it after this glowing review. Pretty much every other review says the same thing; excellent picture quality for the price. It’s quite cheap even for an LCD TV, and definitely way way cheaper than an OLED.

I’m not able to judge picture quality except to say it looks great to me. It’s better than the 2013 era Samsung panel it replaced; brighter, bigger, sharper. The 4K 60fps demo reels look amazing. Of course the default picture settings are oversaturated garbage, but fixing that isn’t too hard.

The key picture setting for this screen is “movie mode”. That means “don’t blow out the colors”. However when you enable that it also for some dumb reason defaults to “warm color temperature”, which shifts all the colors to the red. So set that back to normal. Also it turns off Local Contrast which is one of the nice features of the set, so turn that on to high. Then turn off all the fancy motion interpolation type stuff and it looks pretty great.

I particularly like that there’s a quick setting for “darker / normal / brighter”, which means you might actually change that based on time of day and room lighting. There’s also a game mode for HDMI inputs that has decent latency (19ms, or just about one frame).

One nice feature for this set is that its operating system is Roku. Not some dumb badly designed TV manufacturer stuff, just good ol’ Roku with its open ecosystem of third party channels. I’ve been using Roku devices for years, mostly for Plex and Netflix and Amazon, it’s nice to have it native in the TV. I managed to get ARC (“Audio Return Channel”) working so the sound is still being sent to my fancy AV receiver for decoding.

Speaking of the receiver, I’m less clear on what to do with it now. It’s limited to 1080p (or 2160p @ 30 fps, lol). And seems to block HDR too, or at least my PS4 won’t do HDR through it. I guess I could just plug all my inputs in to the TV and use the receiver solely as a digital audio decoder + amplifier. The wiring is tidier if the receiver is a switch, but then I can’t pass 4K/HDR content through it. If I didn’t already have speakers installed in my walls I think I might just get a nice soundbar and call it a day.

Still haven’t considered the joy of making my old Logitech universal remote drive the new system. Need to settle down how the receiver fits in first.

4K is still such a bandwidth hog I can’t imagine I’ll be using it much if any for watching video. Would like it for games though. HDR interests me much more, it’s great we can finally get higher contrast ratios on TV sets. Apparently there’s competing standards for HDR, awesome. This panel supports both Dolby Vision and HDR10, which I guess are the two most common formats. Hopefully that’s good enough.

Upgrading from an unsupported Ubuntu (18.10 to 19.10)

Update (March 2020): I’ve heard from 10+ people who say these instructions work for them, I think it’s reliable. See also this Ask Ubuntu post. Some folks are finding their way to this blog searching for “An upgrade from ‘cosmic’ to ‘eoan’ is not supported with this tool.” That makes sense, it’s the error the upgrade tool gives. Finally, if you see an error about “Upgrades from 18.04 on the i386 architecture are not supported” that is sadly true; 32 bit support stopped in 18.04. I don’t know how to help you with that.

A couple of weeks ago I blogged about being unable to upgrade from Ubuntu 18.10 (cosmic) to 19.10 (eoan), or even 19.04 (disco). Ubuntu 18.10 is fully end of life and deprecated and the do-release-upgrade tool we normally use to upgrade things refused to run. (Note that this tool is bundled in the ubuntu-release-upgrader-core package) Fortunately one of my readers Alejandro saw my post and suggested a solution. I just tried it and it works. Details below.

The underlying problem is do-release-upgrade on 18.10 was trying to upgrade directly from 18.10 → 19.10. But that’s not possible. And it’s no longer possible to upgrade 18.10 → 19.04 because 19.04 is also not supported. So you’re stuck.

The gist of the solution is to hack a config file to force an 18.10 → 19.04 update. It turns out Ubuntu’s do-release-upgrade tool is a generic program, the version installed on an 18.10 system is not 18.10 specific. Instead it downloads metadata files from the Internet and uses those to figure out what to do. Alejandro’s suggestion boils down to intercepting those metadata files after download and hacking them to point to 19.04 instead of 19.10. The packages needed to do the 18.10 → 19.04 upgrade are still online, do-release-upgrade just wasn’t finding them.

Needless to say, the instructions below come with a big warning this may break your system. It worked for me, but no idea if it will work for you.

18.10 → 19.10 steps

  1. zRun do-release-upgrade on the 18.10 system. This will give you an error about being unsupported. But behind the scenes, the tool will download some metadata files we want to modify.
  2. As root, go in to /var/lib/update-manager and copy the file meta-release to a new file meta-release2. This file was downloaded by do-release-upgrade from the Internet and tells the upgrader how to upgrade.
  3. Edit meta-release2. Remove all entries for eoan entirely. Modify the disco entry so it says Supported: 1
    Update April 2020: reader F.K. write to say “I found that ubuntu repo returns 404 and `do-release-upgrade` can’t run update. I changed all link in meta-release2 for disco to  http://old-releases.ubuntu.com/  and its works.” I can’t test this but it seems plausible of Ubuntu finally moved the disco update files out of the live repo (say, in preparation for the 20.04 release.)
  4. Edit the file /usr/lib/python3/dist-packages/UpdateManager/Core/MetaRelease.py. Change this line of code
    self.metarelease_information = open(self.METARELEASE_FILE, "r")
    To read
    self.metarelease_information = open(self.METARELEASE_FILE + "2", "r")
    That will tell the upgrader to use your modified file instead of the original. (It will also avoid any redownloads overwriting your changes.)
  5. Run do-release-upgrade. It should now be doing an upgrade 18.10 → 19.04. Let that run as normal and reboot.
  6. Congratulations! You’re now running 19.04. Remove the /var/lib/update-manager/meta-release2 you made.
  7. Since 19.10 is supported, all you have to do to upgrade 19.04 → 19.10 is run do-release-upgrade again. No hacks necessary, you’re back on the main path.

Like I said, it worked for me. I’m a little surprised Ubuntu doesn’t support this officially out of the box, it seems simple enough. I suspect they just want to have nothing at all to do with older systems. Still if it were me I’d add some flag to enable at-your-own-risk upgrades for people like me who were stuck.

I think I also could have upgraded my system simply by editing /etc/apt/sources files and relying on apt dist-upgrade. That’s the underlying tool that’s doing all the work anyway. But the Ubuntu upgrade scripts have a little extra stuff in them and it’s nice to be able to run it just in case it’s important.

I’m very grateful to Alejandro for his emailing me the suggestion on the fix.