Cyberpower pwrstatd revisited

Eight years ago I wrote a blog post about the Linux software for my UPS. I was pretty negative about it. What’s funny is now 8 years later I’d forgotten about that post and re-installed that software and thought “hey, that’s not so bad”. Mostly because they publish a dpkg which just works. But revisiting my post I see my complaints still apply: requires root, no source code, aggressive shut down options. They’ve even added a thing now where it schedules the whole UPS to turn itself off 10 minutes after the outage. WTF?

Anyway, reconfiguring the daemon so that it never shuts anything down itself, it’s OK software. Not great, not bad. Mostly what I want is the munin plugin for monitoring the thing. I put one in my blog post; someone picked it up, improved it, and has it on Github now. Open source works!

OTA digital TV notes

Did you know TV shows are broadcast for free over the airwaves? It’s amazing; no need to pay the cableman or satellitelady their rent. At least, if you want to watch ad supported non-premium programming.

OTA digital TV has gotten pretty simple. Basically there’s just MPEG-2 streams floating around at ~600MHz for the taking. I just got a tuner and antenna to pick them up. An HDHomeRun Connect Duo for the tuner and a Clearstream Eclipse antenna. Total cost: $100 to $150. There’s other options, see Wirecutter.

So far so good; without any effort to set up the antenna right, in San Francisco I can get 46 channels. Really it’s 12 channels, but there’s multiplexing so where channel 2.1 is KTVU in HD, channel 2.2 is KTVU-SD and channel 2.3 is a random movies channel.

The diversity of channels is pretty neat. I’m enthralled by channel 60.5 PJKFNX which is showing First Nations Experience, a whole channel of indigenous peoples programming. I think YouTube is the only other place I could find content like that.

Devices

The antenna is nothing exciting. Some flat piece of floppy vinyl with some very stiff coax cable coming out. This particular Eclipse antenna has a 20 dB amplifier attached to it. No idea if the amplification is necessary, or if more would be better.

I like the HomeRun tuner. It takes power and antenna in, has ethernet out. That’s it. The device serves MPEG streams and some diagnostic info via a web interface. Very simple and hackable.

Software

I’m currently watching via the HDHomeRun Windows app. It’s a fine bare-bones viewer, backed with some sort of program guide. They run a DVR service too; I wonder if they even bother recording anything or if it’s more of a Sling-like setup. There’s mobile apps too and I think a new Roku app.

The HDHomeRun includes a basic HTTP API for programmatically getting a list of channels and watching MPEG-TS streams. This caveat is fascinating

The client will have a slightly different concept of how long one second is due to slight differences in the client clock timebase vs the broadcaster clock timebase. The client must adjust its concept of real time to match the incoming video stream

No one watches live TV anymore though. I bought this in part because I finally got a Plex Pass and one of the value-adds for that is Plex has DVR software. Setup of the Plex DVR was easy, the program guide is good, and it seems to record OK. Recording seems to run a transcoder stream at 150% CPU on my system which is odd, I have both commercial skipping and transcoding compression disabled. Maybe it’s just having to collect and assemble the chunks. The final output is a .ts container that has what looks like the broadcast MPEG-2 and AC3 streams in it. I’ve set up recording a show regularly, we’ll see how it does.

One thing I’ve noticed is a lot of the older video I’m seeing is interlaced. And I guess decombing isn’t a standard feature. It’s ugly.

Update the DVR seems to work great. I also enabled commercials skipping and transcoding for smaller storage. It seems to work OK. Commercials skipped even in a live news broadcast, which suggests they’re doing some live detection mechanism and not relying on a content database.

Transcoded to 6Mbps 720p (H.264) using about 1.5 CPU cores out of my system’s 8, so the impact isn’t too bad. Resulting file is 3GB/hour for 720p, which is not great, but for a fast transcode it’s not bad. You can see the settings they’re using here; as the commenter points out, a lower CRF would be a nice space / quality tradeoff. Right now it’s not configurable.

Channel availability

I’m currently getting channels 2, 4, 5, 7, 9, 38, 42, 44, 50, 54, 60, and 66 (with many variants). That’s the major “broadcast” networks (ABC, CBS, NBC, and regretfully FOX) plus a bunch of other random stuff. Most channels have 99%+ signal quality, but 60.5 is only 84%. Still looks fine to watch. Note this testing is being done right before the FCC repack when stations get renumbered to free up some bandwidth.

There’s several sites that’ll let you check what channels you should expect. Results below. To simplify things I’m assuming if I get one channel I get all the other related multiplexed ones.

Note each test has a different list of stations. I think what my antenna auto-detected is a strict subset of all of those. What these online sites tell me is I can probably get a few more channels with better antenna placement. I will note I did a second scan and got the exact same result of 46 channels, so at least it’s stable.

AntennaWeb: 2, 4, 5, 7, 9, 11, 12, 22, 38, 42, 44, 45, 50, 52, 54, 60, 66

FCC: 1, 2, 4, 5, 6, 7, 8, 9, 10, 11, 14, 22, 26, 28, 36, 38, 44, 48, 54, 60, 65, 66, 68

TabloTV: 2, 3, 4, 5, 7, 9, 11, 12, 38, 42, 44, 45, 48, 50, 52, 54, 60, 65, 66

TVFool: see picture below. I’m not sure their database is current, but I love this nerdy visualization.

Facial recognition for the public: Yandex

You can use Yandex Image Search right now as a pretty good facial recognition system for anyone who has labelled photos on the Web. I believe this is the first generally accessible facial recognition system with a large database. Yandex isn’t designed for this purpose. The trick is to upload photos cropped to a face and it’ll work more or less to find similar faces.

This post was inspired by Bellingcat’s recent post on image search which noted Yandex’s image search as something particularly good and particularly called out the facial recognition. I’m just elaborating on what he found.

Update: see this article about ClearView, a company doing facial recognition with scraped photos from Facebook, etc. Also this Dec 2019 video reporting about Yandex facial recognition. The Daily Mail also published an article based on this blog post.

Demonstration with a selfie

Here’s a selfie I took today. It’s never been on the web before this post and appears in no databases as I do these searches. This is a good clear photo of me, which makes the results better, but it works with less good photos too.

Now pretend you didn’t know my name. Put the image in to Yandex. Here’s the result you get.

The first two images are me. Click on either of them and the web page it takes you to has my name on it. Now you know who I am!

It’s not perfect; there’s no name provided directly, someone has to read the linked page to find one. And a person still has to sift through the near-matches of faces. But it’s pretty good. The only limit is Yandex’s database. That’s kind of “the whole web”, but their index is nowhere as comprehensive as Google.

Demonstration with Andrew Yang

Recognizing myself is easy. How about a stranger? I used this today to identify who the “Yang” was in this image from CNBC. It’s supposed to be Andrew Yang, the presidential candidate, but it is not.

Crop the image to Yang’s face, put it in Yandex, and you get this

All five of the first images look like our guy. The first result on the Yandex result page also leads directly to his home page: Geoff Yang. Some CNBC intern is gonna be in so much trouble! (This post is not an endorsement of Andrew Yang.)

Facial recognition vs image search

The new thing here is that Yandex is working not just as a reverse image search tool; it seems to be doing facial matching. None of the images it finds look much like the original image; different aspect ratio, different background, different details. But the faces match. It even matches my selfie both to a much younger version of me without a beard and to a version of me holding a box up obscuring part of my face. That’s pretty wild.

FWIW, here’s what other image search engines do with my selfie. I tested Google, Baidu, Bing, and TinEye. TinEye returned 0 results.

Google
Baidu
Bing

Look at all the beardy white guys with some yellow in the image! None of them are me. It’s not a bad guess; I gave it a head shot, it found me similar head shots. But it didn’t find my face. Yandex finds my face. (Arguably TinEye’s 0 results is the right answer for this query; this image does not exist online.)

Why this matters

We’re on the cusp of a major change in day to day privacy with facial recognition algorithms. AI has now gotten good enough that it can do excellent facial recognition; modern software can take a blurry photo of half a face and match it to other faces of the same person. It’s a pretty scary future, particularly as already put into place by authoritarian governments like in Xinjiang, but it is inevitable and unstoppable.

Right now an ordinary person still can’t, for free, take a random photo of a stranger and find the name for him or her. But with Yandex they can. Yandex has been around a long time and is one of the few companies in the world that is competitive to Google. Their index is heavily biased to Eastern European data, but they have enough global data to find me and Andrew Yang.

If you use Google Photos or Facebook you’ve probably encountered their facial recognition. It’s magic, the matching works great. It’s also very limited. Facebook seems to only show you names for faces that people you have some sort of Facebook connection to. Google Photos similarly doesn’t volunteer random names. They could do more; Facebook could match a face to any Facebook user, for instance. But both services seem to have made a deliberate decision not to be a general purpose facial recognition service to identify strangers.

There is at least one public recognition service out there: FindClone (hat tip to Brad F). You can read more about FindClone here. (It used to be known a FindFace, and SearchFace.) However it’s limited to photos from VKontakte, the Facebook for the Russian speaking world. It’s a big dataset. But it’s not international like the Yandex search engine is.

I imagine within a few years this “look up a name from a face” will become as common as “look up a name from a phone number”. Not quite ubiquitous because everyone finds it creepy, but easy enough to access if you need it.

UPS battery lifetime

I use cheap UPS units for battery backup. Things like the CyberPower CP825LCD, basically power strips with a 12V battery and an inverter. They work OK.

For a few years. The lead acid batteries in them eventually wear out, and after 3-5 years they don’t hold a useful charge anymore. They lose capacity over time, too. Unfortunately these cheap UPS units don’t have a good way to monitor their battery health. You only notice when the power goes out and find you have 5 minutes of battery backup, not 15. Or else the unit just fails entirely and starts beeping confusingly or doesn’t provide power, even when plugged in.

Checking battery health

There is a way to easily check the battery life though, at least with CyberPower UPS units. Their PowerPanel software connects to the USB port on the UPS and gives you info on battery status. Including a self test which calculates battery capacity in the moment

Before I clicked the “self test” button it was saying Battery Capacity was 100%. But that was a guess based on the fact the battery was fully charged. Self test swaps over to battery power for a couple of seconds and then measures drain. This battery really only has 1/3 capacity which means it’s time to replace it.

You can also do a simpler version of this self test by just yanking the plug and seeing how long the UPS lasts. Works better if the UPS has some sort of display of battery life remaining. But I found that a little confusing; this software test works better.

Every UPS I’ve had also runs its own self test regardless of whether there’s any software talking to it or not. I’ve never seen this test report any useful information, though. Maybe when it fails that’s what sets the UPS off on the endless beeping until you change the battery? Dunno. I sure wish they’d invest a little more into communicating to the user this necessary maintenance.

One other neat thing about the software: they report wattage used as well. For some reason the CyberPower LCD units don’t show that on the LCD screen.

Another blessing of the software is it lets you turn off the beep alarm when the power goes out. I already know the power is out, I can tell because of how dark it is. I don’t need every UPS in the house beeping at me.

These UPS units offer software for Windows, Mac, and Linux. The last time I used the Linux software (from APC) it was scary bad code running as root so it could run shutdown. But it did reliably output numbers on battery activity that I could graph with munin.

Replacement batteries

The simple solution to a failing battery is to replace the battery. The CP600LCD I have that just failed is $70 new, or $35 for a battery from CyberPower. Not great. But fortunately lead acid batteries are standard and common; as long as it’s a lead acid battery with 7V, the same Ah capacity, and the same physical size, it’ll work. (Probably any capacity battery will work but if you want it to fit the case, you’ll want the same size.) I ordered an ExpertPower battery for $17.50, or just half of the CyberPower official price.

The one thing to look out for is that the connection terminals come in two sizes: F1 and F2. F2 is wider. My UPS has F2 connectors but the new battery has F1 spades, I’m hoping it’ll still fit well enough to just work. You can also buy very simple adapters, metal clips, but the Amazon price on something that cheap is ridiculous.

Don’t put your old battery in the garbage, they are hazardous waste. Instead dispose of them like you would a car battery. Most trash collection companies have a special pickup or dropoff option for them. Places like Auto Zone and Home Depot often take car batteries for recycling, too.

Better batteries

Battery tech has come a long way; why are we still using old lead acid batteries? Mostly because they’re cheap, and UPS units don’t need to be compact or light. There are other battery tech options though.

Our ordinary lead acid batteries are “VRLA”, sealed. There’s a different type of “flooded” lead acid battery that’s open. They work for more cycles, but they are constantly venting hydrogen and needing distilled water added to it. So not really practical for home use.

I’ve never seen a NiCd or NiMH consumer UPS. The articles I’ve read say they are just too expensive for the performance, although I’ll note that didn’t stop laptop manufacturers in the 90s / early 00s. There are some hacker projects you can find building cheap 5V UPS backups with NiMH AA cells for low power devices like Raspberry PIs. Sounds fun.

As for lithium, I think it’s just too expensive. APC does build one consumer lithium-ion UPS, the APC BG500. But at $270 for a 500VA backup it has a hard time competing with the $50ish 500VA from lead acid batteries. It’s smaller and the battery will last twice as many years, sure, but for 5x the price? There’s a range of high end rackmount lithium UPS units for datacenters where compactness matters, but now we’re talking thousands of dollars.

IoTaWatt energy monitoring

I’ve been playing with an IoTaWatt energy monitor. This is a bit of hacker-friendly kit that measures electricity usage on each circuit in your breaker box. It gives you real time wattage data with 5 second samples and makes it much, much easier to understand where power is going in your house.

There’s a lot of products on the market now that do this kind of thing. Many are full consumer turnkey products like Neurio or Sense. They have cloud based monitors and some fancy pattern recognition to tell you things like “your refrigerator is failing”. I opted for the more hacker-friendly one, both to keep the data private and to be able to play with it more myself. See also OpenEnergyMonitor. I tell you, what I really want is something like a Leviton Smart Load Center, where the panel and breakers do this kind of monitoring themselves. (Price seems to be ~$4000 for a house, compared to $1500 for traditional dumb breakers.)

This post is long because I’m in the middle of learning about this stuff. But after running it just for 24 hours I learned some things like I have a leak in my water system somewhere, that my office uses 80W even when everything is “turned off”, and just how much electricity videogames use when you’re playing them. Having precise numbers for all this stuff is great.

How it works

The key sensor technology is the Circuit Transformer, a ~$10 metal coil you clamp around the hot wire of a circuit you want to monitor. The magnetic field in the main circuit induces current in the CT, which you can then read out the other end as either variable voltage or amperage. The common split-core CTs are installed without disturbing the wires in the breaker box at all, and nothing is electrically connected. Even so it’s important to keep the CTs plugged in to the controller box; if they aren’t they can build up a dangerous charge.

The CTs are plugged in with 3.5mm jacks to a controller box. The IoTaWatt is an Arduinio-class system that basically takes 14 CT sensors as input, stores the data on an SD card, and has a web interface over WiFi. The custom software (not Unix) includes a decent graphing interfaces and enough APIs to load the data up to some other system like Emoncms or your own stuff. It seems pretty good.

Installation

A temporary installation isn’t as hard as I feared. You take the panel for your breaker box off, exposing all the wires. Then you clip the CT sensors of the appropriate sizes around the wires you want to monitor. Done! You don’t even have to turn the power off first although obviously you should to be safe.

Of course it’s not so easy. The tricky part in North America is we use split phase power. We get 240V from the power company but it’s split into two out of phase 120V legs. The CTs only monitor a single leg, so if you’re measuring a 240V circuit (like a well pump or a dryer) you need to use two CTs, or pass both hot wires through a single CT, or double up the measurements. Also it’s easy to install a CT backwards, so the values are all the wrong sign. Fortunately the IoTaWatt has many software tools to compensate.

The other problem is a permanent installation. All the CTs and wires take up a lot of room in the breaker box. And the IoTaWatt also has to go somewhere, and be plugged in to two wall transformers. If you have a big breaker box you could just leave it inside (if code allows), but I’m not sure how well WiFi works there. I haven’t figured out a good permanent install.

What I measured

I just went with a simple temporary install to get some basic measurements. I only measured for a subpanel in my house and wasn’t able to get some of the more interesting things like the refrigerator. What I did get:

  • My office: a simple 110V circuit. This is where I spend all my time and includes my PC, my PS4, and my TV. Also overhead lights.
  • Furnace fan: simple 110V circuit? Gas heat, so fan only.
  • Pool: full 240V circuit. I’m only monitoring half of it, so I have to double the numbers reported. This may not be entirely accurate if the load is not balanced. But my 385W pump load shows up as 200W on the graph, so doubling is roughly correct.
  • Well pump: a 240V only circuit. I should double the numbers reported.
  • Septic pressure dose pump: another 240V only circuit, should double.
  • Subpanel supply: a full 240V circuit, with two monitors installed. I believe it’s correct to sum the two numbers when measuring the load.

Data and Graphs

Here’s some data I collected after running the monitor for 24 hours.

Power usage for individual circuits (already doubled as needed):

  • My office: 4600 Wh
  • Furnace fan: 1200 Wh
  • Pool: 3200 Wh
  • Well: 1400 Wh
  • Septic: 700 Wh
  • Total measured: 11,000 Wh
  • Subpanel supply 1: 7400 Wh
  • Subpanel supply 2: 9300
  • Total subpanel: 16,700 Wh (or about $4/day)

Office and Pool

Office and Pool

Here’s a graph of power usage over the course of a day. The yellow line is my office, the blue line is the pool.

The pool is easy to understand. Note the graph is at half scale, I forgot to click “double” in the setup. The filter pump turns on around 2am, has a brief priming jolt, then runs steadily at ~400W until 6am. There’s a 30 minute window of very high usage, 1800W, that’s when the second pump turns on to run the sweeper. There’s also an occasional usage of power during the day, well below 100W. That’s a little pool cover pump I have that runs when there’s enough water. It was raining this day. The pool uses 3200Wh a day or $8.

My office is harder to understand. At night it’s a steady 80W; that’s with most everything turned off or in sleep mode, but I still have a bunch of network equipment and vampire loads from TV gear. The load in the morning of about 200W is when I turn on my computer and do low power stuff like reading web pages. It spikes up to 400W when I’m playing a game on my PC. The 350W load is when my PC is relatively idle but I’m playing a game on my PS4. The TV seems to also spike the whole circuit load up to 200W. There’s also some overhead lights in the mix. Putting this all together, I can estimate these loads:

  • Constant load 24/7: 80W
  • Idle PC use in morning: +120W
  • PC gaming: +300W
  • PS4 gaming: +250W
  • Watching TV: +100W
  • Full day: 4600Wh, or ~200W average.

Furnace, Well, Septic

Furnace, well, septic

The furnace fan (in red) uses 120W when it’s on. It doesn’t run at night. It averaged about 50W this day.

The septic pump runs a few times a day; this value should be doubled. It’s sorta correlated to water usage but not really, it’s not as simple as “every time I flush the toilet”. It’s a high load when it runs, 2000W, but only briefly. Average daily usage is 30W.

The water pattern is more troubling. (Again, values in graph should be doubled). My well pump is coming on every 30 minutes and it’s basically uncorrelated to water usage in the house. That to me says I have a leak somewhere. I turned off the outside water while monitoring this and the frequency went from ~25 minutes to ~33 minutes, so while there may also be a smaller leak outside the big loss is coming from the indoor plumbing. I’m pretty sure I don’t have any leaky faucets or running toilets. Maybe it’s the well itself and the water system’s pressure tank. Apparently a common failure mode is that the check valve leaks pressurized water back up into the well, so it has to refill itself every once in awhile. The power usage isn’t so bad (60W average, or $15 a month) but I don’t want to waste water. And it’d be better not to have the wear and tear on the pump.

Subpanel Inputs

This is the graph of the power going into the panel. Both legs of the 240V circuit, I believe it’s correct to sum these. Not sure why they aren’t balanced more? The yellow line seems to go higher when I’ve got lights turned on and the office is busy, so maybe it’s uneven 110V loads. Not sure it really matters anyway. Note that the 5 circuits I’m monitoring in detail only add up to 11,000 Wh of the 16,700 Wh the panel supplied this day. I’ve got a bunch of other circuits I’m not monitoring, I guess they add up to about 1/3 the usage.

Conclusion

I like this IoTaWatt! Even with a temporary install I think I could learn enough in a few days to model out all the power usage of my house. Idle load, average load, and max load for every circuit in the house. That will be hugely helpful when designing a power backup for PG&E’s stupid outages, whether a generator or a battery.

What I really want is to have this kind of monitoring all the time, storing history in the background. Would love to see how my usage patterns vary over time, or spot new water leaks, or realize when the refrigerator is failing. I think the IoTaWatt is sufficient to do that, at least if you can find a nice way to install it permanently.

I do think an integrated system like the Leviton Load Center makes more sense though. I wonder if their system is in any way open? I fear the answer is no. No hint it works with any of the open source software, and Leviton doesn’t even have it working with Alexa yet. I bet there’s a way to scrape data out of the web app.

Bonus

Another graph, this of the next 24 hours. All the doubling, etc is done correctly.

Backup cell ISP notes

My Internet in Grass Valley is flaky enough that I’m looking at getting a permanent mobile service as a whole-house backup. Right now I just use my phone as a hotspot and switch my computer to it when necessary. That works OK but it only gets my own computer, I’m curious about switching the whole house. (Also dreading it; what if one of the devices decides to download a 10 GB update while on mobile?)

Ethernet / router options

The Ubiquiti EdgeMAX EdgeOS router I uses supports having a second WLAN connection as either load balancing or pure failover. Obviously I’d want failover. The trick is that you need to plug the mobile device into an ethernet port on the router. So the usual wireless hotspot doesn’t work.

The one device that looks exactly right is the Netgear 4G LTE Modem. $110 for a SIM card in, ethernet out. Perfect! See review here. Except it explicitly says it does not support Verizon, my carrier of choice. Some of the user comments suggest it does work on Verizon but my guess is only the LTE works and if you have to fall back to 3G it can’t do CDMA. Not ideal. I could use it with AT&T. Also it does not support LTE Advanced.

This review site’s roundup of ethernet-out devices is discouraging. Only Netgear makes the simple ethernet-out product. There’s a couple of things that are also wireless hotspots or routers that have an ethernet port, maybe they could be adopted. Or something really expensive like this new 5G hotspot. One final option would be to get a USB device and use a Raspberry Pi to output ethernet, assuming it’s possible.

Another option that looks slick is the Ubiquiti Redundant WAN over LTE. $200 for some solid-looking LTE hardware that plugs into your ethernet. Also has a setup with AT&T for service already, $15/mo plus $10/GB. The only point of confusion for me is how the failover happens. The device says it’s for UniFi networks, so I imagine it does some Ubiquiti-proprietary coordination with the router (UniFi Security Gateway) to act as a failover. If so then it probably won’t work with my old EdgeOS device.

Back to hotspots

So maybe using a hotspot is the right idea and I should just get a dedicated hotspot device and service. It’s manual failover and won’t cover my whole house, but there’s lots of product choices. The Inseego MiFi 8800L Jetpack (aka Verizon Jetpack) looks terrific. It even supports LTE Advanced! It doesn’t support Verizon 3G which is weird for a product Verizon is explicitly selling. Maybe that’s OK though, 3G is on the way out in the US.

OTOH just using my phone works well enough for the few times I need to do this. Never been sure I really needed a dedicated device. I’m using a Pixel 3 and it supports LTE Advanced.

Mobile data providers

If I get a dedicated device I need a data contract and SIM for it.

I can add a second line to my Verizon account. $10/mo and then $10/GB. I’m unclear but I think this will share my phone’s data balance? I always have way more data than I need, so that seems like a good choice.

I took a quick look at AT&T prepaid MVNOs, as something I could use with that Netgear modem. The best I could find in the US was Red Pocket Mobile. $10/mo for 1GB, and then $10/GB for topups. Or $25/mo for 5GB if you know you’re going to use it a lot. I did not look at getting a proper monthly account with AT&T, or adding a line to existing AT&T service. I suspect the pricing is the same as Verizon.

Conclusion

Three options

  1. Stick with using phone as my backup. A hassle, and manual, but it’s paid for already.
  2. Get a Jetpack mobile hotspot and switch to it manually. More convenient, but still a hassle. Small extra monthly fee.
  3. Get the Netgear LTE Modem and an AT&T MVNO. Give me proper failover, although there’s the risk we then blow through the data thanks to some ignorant device. Have to buy a new contract, so can’t use existing data on the Verizon account.

Option 1 sure is the least demanding up-front.