Author: nelsonminar

Some rough watt-hour numbers

~ nelsonminar ~ Leave a comment

PG&E’s incompetence means I’m thinking about electricity storage recently. Some rough numbers:

Devices you charge and use:

Batteries you can charge things from:

There’s a huge variety in the stuff I lumped together as “charger”, particularly how well they furnish AC or DC for usage. Your car battery is only as good as the inverter you plug into the stupid cigarette lighter, for instance. That EGO Power Station is designed to be a full portable AC power solution that can put out 3000W of clean power (well, for 20 minutes). Briefcase solar systems can also recharge themselves from solar panels.

I currently own a Powercore and two spare UPS units, for a total of about 275Wh. With that I can charge my laptop 4 times, my iPad 8 times, or my phone 25 times. I also have the car which is good for about 3x more charging just off the battery, which doesn’t count idling the engine to keep its battery charged.

My electric bill says my house averaged using 35,000 Wh a day for the last year. A lot of that is optional load; in a low power situation I wouldn’t be running the pool pump, the electric oven, etc.

A single rooftop solar panel generates roughly 300W maximum, or 1370 Wh per day (average over the course of a year in NorCal).

Archiving and deleting 23andMe

~ nelsonminar ~ Leave a comment

I decided to delete my 23andMe profile. All told it took about a week and I have a copy of everything I care about. Could have gone faster but I was careful and slow about it.

I did the test in 2011 with mixed feelings; genetic data is so super private and yet also so readily available to a determined attacker. I was curious what the reports would say. And I kind of wanted to support the company, I believed their research mission.

The consumer product never quite excited me. Other than one very significant thing (story forthcoming) it’s told me nothing useful. A lot of the stuff they highlight is dubious. I don’t see any reason to continue to pay for access to it.

And then there’s the law enforcement fuckery. Police keep getting more and more aggressive about accessing genetic databases. It’s not clear the courts will protect people. Time to delete the data.

Note that deletion isn’t really deletion. 23AndMe seems to make a good faith effort to remove data and destroy samples. But they may have shared some data and it’s no longer entirely under their control. Also your entire genetic data is stored “to meet CLIA requirements”, apparently some regulatory requirement where the government wants labs to keep data so they can evaluate lab quality. In theory all that stuff should be anonymized but I wouldn’t trust that against a determined attacker. If I’d known all this back at the beginning I never would have done the test.

So, what to do when deleting your 23AndMe account? Here’s what I did. Note I have very old data in their system, from a genetic test in 2011.

  1. Contact any connections you have and let them know you’re deleting your data. If you shared something with a family member it might be nice to give them a couple of days to take one last look at your data.
  2. Download all user data. This page is accessible from Settings, at the bottom under Data. It has buttons for downloading various sections. The reports data wants to print; I printed that to a PDF file.
  3. Be sure to download raw genetic data. It doesn’t download immediately; it creates a request and you get an email when it’s ready. (Mine took just a few minutes and was 24MB). This is the most important data on the site, the raw genetic test output. It’s all your SNPs. It’s possible to analyze this data with lots of tools, both online and offline.
  4. Download your archived reports. This only applies to pre-2016 accounts, before the FDA settlement, and only contains old science reports.
  5. Browse your current reports. One last trip down memory lane. It turns out I do not have the Neanderthal trait that makes people sneeze after eating dark chocolate. Thanks, 23AndMe, that’s definitely worth risking my genetic privacy to know!
  6. Consider downloading / printing any detailed reports you are interested in. They seldom contain any more personal data than is in the summary report but the surrounding context may be useful.
  7. Consider downloading Profile Data. This is a record of inputs you’ve given the site, like address changes and your self-reported phenotype. You have to request a download; it took two days for me. Mine was a few uninteresting tiny CSV files so I’m not sure it was worth the bother.
  8. Request account closure. That’s the final step, the deletion. The big red button is at the bottom of this page. They send you an email to confirm with some caveats.

Once I submitted the final deletion my account looked inaccessible the moment I tried to log in. Well I could log in, but it looked like an empty profile. The site didn’t promise any final notification when the deletion is completed, no idea whether I’ve simply been marked as status: deleted or they actually scrubbed my data. Presumably destroying biological samples takes some time.

As noted above, various copies of your genetic data will be kept for years and there’s nothing you can do to delete it. I believe these are all anonymized and won’t be easy to associate with your name, but that still seems pretty crappy to me.

I will give 23AndMe credit, nowhere in this process was there some retention process slowing me down. They never even asked why I was deleting! I appreciated that; it’s much harder to, say, cancel cable TV.

23AndMe’s deletion details

This is the disclosure in the email from 23AndMe about what deletion means. Note the third bullet item about “will retain your Genetic Information”; despite my request they are not actually deleting my data.

We received your request to permanently delete your 23andMe account and associated data. The following apply when you submit your deletion request:

  • If you chose to consent to 23andMe Research by agreeing to an applicable 23andMe Research consent document, any research involving your data that has already been performed or published prior to our receipt of your request will not be reversed, undone, or withdrawn.
  • Any samples for which you gave consent to be stored (biobanked) will be discarded.
  • 23andMe and the contracted genotyping laboratory will retain your Genetic Information, Date of Birth, and sex as required for compliance with legal obligations, pursuant to the federal Clinical Laboratory Improvement Amendments of 1988 and California laboratory regulations.
  • 23andMe will retain limited information related to your data deletion request, such as your email address and Account Deletion Request Identifier, as necessary to fulfill your request and for the establishment, exercise or defense of legal claims.

Once you confirm your request to delete all of your data, 23andMe will begin processing your request. This decision cannot be cancelled, undone, withdrawn, or reversed.

Better NVidia driver updates

~ nelsonminar ~ Leave a comment

NVidia makes excellent graphics cards with decent drivers.

They also make shitty marketing-ware bloated with crap you don’t want. I’m referring to GeForce Experience, the wrapper around the drivers that runs on Windows. It claims its primary job is updating your drivers. But somehow it needs 100s of megabytes to do that, has a bunch of stupid features in it you don’t want, and requires a login to an NVidia website account to work. Worse, the login requires a CAPTCHA. Right there on my Windows desktop. Stupidest damn thing. Of course the real reason for all this is building ad profiles and marketing and blah blah blah. See also: “NvTelemetry”.

One option for GeForce Experience is simply not to use it. You can still manually download drivers online and install them by hand. For now. But that’s a hassle.

A better option is TinyNvidiaUpdateChecker, a very minimal app for updates. It looks at your machine for the installed version of the drivers, looks online for newer versions, and will install an update for you. It runs super fast, requires no login, and just does its job without getting in the way.

The one drawback is it really is minimal. The default setup is to run once and display its UI in a console window (lol). There’s a guide for making it run at system startup instead, which boils down to “make a shortcut with the –quiet argument and make it run at startup”. It’d be nice if it had an installer and an optional system tray mode.

There’s also an advanced feature called the minimal installer. That breaks apart the NVidia update and only installs some pieces. The goal is to avoid GeForce Experience and NvTelemetry, which are laudable. But the docs makes it sound like you might also not get PhysX, HD Audio, etc. That doesn’t sound great to me, but maybe I misunderstand.

Updating nameservers in Ubuntu 19, Pi Hole edition

~ nelsonminar

For some reason, every time I reboot my Ubuntu 19 box had /etc/resolv.conf configured to point the name server at 127.0.0.1. There’s no nameserver running there, so it fails. I want it set to 192.168.0.1. Where is this coming from? Is it related to having had Pi Hole installed at one time? systemd? What?

tl;dr: there’s like 5 things in an Ubuntu 19 system that might be modifying your name server, and tracking it down is really terrible.

In the old days you edited /etc/resolv.conf and were done. (libc knows to read this file and use it when resolving names. Crazy, huh?) That works transiently but is undone if you reboot. If you look at the file there’s a dire notice

# Dynamic resolv.conf(5) file for glibc resolver(3) generated by resolvconf(8)
#     DO NOT EDIT THIS FILE BY HAND -- YOUR CHANGES WILL BE OVERWRITTEN
# 127.0.0.53 is the systemd-resolved stub resolver.
# run "systemd-resolve --status" to see details about the actual nameservers.

nameserver 127.0.0.1

OK, so systemd is a DNS resolver too now? Awesome. Don’t let the 127.0.0.53 surprise you; that’s basically another name for 127.0.0.1, localhost. Only they’re being sorta clever. But my system is broken; it’s not set to .53, it’s set to .1.

Then you go down the rabbit hole. That command systemd-resolve --status is giving you systemd’s idea of how to resolv names. That in turn seems to be configured by files in /etc/netplan, something you may have created if you configured static networking. Changing those will (presumably) alter the behavior of systemd’s DNS server.

But my problem is my /etc/resolv.conf was being regenerated at boot time to point to 127.0.0.1, not 127.0.0.53. systemd’s resolver is never involved. How to fix that? Second rabbit hole; there’s a system called resolvconf that might be overwriting it with information in /etc/resolvconf. Only what’s on my system isn’t the real (old) resolvconf, it’s actually systemd’s resolvectl running in a resolvconf-compatibility mode. I fiddled with it for awhile and I still don’t understand how this stuff works, but it seems to not be writing this entry.

I finally did a grep and found even though I’d uninstalled Pi Hole, it left behind a /etc/init.d/pihole-FTL script. Which is running resolvconf at boot time to re-set the nameserver to 127.0.0.1. This script shouldn’t still be on my system, and removing it should stop the clobbering. So I removed and rebooted.

Hah, joke’s on me! Every reboot, the 127.0.0.1 entry gets rewritten. Where is it coming from? I found it in /run/resolvconf/interface/enp3s0.dhcp. I’m not using DHCP, I have a static IP address. But that encouraged me to pay attention to /etc/dhcpcd.conf which, yes, is the actual source of the text overriding resolv.conf. I changed it from “static domain_name_servers=127.0.0.1” to 192.168.0.1 and name service works on reboot! Who knows why this matters.. dhcpcd is still running; it appears to be used to configure interfaces with static addresses even if DHCP isn’t involved.

But systemd still has one joke left on me. Now when I cat /etc/resolv.conf there are two name servers.

nameserver 192.168.0.1
nameserver 127.0.0.53

I have no idea why systemd decided to inject its .53 in there. It didn’t when the DHCP wrote 127.0.0.1 in there, but change that number to 192.168.0.1 and now systemd’s all in a hurry to put itself there. It may be coming from files I’m afraid to edit in /run. Anyway now half the queries go directly to my name server, half are looped through systemd first.

Fortunately the systemd loop seems to be working. I can’t dig @127.0.0.53 but no ordinary query ever hangs in a way consistent with a broken nameserver. systemd-resolve –status suggests that resolver is configured usefully (forwarding all requests to 192.168.0.1). So I’m just gonna leave it alone since it’s working, even if it’s not what I want.

The problem with systemd isn’t just that it’s a giant beast that tries to do everything. It’s that it’s also magic and poorly documented.

Linux watchdogs in 2019

~ nelsonminar

My home server is dying unexplainedly. Totally locks solid, nothing in the logs, very confusing. While I figure out what’s wrong (power supply?) I decided to go implement a watchdog to reboot the system if it fails.

This turns out to be hard in 2019, with Ubuntu 19. tl;dr there are two choices: systemd or the good ol watchdog daemon. See also: softdog.

systemd has watchdog support. Unlike many things in the systemd hydra, this one makes sense to me to integrate. It’s configured in /etc/systemd/system.conf. However it’s not well documented and relatively limited, so I decided not to use it. Even so my system has a PID 77 named [watchdogd] that I think is part of systemd. No idea what it’s doing, if anything.

watchdog is the old school Linux daemon that does watchdoggy things. Ubuntu 19 offers version 5.15-2, and if you install it by default it doesn’t do much. You have to configure it via /etc/watchdog.conf, ensure it loads at startup, and (maybe) install a kernel module to help it.

watchdog works by running a bunch of different tests. It’ll try pinging a network address, check if a file has been modified, see if the load average is too high or if there’s no RAM available. If not it’ll reboot the system. The shutdown is internal, it doesn’t fork any processes.

By default the Ubuntu watchdog.conf has no tests enabled. I think this means it does nothing at all. (It’s possible there’s some “if the system is totally dead then reboot” thing still hiding in there, but if so I don’t see it.) To be useful you want to configure various tests. I have it pinging my router; in theory my computer is still working even if the router is down, but in practice it seems more likely my server’s networking has died. (There’s a systemd upgrade bug.). I’m sure this will end up shooting me in the foot some day.

This is what a watchdog shutdown looks like in the syslog

Oct 13 18:57:31 ub watchdog[30919]: no response from ping (target: 192.168.3.1)
Oct 13 18:58:33 ub watchdog[30919]: message repeated 31 times: [ no response from ping (target: 192.168.3.1)]
Oct 13 18:58:33 ub watchdog[30919]: Retry timed-out at 62 seconds for 192.168.3.1
Oct 13 18:58:33 ub watchdog[30919]: shutting down the system because of error 101 = 'Network is unreachable'

BTW, this is the point to note that in theory if you screw up a watchdog config bad enough, the system might reboot itself so fast you can never get in and fix it without rebooting single-user mode at a console. Fortunately the default config is to reboot after 1-2 minutes of failure, giving you time to get in and fix anything dumb or disable watchdog entirely.

What happens if the machine is so locked up that the user space watchdog process can’t run at all? What will trigger a reboot then? Enter kernel support for watchdogs, a feature that goes back to 2002. The basic idea is if some process ever writes to a file named /dev/watchdog, if that file is not written to once a minute the kernel will reboot itself. The kernel’s own watch on itself is implemented at a low level with some sort of CPU timer. Serious systems have extra hardware for this kind of self monitoring, but this method should work reasonably well on a consumer PC unless the kernel itself or the whole CPU locks up.

However if you look on Ubuntu you don’t have a /dev/watchdog file. You have to install it. The simple way is to “modprobe softdog”. Getting this to happen at boot time is remarkably difficult because the module is blacklisted and systemd refuses to load it. The best workaround is to modify /etc/default/watchdog to load “softdog” as a module, fortunately they thought ahead on the need for this. Once you can do that you can enable the test in watchdog.conf.

Putting it all together, here’s how to enable a watchdog in an Ubuntu 19 system

  • apt install watchdog
  • edit /etc/default/watchdog to load the “softdog” module
  • edit /etc/watchdog.conf to enable the tests you want. I enabled ping and watchdog-device.
  • run “systemctl start watchdog” to enable it (or reboot)
  • check the syslog to see that watchdog is logging your active tests and looks reasonable

Epic Launcher: running copied games

~ nelsonminar

The Epic Launcher on Windows stores its games in C:\Program Files\Epic Games. However if you copy a game from one computer to another, the Epic Launcher on the new computer won’t recognize it or let you play it. Super annoying if you want to avoid re-downloading 60GB of stuff you already have a copy of. (Steam does not have this nuisance.)

The workaround for this is to first start installing the game on the new computer using the Epic Launcher. Let it download for a few seconds, then abort the install (don’t just pause it) from the launcher. Then go in to the folder and delete the game directory on the new computer. And then copy the files from the old computer to the new. Finally in the Epic Launcher click “Resume” and it will verify the files rather than re-download and let you play the game.

This solution is described in How to Move Fortnite to Another Folder, Drive, or PC. The problem is also discussed in EpicGamesLauncher – Recognising already installed files which includes some hints about a C:\ProgramData directory that may contain the hidden state that’s missing if you just copy the program files. I didn’t pursue that avenue for making things work but it’s a good idea.

Upgrading systemd kills Ubuntu networking

~ nelsonminar

Two or three times this year I’ve had a bug in a relatively fresh Ubuntu install where upgrading systemd kills the networking. I’ve got no idea what’s wrong and no physical console on the machine to inspect it. It feels like something simple like the upgrade script shut down networking but then didn’t bring it back again, but of course it could be anything.

So far this is happening only on my 18.10/19.04 box in San Francisco. Fortunately I have physical access so at least I can reboot it (usually). I haven’t seen it happen on my 18.04 box in a datacenter.

Ubuntu has a couple of recent bugs about this. Bug 1803391 boils down to a one time bug in an upgrade script in systemd. Bug 1782709 is open and confirmed with many reports and no clear idea what might be wrong. It looks like the kind of bug report that stays open for 4 years until someone closes it as irrelevant :-(

I do not want to learn so much about the guts of systemd and dpkg to diagnose this. systemd is particularly gnarly to debug since it does so much. Among other things it’s also capturing system logs, so there’s a risk whatever is being logged about the error is disappearing during the upgrade. Or not, who knows?

Again, I’ve made my peace with systemd, but this bug is a great example of the danger of centralizing so many functions into one component.

Edit I’ve placed a hold on systemd upgrades for now. Never upgrading systemd is not a solution, but this will serve as a reminder for me to only do the upgrade if I’m able to access the physical machine.