Upgrading systemd kills Ubuntu networking

Two or three times this year I’ve had a bug in a relatively fresh Ubuntu install where upgrading systemd kills the networking. I’ve got no idea what’s wrong and no physical console on the machine to inspect it. It feels like something simple like the upgrade script shut down networking but then didn’t bring it back again, but of course it could be anything.

So far this is happening only on my 18.10/19.04 box in San Francisco. Fortunately I have physical access so at least I can reboot it (usually). I haven’t seen it happen on my 18.04 box in a datacenter.

Ubuntu has a couple of recent bugs about this. Bug 1803391 boils down to a one time bug in an upgrade script in systemd. Bug 1782709 is open and confirmed with many reports and no clear idea what might be wrong. It looks like the kind of bug report that stays open for 4 years until someone closes it as irrelevant :-(

I do not want to learn so much about the guts of systemd and dpkg to diagnose this. systemd is particularly gnarly to debug since it does so much. Among other things it’s also capturing system logs, so there’s a risk whatever is being logged about the error is disappearing during the upgrade. Or not, who knows?

Again, I’ve made my peace with systemd, but this bug is a great example of the danger of centralizing so many functions into one component.

Edit I’ve placed a hold on systemd upgrades for now. Never upgrading systemd is not a solution, but this will serve as a reminder for me to only do the upgrade if I’m able to access the physical machine.