After 10 years of fiddling with remote servers, I finally killed one so I couldn’t access it on reboot. The system came up fine but not the network. The nice remote admin logged in and found that the Ethernet device was now named /dev/p4p1 and the config files were expecting /dev/eth0. He updated the config and it works, for now. I’m scared to reboot again since I don’t understand what caused it to change in the first place.
tl;dr: installing biosdevname causes devices to be renamed at boot.
Summary: I installed biosdevname and I think that caused my ethernet device to get a different name after reboot. A fresh Ubuntu 14.04 system uses biosdevname to give Ethernet devices names like /dev/p4p1. These names are based on the physical location of the adapter on the PCI bus. An old Ubuntu system upgraded to 14.04 may not install biosdevname and instead use older working names like /dev/eth0 that are configured by MAC address in /etc/udev/rules.d/70-persistent-net.rules.
Some detailed notes on what may have broken it:
- The system started as Ubuntu 10.10, I did two do-release-upgrades to bring it to 14.04. After installing 14.04 it booted correctly at least once, configured as /dev/eth0 for ethernet. Now it wants /dev/p4p1
- After the working boot I installed a few packages that I regret installing. Specifically biosdevname libsystemd-daemon0 systemd-shim systemd-services discover linux-image-3.13.0-36-generic.
- I removed the 3-13.0-36 kernel and then had to do some awkward stuff to fix grub to use the linux-image-3.13.0-37-generic I already had. I think I got that right.
- I don’t think the systemd packages would cause any problems, but I don’t know for sure. I don’t think I’m running systemd as a whole.
- I’m not sure what discover does. My hope is it’s an install-time thing only.
- biosdevname’s whole job is to give names to devices, so it seems a likely troublemaker. One theory is that the mere presence of it being installed alters the boot methods of Ubuntu in some way. But that’s a total guess. This forum post gives some credence to things.
- /proc/cmdline says
BOOT_IMAGE=/vmlinuz-3.13.0-37-generic root=UUID=e7833d2a-17a4-4598-95e2-65d354589db9 ro quiet splash
Some notes on how it’s supposed to work:
- /etc/udev/rules.d/70-persistent-net.rules is supposed to give a persistent name to the device. Mine seems to be saying “call this eth0”. That may be a legacy from the 10.10 or 12.04 history of the system, and was apparently working until yesterday
- I have another system that began life as ubuntu 14.04. Its ethernet is /dev/p2p1. It has no files in /etc/udev/rules.d
- My system appears to be booting with Upstart, not systemd. (PID 1 is init, not systemd). Details on systemd and Ubuntu
- These Arch Linux notes on the upgrade to the new persistent naming scheme are interesting. Also these systemd docs. But since I’m not using systemd, they are probably not relevant.
- This GenToo doc on udev upgrades is also useful.
- Here’s an Ubuntu bug saying the 14.04 docs have not been updated to reflect that 70-persistent-net.rules is no longer used. Also here
- biosdevname installs /lib/udev/rules.d/71-biosdevname.rules which may override that 70-persistent-net.rules thing.
- Apparently the kernel has a command line parameter biosdevname=0 to disable renaming. (net.ifnames=0 in Fedora). 71-biosdevname.rules also can be edited to do that.
After 2+ hours of reading I convinced myself it was safe to reboot. It was! I removed the /etc/udev/rules.d/70-persistent-net.rules file that was effectively being ignored after biosdevname was installed.
I filed a bug: https://bugs.launchpad.net/ubuntu/+source/biosdevname/+bug/1382600
Related: bless Arch Linux. I’ve never run it, but their incredibly detailed Wiki documentation on how Linux systems work is so useful. Sometimes you have to go to the low level tools manufacturers to understand how something works.