For some reason, every time I reboot my Ubuntu 19 box had /etc/resolv.conf configured to point the name server at 127.0.0.1. There’s no nameserver running there, so it fails. I want it set to 192.168.0.1. Where is this coming from? Is it related to having had Pi Hole installed at one time? systemd? What?
tl;dr: there’s like 5 things in an Ubuntu 19 system that might be modifying your name server, and tracking it down is really terrible.
In the old days you edited /etc/resolv.conf and were done. (libc knows to read this file and use it when resolving names. Crazy, huh?) That works transiently but is undone if you reboot. If you look at the file there’s a dire notice
# Dynamic resolv.conf(5) file for glibc resolver(3) generated by resolvconf(8) # DO NOT EDIT THIS FILE BY HAND -- YOUR CHANGES WILL BE OVERWRITTEN # 127.0.0.53 is the systemd-resolved stub resolver. # run "systemd-resolve --status" to see details about the actual nameservers. nameserver 127.0.0.1
OK, so systemd is a DNS resolver too now? Awesome. Don’t let the 127.0.0.53 surprise you; that’s basically another name for 127.0.0.1, localhost. Only they’re being sorta clever. But my system is broken; it’s not set to .53, it’s set to .1.
Then you go down the rabbit hole. That command
systemd-resolve --status is giving you systemd’s idea of how to resolv names. That in turn seems to be configured by files in
/etc/netplan, something you may have created if you configured static networking. Changing those will (presumably) alter the behavior of systemd’s DNS server.
But my problem is my
/etc/resolv.conf was being regenerated at boot time to point to 127.0.0.1, not 127.0.0.53. systemd’s resolver is never involved. How to fix that? Second rabbit hole; there’s a system called resolvconf that might be overwriting it with information in /etc/resolvconf. Only what’s on my system isn’t the real (old) resolvconf, it’s actually systemd’s resolvectl running in a resolvconf-compatibility mode. I fiddled with it for awhile and I still don’t understand how this stuff works, but it seems to not be writing this entry.
I finally did a grep and found even though I’d uninstalled Pi Hole, it left behind a
/etc/init.d/pihole-FTL script. Which is running resolvconf at boot time to re-set the nameserver to 127.0.0.1. This script shouldn’t still be on my system, and removing it should stop the clobbering. So I removed and rebooted.
Hah, joke’s on me! Every reboot, the 127.0.0.1 entry gets rewritten. Where is it coming from? I found it in /run/resolvconf/interface/enp3s0.dhcp. I’m not using DHCP, I have a static IP address. But that encouraged me to pay attention to /etc/dhcpcd.conf which, yes, is the actual source of the text overriding resolv.conf. I changed it from “static domain_name_servers=127.0.0.1” to 192.168.0.1 and name service works on reboot! Who knows why this matters.. dhcpcd is still running; it appears to be used to configure interfaces with static addresses even if DHCP isn’t involved.
But systemd still has one joke left on me. Now when I cat /etc/resolv.conf there are two name servers.
nameserver 192.168.0.1 nameserver 127.0.0.53
I have no idea why systemd decided to inject its .53 in there. It didn’t when the DHCP wrote 127.0.0.1 in there, but change that number to 192.168.0.1 and now systemd’s all in a hurry to put itself there. It may be coming from files I’m afraid to edit in /run. Anyway now half the queries go directly to my name server, half are looped through systemd first.
Fortunately the systemd loop seems to be working. I can’t dig @127.0.0.53 but no ordinary query ever hangs in a way consistent with a broken nameserver. systemd-resolve –status suggests that resolver is configured usefully (forwarding all requests to 192.168.0.1). So I’m just gonna leave it alone since it’s working, even if it’s not what I want.
The problem with systemd isn’t just that it’s a giant beast that tries to do everything. It’s that it’s also magic and poorly documented.