Wanted to document this learning as more than some tweets.
The amazing Julia Evans launched Mess with DNS, a little sandbox where you can register some domain names and then see what queries their DNS server of record gets. Great tool for learning.
I learned something! When I did a single A query at home on Sonic.net using their DHCP servers, Mess With DNS sees six queries all at once
That seems like an awful lot. There’s several mysteries here, but the one I focussed on was that query for
_.lily6.messwithdns.com. Underscores are generally understood to not be valid in domain names, although the underlying DNS protocols do seem to allow them in some places. But why would a query for a normal subdomain also produce a query for _?
A simple way to reproduce this is
$ dig @220.127.116.11 nelson.lily6.messwithdns.com a
That’s a different Sonic resolver than I use. It only results in two queries, not six, but it still generates that underscore query.
First, a reply from a product manager at Cloudflare who very kindly looked in his logs for similar queries and reported.
Over the last ~5 minutes I see ~12 queries per second on average (globally) to 18.104.22.168 for _.www.google.com and ~3 QPS for _.www.cloudflare.com – which is effectively noise.
So that’s interesting. Those queries do exist but they are not very common.
As for what causes them, the Hacker News discussion came through. Several folks all pointed out this is something Bind9 does with the “QNAME optimization” option turned on. One Sonic engineer even explicitly confirmed that was the DNS server they use and the source of the queries.
So what do they mean? QNAME Minimization is a privacy thing in recursive resolvers. If you’re querying
foo.bar.example.com at some point you end up asking the .com root server for information. But there’s no need for them to see the whole domain name, so you just send the minimum you need (
example.com, I believe.)
qname minimization, even in relaxed mode, can fail on some very broken domains. In relaxed mode, instead of asking for “foo.bar NS” ask for “_.foo.bar A” to either get a delegation or NXDOMAIN. It will require more queries than regular mode for proper NXDOMAINs.
So that seems to be the answer. It’s not great that this workaround for “some very broken domains” ends up doubling the name server traffic for the rest of the Internet, but here we are.
For completeness, several people pointed to RFC 2782 or RFC 8552. (It’s one of first Google search results for “underscore dns”.) That RFC is about using _ to mangle some other tokens to not collide with domain names; they are used when querying things like _tcp or _udp. I’m pretty sure that’s not what’s going on here.
I’m grateful to have gotten replies from someone at CloudFlare, my ISP, and the author of the QNAME Minimization RFC. It’s nice to be able to get an answer from actual experts.