Score:0

Ubuntu 20.04 resolved waits for refused IPV6 result with dnsmasq

cn flag

TL;DR

My DNS queries are slow because systemd-resolved queries the DNS server on IPv4 successfully but repeatedly queries on IPv6 after the DNS server responds with a REFUSED. Is this a resolved config issued? A dnsmasq issue? or a bug?

I have a stock ubuntu 20.04 install connected to dnsmasq running on an air gapped appliance (ubiquity edgerouter) with top level domain 'bar' (air gapped remember but it doesn't seem, to matter what this is--ie .com etc). DNS requests resolve quickly via Mac and Windows clients. For ubuntu, DNS queries take about 5 seconds to resolve. Searching around, Ubuntu DNS issues are discussed in plenty of places but the underlying behavior I'm seeing has not been to my knowledge.

Top level issue: if I ping a machine using ubuntu, it takes about 5 seconds for the responses to come back:

$ ping foo
# about 5 seconds go by
PING foo.bar (10.2.1.132) 56(84) bytes of data.
64 bytes from foo.bar (10.2.1.132): icmp_seq=1 ttl=63 time=1.10 ms
64 bytes from foo.bar (10.2.1.132): icmp_seq=2 ttl=63 time=1.02 ms
...

Which seems like an obvious timeout issue. Interestingly it does resolve though. Running resolvectl status gives me the standard result including that DNS is being provided by the dnsmasq-issued DHCP lease.

$ resolvectl status

...
lots of stuff
...

Link 3 (enp0s25)
      Current Scopes: DNS     
DefaultRoute setting: yes     
       LLMNR setting: yes     
MulticastDNS setting: no      
  DNSOverTLS setting: no      
      DNSSEC setting: no      
    DNSSEC supported: no      
         DNS Servers: 10.1.1.1
          DNS Domain: ~.      
                      bar

Stopping resolved and running in debug mode appears to show the issue. The ubuntu machine queries the DNS server via IPv4 and it receives an answer immediately. It then repeatedly queries via IPv6 which the DNS server responds with REFUSED until it times out.

$ sudo systemctl stop systemd-resolved
$ sudo SYSTEMD_LOG_LEVEL=debug /lib/systemd/systemd-resolved
...
lots of startup logging
...
Got DNS stub UDP query packet for id 40457
Looking up RR for foo.bar IN A.
Switching to DNS server 10.1.1.1 for interface enp0s25.
Cache miss for foo.bar IN A
Transaction 42924 for <foo.bar IN A> scope dns on enp0s25/*.
Using feature level UDP+EDNS0 for transaction 42924.
Using DNS server 10.1.1.1 for transaction 42924.
Sending query packet with id 42924.
Processing query...
Got DNS stub UDP query packet for id 59119
Looking up RR for foo.bar IN AAAA.
Cache miss for foo.bar IN AAAA
Transaction 21734 for <foo.bar IN AAAA> scope dns on enp0s25/*.
Using feature level UDP+EDNS0 for transaction 21734.
Using DNS server 10.1.1.1 for transaction 21734.
Sending query packet with id 21734.
Processing query...
Processing incoming packet on transaction 42924 (rcode=SUCCESS).
Verified we get a response at feature level UDP+EDNS0 from DNS server 10.1.1.1.
Added positive unauthenticated cache entry for foo.bar IN A 7200s on enp0s25/INET/10.1.1.1
Transaction 42924 for <foo.bar IN A> on scope dns on enp0s25/* now complete with <success> from network (unsigned).
Sending response packet with id 40457 on interface 1/AF_INET.
Freeing transaction 42924.
Processing incoming packet on transaction 21734 (rcode=REFUSED).
Server returned REFUSED, switching servers, and retrying.
Retrying transaction 21734.
Cache miss for foo.bar IN AAAA
Transaction 21734 for <foo.bar IN AAAA> scope dns on enp0s25/*.
Using feature level UDP+EDNS0 for transaction 21734.
Sending query packet with id 21734.
Processing incoming packet on transaction 21734 (rcode=REFUSED).
Server returned REFUSED, switching servers, and retrying.
...
repeats for ~5 seconds
...
Transaction 44267 for <foo.bar IN AAAA> on scope dns on enp0s25/* now complete with <attempts-max-reached> from network (unsigned).
Freeing transaction 44267.

Also suggesting that this is the issue:

$ nslookup foo
Server:     127.0.0.53
Address:    127.0.0.53#53

Non-authoritative answer:
Name:   foo.bar
Address: 10.2.1.132
...
about 5 seconds go by
...
;; connection timed out; no servers could be reached

# this returns immediately
$ nslookup -query=A foo
Server:     127.0.0.53
Address:    127.0.0.53#53

Non-authoritative answer:
Name:   foo.bar
Address: 10.2.1.132

# while this times out in about 5 sec.
$ nslookup -query=AAAA foo
;; connection timed out; no servers could be reached

Questions:

  1. If the DNS server was providing a good IPv4 answer, why would the ubuntu machine wait while repeatedly trying unsuccessfully to obtain a IPv6 answer?

  2. Is this an ubuntu configuration issue or a dnsmasq issue?

Thanks in advance.

tegtmeye avatar
cn flag
BTW, I found a someone posting a similar issue to the dnsmasq mailing list: https://lists.thekelleys.org.uk/pipermail/dnsmasq-discuss/2019q2/013094.html
tegtmeye avatar
cn flag
Some additional content that makes me think there is an issue with both resolved and dsnmasq. The fact that dnsmasq is returning REFUSED appears to be an error (see RFC 4074 "Common Misbehavior Against DNS Queries for IPv6 Addresses"). If it doesn't know what the IPv6 address is, its should return an empty reply as per the RFC but maybe I'm misinterpreting. That said, resolved shouldn't repeatedly bang on the DNS server after it refused to answer you. See also: https://lists.isc.org/pipermail/bind-users/2018-April/100056.html https://bugs.centos.org/view.php?id=16317
Score:0
jp flag

I have encountered exactly the same issue while configuring systemd-resolved and dnsmasq.

For your question 1, I am not able to find out why systemd-resolved query for AAAA record repeatedly while there is no reply from dnsmasq.

For question 2, the dnsmasq is not correctly configured to add a record for foo.bar. You only add an A record for this domain name, while systemd-resolved query for both A and AAAA record at the same time. dnsmasq has no idea in AAAA record for this domain thus it forward this query to upsteam server. I am not clear about what will happen next but dnsmasq does not give a expected answer.

You did not mention the way you configure dnsmasq. In my test, any config below will work as expected. Note there is a difference between 1 and 2, 3.

  1. address=/foo.bar/10.2.1.132/
  2. local=/bar/, addn-hosts=/tmp/hosts and add 10.2.1.132 foo.bar to /tmp/hosts
  3. host-record=foo.bar,10.2.1.132 and local=/bar/

From dnsmasq man page for option --local

Also permitted is a -S flag which gives a domain but no IP address; this tells dnsmasq that a domain is local and it may answer queries from /etc/hosts or DHCP but should never forward queries on that domain to any upstream servers. --local is a synonym for --server to make configuration files clearer in this case.

mangohost

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.