Ubuntu 20.04 resolved waits for refused IPV6 result with dnsmasq

TL;DR

My DNS queries are slow because systemd-resolved queries the DNS server on IPv4 successfully but repeatedly queries on IPv6 after the DNS server responds with a REFUSED. Is this a resolved config issued? A dnsmasq issue? or a bug?

I have a stock ubuntu 20.04 install connected to dnsmasq running on an air gapped appliance (ubiquity edgerouter) with top level domain 'bar' (air gapped remember but it doesn't seem, to matter what this is--ie .com etc). DNS requests resolve quickly via Mac and Windows clients. For ubuntu, DNS queries take about 5 seconds to resolve. Searching around, Ubuntu DNS issues are discussed in plenty of places but the underlying behavior I'm seeing has not been to my knowledge.

Top level issue: if I ping a machine using ubuntu, it takes about 5 seconds for the responses to come back:

$ ping foo
# about 5 seconds go by
PING foo.bar (10.2.1.132) 56(84) bytes of data.
64 bytes from foo.bar (10.2.1.132): icmp_seq=1 ttl=63 time=1.10 ms
64 bytes from foo.bar (10.2.1.132): icmp_seq=2 ttl=63 time=1.02 ms
...

Which seems like an obvious timeout issue. Interestingly it does resolve though. Running resolvectl status gives me the standard result including that DNS is being provided by the dnsmasq-issued DHCP lease.

$ resolvectl status

...
lots of stuff
...

Link 3 (enp0s25)
      Current Scopes: DNS     
DefaultRoute setting: yes     
       LLMNR setting: yes     
MulticastDNS setting: no      
  DNSOverTLS setting: no      
      DNSSEC setting: no      
    DNSSEC supported: no      
         DNS Servers: 10.1.1.1
          DNS Domain: ~.      
                      bar

Stopping resolved and running in debug mode appears to show the issue. The ubuntu machine queries the DNS server via IPv4 and it receives an answer immediately. It then repeatedly queries via IPv6 which the DNS server responds with REFUSED until it times out.

$ sudo systemctl stop systemd-resolved
$ sudo SYSTEMD_LOG_LEVEL=debug /lib/systemd/systemd-resolved
...
lots of startup logging
...
Got DNS stub UDP query packet for id 40457
Looking up RR for foo.bar IN A.
Switching to DNS server 10.1.1.1 for interface enp0s25.
Cache miss for foo.bar IN A
Transaction 42924 for <foo.bar IN A> scope dns on enp0s25/*.
Using feature level UDP+EDNS0 for transaction 42924.
Using DNS server 10.1.1.1 for transaction 42924.
Sending query packet with id 42924.
Processing query...
Got DNS stub UDP query packet for id 59119
Looking up RR for foo.bar IN AAAA.
Cache miss for foo.bar IN AAAA
Transaction 21734 for <foo.bar IN AAAA> scope dns on enp0s25/*.
Using feature level UDP+EDNS0 for transaction 21734.
Using DNS server 10.1.1.1 for transaction 21734.
Sending query packet with id 21734.
Processing query...
Processing incoming packet on transaction 42924 (rcode=SUCCESS).
Verified we get a response at feature level UDP+EDNS0 from DNS server 10.1.1.1.
Added positive unauthenticated cache entry for foo.bar IN A 7200s on enp0s25/INET/10.1.1.1
Transaction 42924 for <foo.bar IN A> on scope dns on enp0s25/* now complete with <success> from network (unsigned).
Sending response packet with id 40457 on interface 1/AF_INET.
Freeing transaction 42924.
Processing incoming packet on transaction 21734 (rcode=REFUSED).
Server returned REFUSED, switching servers, and retrying.
Retrying transaction 21734.
Cache miss for foo.bar IN AAAA
Transaction 21734 for <foo.bar IN AAAA> scope dns on enp0s25/*.
Using feature level UDP+EDNS0 for transaction 21734.
Sending query packet with id 21734.
Processing incoming packet on transaction 21734 (rcode=REFUSED).
Server returned REFUSED, switching servers, and retrying.
...
repeats for ~5 seconds
...
Transaction 44267 for <foo.bar IN AAAA> on scope dns on enp0s25/* now complete with <attempts-max-reached> from network (unsigned).
Freeing transaction 44267.

Also suggesting that this is the issue:

$ nslookup foo
Server:     127.0.0.53
Address:    127.0.0.53#53

Non-authoritative answer:
Name:   foo.bar
Address: 10.2.1.132
...
about 5 seconds go by
...
;; connection timed out; no servers could be reached

# this returns immediately
$ nslookup -query=A foo
Server:     127.0.0.53
Address:    127.0.0.53#53

Non-authoritative answer:
Name:   foo.bar
Address: 10.2.1.132

# while this times out in about 5 sec.
$ nslookup -query=AAAA foo
;; connection timed out; no servers could be reached

Questions:

1. If the DNS server was providing a good IPv4 answer, why would the ubuntu machine wait while repeatedly trying unsuccessfully to obtain a IPv6 answer?

2. Is this an ubuntu configuration issue or a dnsmasq issue?

Thanks in advance.

I have encountered exactly the same issue while configuring systemd-resolved and dnsmasq.

For your question 1, I am not able to find out why systemd-resolved query for AAAA record repeatedly while there is no reply from dnsmasq.

For question 2, the dnsmasq is not correctly configured to add a record for foo.bar. You only add an A record for this domain name, while systemd-resolved query for both A and AAAA record at the same time. dnsmasq has no idea in AAAA record for this domain thus it forward this query to upsteam server. I am not clear about what will happen next but dnsmasq does not give a expected answer.

You did not mention the way you configure dnsmasq. In my test, any config below will work as expected. Note there is a difference between 1 and 2, 3.

1. address=/foo.bar/10.2.1.132/
2. local=/bar/, addn-hosts=/tmp/hosts and add 10.2.1.132 foo.bar to /tmp/hosts
3. host-record=foo.bar,10.2.1.132 and local=/bar/

From dnsmasq man page for option --local

Also permitted is a -S flag which gives a domain but no IP address; this tells dnsmasq that a domain is local and it may answer queries from /etc/hosts or DHCP but should never forward queries on that domain to any upstream servers. --local is a synonym for --server to make configuration files clearer in this case.