Score:0

Google Public DNS always returning NXDOMAIN for certain SLDs

fj flag

Problem: Google's Public DNS returns NXDOMAIN for certain SLDs.

Proof of problem:

dig vpn.example.com @8.8.8.8

   ; \<\<\>\> DiG 9.11.5-P4-5.1+deb10u8-Debian \<\<\>\> vpn.example.com @8.8.8.8
    ; global options: +cmd
    ; Got answer:
    ;; -\>\>HEADER\<\<- opcode: QUERY, status: NXDOMAIN, id: 8324
    ;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 1
    
    ; OPT PSEUDOSECTION:
    ; EDNS: version: 0, flags:; udp: 512
    ;; QUESTION SECTION:
    ;vpn.example.com.        IN    A
    
    ;; AUTHORITY SECTION:
    example.com.    1800    IN    SOA    ns1.example.com. root.example.com. 1675851775 28800 7200 604800 86400
    
    ;; Query time: 134 msec
    ;; SERVER: 8.8.8.8#53(8.8.8.8)
    ;; WHEN: Thu Feb 09 09:52:06 EET 2023
    ;; MSG SIZE  rcvd: 93

as you can see the query status is NXDOMAIN. Asking the authoritative DNS server listed in the AUTHORITY section, however, points to a correct answer:

dig vpn.example.com @ns1.example.com

; \<\<\>\> DiG 9.11.5-P4-5.1+deb10u8-Debian \<\<\>\> vpn.example.com @ns1.example.com**
;; global options: +cmd
;; Got answer:
;; -\>\>HEADER\<\<- opcode: QUERY, status: **NOERROR**, id: 37073
;; flags: qr aa rd; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1
;; WARNING: recursion requested but not available

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 600
;; QUESTION SECTION:
;vpn.example.com.        IN    A

;; ANSWER SECTION:
vpn.example.com.    3600    IN    A    XXX.XX.X.XXX

;; Query time: 128 msec
;; SERVER: XXX.XX.XX.XXX#53(XXX.XX.XX.XXX)
;; WHEN: Thu Feb 09 09:58:05 EET 2023
;; MSG SIZE  rcvd: 64

Other public DNS servers (opendns, cloudflare, etc) all resolve the SLD.

The authoritative DNS server (all 4 NS records) is consistent in the responses:

for i in $(seq 1 30)
do
query=$(dig +short us1.vpn.example.com @ns1.example.com)
if [[ -z "$query" ]]
then echo "NO ANSWER"
else
echo "ANSWER"
fi
sleep 2
done | sort | uniq -c

30 ANSWER

I tried the following in two different tabs:

TAB1 client side //

while true; do dig +short vpn.example.com @8.8.8.8; sleep 1; done

TAB2 dns server side //

tcpdump -vvvvv -w /tmp/dns.pcap udp and port 53

TAB2 dns server side //

tcpdump -n -t -r /tmp/dns.pcap | grep vpn

and tried to discern any IPs of the subnets listed here : https://developers.google.com/speed/public-dns/faq#locations_of_ip_address_ranges_google_public_dns_uses_to_send_queries

and found none for that specific host. How can I further debug this? Thanks for any forthcoming suggestions!

cn flag
Fwiw, I'm unable to reproduce the problem. Does it work for you as well now? Either way it does look quite odd how you had a negative response with the same serial that all the authoritative servers still serve (and all the authoritative servers, at least now, seem to serve the positive answer that you expected, as does 8.8.8.8 for me).
vidarlo avatar
ar flag
First, redacting things doesn't change history, so everything is still visible. Second, please use *either* a domain under your control, or one of the example's, e.g. `example.com` or `example.org` to make it clear that it's actually an example.
Score:2
fr flag

The authoritative DNS server (all 4 NS records) is consistent in the responses:

No, it is not. The server ns1.exmaple.com occasionally flips between returning the A record and returning NXDOMAIN for this name. (It seems that making a query via TCP, using dig +vc, is a reliable way to make it start responding with NXDOMAIN over both protocols.)

$ dig vpn.example.com @ns1.example.com
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 24350
;; flags: qr aa rd; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1
;; WARNING: recursion requested but not available

$ dig +vc vpn.example.com @ns1.example.com
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 63787
;; flags: qr aa rd; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 1
;; WARNING: recursion requested but not available

$ dig vpn.example.com @ns1.example.com
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 10815
;; flags: qr aa rd; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 1
;; WARNING: recursion requested but not available

In this situation it is normal to have some cache inconsistency (as seen by Tomek), as Google DNS is not just anycast globally – each location has its own multiple resolvers with independent caches behind a load balancer, so even if you're seeing the same NSID you're still getting replies from a different backend server every time. (As a side note, don't forget the cache flush page.)

It is possible that ns1.example.com is similarly handled by more than one server behind a load balancer, some of which give the correct result and some do not.

Krasimir Velkov avatar
fj flag
good catch about querying over TCP
Score:0
fj flag

This turned out to be a PIPEBackend issue with the powerdns authoritative server. Thanks to all involved!

Score:0
fr flag

I can intermittently reporoduce it on all Google DNS addresses:

triss:~> dig vpn.obfuscated.com @2001:4860:4860::8844 +short
XXX.XX.X.XXX
triss:~> dig vpn.obfuscated.com @2001:4860:4860::8844 +short
XXX.XX.X.XXX
triss:~> dig vpn.obfuscated.com @2001:4860:4860::8844 +short
XXX.XX.X.XXX
triss:~> dig vpn.obfuscated.com @2001:4860:4860::8844 +short
XXX.XX.X.XXX
triss:~> dig vpn.obfuscated.com @2001:4860:4860::8844 +short
triss:~> dig vpn.obfuscated.com @2001:4860:4860::8844 +short
triss:~> dig vpn.obfuscated.com @2001:4860:4860::8844 +short
XXX.XX.X.XXX
triss:~> dig vpn.obfuscated.com @2001:4860:4860::8844 +short
triss:~> dig vpn.obfuscated.com @2001:4860:4860::8844 +short
XXX.XX.X.XXX
triss:~> dig vpn.obfuscated.com @2001:4860:4860::8844 +short
XXX.XX.X.XXX

Looks like some cache inconsistency at Google.

Krasimir Velkov avatar
fj flag
I have readily available locations where I can check the dns.google NSID to see if this is up to different anycast resolvers playing foul: ```dig @dns.google. +nsid | grep NSID ; NSID: 67 70 64 6e 73 2d 6f 72 64 ("gpdns-ord") ; dig @dns.google. +nsid | grep NSID ; NSID: 67 70 64 6e 73 2d 73 6f 66 ("gpdns-sof") ; dig @dns.google. +nsid | grep NSID ; NSID: 67 70 64 6e 73 2d 6c 68 72 ("gpdns-lhr")``` however all of these report the same thing - NXDOMAIN
fr flag
The content has been obfuscated after resolution.
I sit in a Tesla and translated this thread with Ai:

mangohost

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.