Is random response from DNS authoritative server normal

We are trying to setup Lets Encrypt certificate issuance using cert-manager and dns01 solver. We are using Dreamhost as our DNS provider and we created glue component that bridges between RFC-2136 cert-manager and Dreamhost API.

We are experiencing issue that although the required (TXT) records are being added the authoritative DNS server are returning them at random - with two requests one after another one returning answer and other returning nothing. This causes cert-manager to wait unpredictable amount of time until by luck it is able confirm presence of TXT record and in some cases causes Lets Encrypt domain check to fail.

Is this something to be expected from DNS authoritative server? Or is that something we should raise with our DNS provider?

Bellow is example of such situation (for now the sepcific domain is retracted). As you can see

dig @ns1.dreamhost.com. _acme-challenge.keycloak.tenant-a.k8s-dev.redacted.redacted. TXT 

; <<>> DiG 9.16.15-Ubuntu <<>> @ns1.dreamhost.com. _acme-challenge.keycloak.tenant-a.k8s-dev.redacted.redacted. TXT
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 38336
;; flags: qr aa rd; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 1
;; WARNING: recursion requested but not available

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
;; QUESTION SECTION:
;_acme-challenge.keycloak.tenant-a.k8s-dev.redacted.redacted. IN    TXT

;; AUTHORITY SECTION:
redacted.       300 IN  SOA ns1.dreamhost.com. hostmaster.dreamhost.com. 2022060209 18661 600 1814400 300

;; Query time: 20 msec
;; SERVER: 162.159.26.14#53(162.159.26.14)
;; WHEN: czw cze 02 19:40:17 CEST 2022
;; MSG SIZE  rcvd: 152

dig @ns1.dreamhost.com. _acme-challenge.keycloak.tenant-a.k8s-dev.redacted.redacted. TXT 

; <<>> DiG 9.16.15-Ubuntu <<>> @ns1.dreamhost.com. _acme-challenge.keycloak.tenant-a.k8s-dev.redacted.redacted. TXT
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 41439
;; flags: qr aa rd; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1
;; WARNING: recursion requested but not available

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
;; QUESTION SECTION:
;_acme-challenge.keycloak.tenant-a.k8s-dev.redacted.redacted. IN    TXT

;; ANSWER SECTION:
_acme-challenge.keycloak.tenant-a.k8s-dev.redacted.redacted. 300 IN TXT "Uyr1nHC1CRWQcmOWDvObc4RMd-mNhKaE9bbNZTf3L2k"

;; Query time: 16 msec
;; SERVER: 162.159.26.14#53(162.159.26.14)
;; WHEN: czw cze 02 19:40:18 CEST 2022
;; MSG SIZE  rcvd: 144

As You can see

• Timestamps of responses are lmost the same
• One response contains TXT records, other do not
• The same dns server (162.159.26.14 - ns1.dreamhost.com) is responding and I believe this is authoritative server

The same dns server (162.159.26.14 - ns1.dreamhost.com) is responding and I believe this is authoritative server

Just like with web servers, a single IP address could be hiding multiple DNS servers answering your requests. Even queries from the same location could be distributed to multiple nodes through a load-balancer or a multipath route. (Sometimes the "ns1" and "ns2" are entirely for show, with both leading to the same pool of servers.)

Yes, ideally all authoritative servers in the cluster should know the exact same data, but depending on how they're implemented, the cluster could have some servers temporarily out-of-sync with Dreamhost's authoritative database for various reasons. For example (entirely hypothetical – I don't know how Dreamhost's systems work), "reload" requests might be spread in time to reduce load on the database.

after I checked from different country (using cloud VPS) I got different pattern of answers

On a broader scale, when querying from different locations, BGP anycast could lead you to entirely different clusters. A good example is either public DNS resolvers or the root servers – there are many instances of 1.1.1.1 around the world, and there are many instances of "f.root-servers.net". If Dreamhost uses anycast to host "ns1" at multiple physical locations (which they probably do, for reduced latency), then it is even more likely that they will be out-of-sync for a short period of time after you make changes. (That's one area where "DNS propagation" isn't a lie.)

Many DNS servers support the special hostname.bind and/or id.server queries to which they reply using their individual names. Try this from different locations:

dig +short @ns1.dreamhost.com hostname.bind. chaos txt

But overall, none of the above really changes things – your problem is not much different from having actually separate servers and keeping them in sync. For example, even if you just had ordinary ns1/ns2/ns3 servers using traditional DNS AXFR replication, whenever new data is loaded into the master server it can take a few seconds for it to send out NOTIFYs to replicas and for them to transfer the changes. Resolvers looking at your NS records are completely unaware of this and could randomly select either a server that already has the new data or a server that does not.

So regardless of how your DNS host works and how many servers it has, you should never expect updates to be 100% instant; find out whether the provider publishes the expected time, or just wait an arbitrary 5 or 10 seconds.