Intermittent DNS Resolution Failures with CoreDNS and IPTables

Alex Mercer

5/25/24, 2:34 AM

We are using CoreDNS as a sidecar in our service mesh. CoreDNS is listening on port 15053. Both the application and CoreDNS are docker containers running inside an AWS ECS task using awsvpc mode. To direct all DNS requests from the application to CoreDNS, I am using below mentioned IPTable rules. However, I have been encountering intermittent DNS resolution failures with certain applications.

I am using following IPTable rules:

-A OUTPUT -p udp -m udp --dport 53 -m owner --uid-owner 65534 -j RETURN
-A OUTPUT -p udp -m udp --dport 53 -m owner --gid-owner 65534 -j RETURN
-A OUTPUT -p udp -m udp --dport 53 -j REDIRECT --to-ports 15053
-A OUTPUT -p tcp -m tcp --dport 53 -j REDIRECT --to-ports 15053

I checked CPU, Memory of both containers and instance, those are below 30%. I have also verified that CoreDNS is running and reachable. I also checked the presence of DNS records by directly querying to CoreDNS via dig. These errors occur sporadically, approximately once in every thousand requests.

I enabled CoreDNS logs and it appears that the requests are not reaching CoreDNS, suggesting a potential issue with IPTables. I have enabled IPTables logging but could not find any errors or drop packets in the logs. I also verified conntrack, open file limits etc. Unfortunately, I am unable to determine the specific ulimits that may be causing these intermittent errors.

100

0 + 1

domain-name-system

iptables

linux-networking

Greg Askew

5/25/24, 4:23 AM

` I have enabled IPTables logging but could not find any errors or drop packets in the logs.` Broadly - if a product can't track basic things like a packet from point A to B, it's time to get a different product. I'm assuming this is UDP. Given that it only requires a thousand attempts, it should only take a few minutes to create a test case that does a packet capture and identifies the offending component where the packet is being dropped. End to end, multi-point correlated packet capture.

Elon Musk

I sit in a Tesla and translated this thread with Ai:

EN: Intermittent DNS Resolution Failures with CoreDNS and IPTables

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.