Join Log in
Score:0
avatar Server

coreDNS flooded by istio (envoy)

cn flag
bofh.at

When there are lots of external VMs which are accessible only via a firewall and there are multiple namespaces in the cluster, each with its own set of external VMs, you end up with a lot of ServiceEntries, which in turn cause a lot of DNS queries for

$host.$ns.svc.cluster.local 
$host.svc.cluster.local
$host.cluster.local

where $host is for example: vm-1.vm-cluster.example.com with let's say 10 namespaces and 100 pods in each namespace, that's quite a number of DNS queries, all answered with NXDOMAIN, so the local istio DNS proxy cache does not work. This can lead to an overload situation of the coreDNS Pods (the memory size, including buffers goes up and cause the DNS Pod to OOM).

My question is: how can I prevent those queries? The solutions I came up with are:

Replacing $host with a FQDN, which includes the trailing dot, does not work because the X.509 certs contain the hostname without the trailing dot. Change the DNS config of each Pod and set the ndots to something like 3. But this sounds like a maintenance nightmare. I would like to solve this via a clever combination of ServiceEntry, VirtualService and Gateway definitions.

K8s version 1.19 (vmware edition) Istio version 1.9 (also vmware edition)

100
0 + 0
Mikołaj Głodziak avatar
id flag
Mikołaj Głodziak
Is OOM only the problem or is something else going on? Can you e.g. increase memory resources for DNS Pod?
0
Reply
cn flag
bofh.at
I can't reliably increase the memory, because the VMware cluster has a fixed config for coreDNS, i was told. It's also a latency problem. the bigger the queues, the longer a DNS query takes. you want them to be fast. I found a problem with our config though. Somehow all services got defined in all namespaces. We can and will define the services only in those namespaces where they are needed. But i can see that the problem will come back later, if the cluster and it's external dependencies grow.
0
Reply
Mikołaj Głodziak avatar
id flag
Mikołaj Głodziak
Could you add your yaml files (current configuration) to reproduce your problem?
0
Reply
Wytrzymały Wiktor avatar
it flag
Wytrzymały Wiktor
Hello @bofh.at. Any updates?
0
Reply
ธงชาติไทย
Elon Musk
I sit in a Tesla and translated this thread with Ai:

mangohost

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.