KubeADM cluster: how to configure DNS properly

I have the following project that I use to create my own Kubernetes cluster on the local machine (macOS) via KubeAdm: https://github.com/sasadangelo/k8s-cluster Basically, I started from this project and did the following update:

• Configure the Vagrant deployment via a YAML file.
• Support the K8s 1.6 APIs, K8s 1.18, and Calico 3.8.8
• Automatic K8s dashboard deploy.

Now I am fine with this cluster but I have some DNS issues I would like to understand how to solve. The tool uses a Vagrant file to start 3 Vagrant boxes then I use mainly 3 scripts:

• configure_box.sh, I use to install on each box the required software like docker, kubeadm, kubelet, kubectl, etc.
• configure_master.sh, used to configure only the master node, install network plugin, and run the kubeadm init.
• configure_worker.sh, used to configure the worker node joining the cluster.

My problem is that resolv.conf of my Pods looks like this:

nameserver 10.96.0.10
search default.svc.cluster.local svc.cluster.local cluster.local <other domain coming from host machine>
options ndots:5

This file has two problems:

1. doesn't allow me to access the Internet. In fact, I have to add manually add it adding this line at the beginning of the file:

nameserver 8.8.8.8

and remove it when not required anymore (if I keep it internal DNS doesn't work)

2. in the search clause there are domains coming from my host machines: <other domain coming from host machine> that cause issues.

I read a lot of documentation but I still have problems figuring out how to change my code to avoid these two problems.

1. doesn't allow me to access the Internet.

Your default configuration seems fine. As you've found, you don't want to change your pod resolv.conf as it needs to use the coredns service within the cluster for local dns lookups. The coredns service should be able to forward dns requests to the resolver configured in your k8s-master /etc/resolv.conf. It seems like it may be a problem with the k8s-master server internet access or with the virtualbox networking. I cloned your repo and was able to resolve internet dns from pods running on the cluster as expected.

If you take a look at the coredns configuration you will see that it's set to use /etc/resolv.conf for any unspecified domains. From k8s-master you can run kubectl -n kube-system get configmap coredns -o yaml to see the coredns config. There will be a line, forward . /etc/resolv.conf. The kubernetes docs here explain the coredns corefile configuration. You will also find that the resolve.conf in the coredns pod is the same as on the host. To verify this, you can run ps -aux | grep coredns to get the pid of one of the coredns processes. Then run nsenter -t <pid> -n cat /etc/resolv.conf and you should see that the content is the same as the host system. It should have a resolver from the virtualbox network. You can test this resolver from the host or within the coredns process to validate if it can resolve dns. You could also test against the google resolver from within the coredns process to see what you get. I would try a dns lookup as well as just a ping. This should help you narrow down what's causing your issue. Again, I tried your vagrantfile on a brand new ubuntu 20.04 server and it worked fine.

2. in the search clause there are domains coming from my host machines that cause issues.

This is also the expected configuration. I wouldn't typically expect this to cause issues, but I don't know your exact configuration or what you're trying to achieve. If the solution requires that you modify the list of search domains in the pod, the place to do that would be in the pod dns policy. Hopefully solving your first issue will allow you to avoid having to customize this.