I have installed kubernetes v1.23.0 with Arch Linux as distribution. The cluster consists of a master and a node. Booth systems are KVM based VMs.
When a pod wants to make a DNS query, it gets a timeout when the service forwards the requests to a pod instance of coredns which is running on another kubernetes node.
So I suspect that the network provider is not working properly or some settings (kernel modules, sysctl, ect) are not set, because when the request is forwarded to the locally running coredns pod instance the client gets a response. Here my debugging steps:
- Before I started debugging, I increased the loglevel of coredns by adding
log to the configmap of coredns.
# kubectl get -n kube-system configmaps coredns -o yaml
apiVersion: v1
data:
Corefile: |
.:53 {
log
errors
health {
lameduck 5s
}
ready
kubernetes cluster.local in-addr.arpa ip6.arpa {
pods insecure
fallthrough in-addr.arpa ip6.arpa
ttl 30
}
prometheus :9153
forward . /etc/resolv.conf {
max_concurrent 1000
}
cache 30
loop
reload
loadbalance
}
- I deployed as debugging environment my network container with network tools like
dig, nslookup and so on to test the different coredns instances.
kubectl apply -f https://raw.githubusercontent.com/volker-raschek/network-tools/master/network-tools.yml
- The following pods and service of coredns is available:
kubectl get pod,service -n kube-system -l k8s-app=kube-dns -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
pod/coredns-64897985d-cgxmv 1/1 Running 0 24h 10.85.0.4 archlinux-x86-64-000 <none> <none>
pod/coredns-64897985d-l9ftl 1/1 Running 0 24h 10.85.0.3 archlinux-x86-64-001 <none> <none>
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR
service/kube-dns ClusterIP 10.96.0.10 <none> 53/UDP,53/TCP,9153/TCP 24h k8s-app=kube-dns
- Execute a shell in my network pod and tries to query the IP adress of
google.com via the coredns service. How to recognize the command takes different lengths of time. Unfortunately, I could not reproduce a timeout error via the service:
# kubectl exec network-tools -- time dig A google.com @10.96.0.10 +short
142.250.185.238
real 0m 5.02s
user 0m 0.02s
sys 0m 0.00s
# kubectl exec network-tools -- time dig A google.com @10.96.0.10 +short
142.250.185.238
real 0m 0.03s
user 0m 0.01s
sys 0m 0.00s
# kubectl exec network-tools -- time dig A google.com @10.96.0.10 +short
142.250.185.238
real 0m 10.03s
user 0m 0.01s
sys 0m 0.01s
- Now I limit the query to the different coredns pods. Note that the pod
coredns-64897985d-cgxmv with the IP 10.85.0.4 is running on a different node.
pod/coredns-64897985d-l9ftl / 10.85.0.3
kubectl exec network-tools -- time dig A google.com @10.85.0.3 +short
142.251.36.206
real 0m 0.09s
user 0m 0.00s
sys 0m 0.01s
pod/coredns-64897985d-cgxmv / 10.85.0.4
Here is the timeout error when explicitly using the coredns pod of another node.
# kubectl exec network-tools -- time dig A google.com @10.85.0.4 +short
; <<>> DiG 9.16.20 <<>> A google.com @10.85.0.4 +short
;; global options: +cmd
;; connection timed out; no servers could be reached
Command exited with non-zero status 9
real 0m 15.02s
user 0m 0.02s
sys 0m 0.00s
command terminated with exit code 9
- The following logs were written by the
coredns pods:
pod/coredns-64897985d-l9ftl / 10.85.0.3
# kubectl logs -n kube-system coredns-64897985d-l9ftl
.:53
[INFO] plugin/reload: Running configuration MD5 = db32ca3650231d74073ff4cf814959a7
CoreDNS-1.8.6
linux/amd64, go1.17.1, 13a9191
[INFO] Reloading
[INFO] plugin/health: Going into lameduck mode for 5s
[INFO] plugin/reload: Running configuration MD5 = 3d3f6363f05ccd60e0f885f0eca6c5ff
[INFO] Reloading complete
[INFO] 127.0.0.1:54962 - 9983 "HINFO IN 4683476401105705616.5032820535498752139. udp 57 false 512" NXDOMAIN qr,rd,ra 132 0.058383302s
[INFO] 10.85.0.1:24999 - 26748 "A IN google.com. udp 51 false 4096" NOERROR qr,rd,ra 1549 0.070006969s
[INFO] 10.85.0.1:6142 - 9467 "A IN google.com. udp 51 false 4096" NOERROR qr,aa,rd,ra 1549 0.000959536s
[INFO] 10.85.0.1:2544 - 20425 "A IN google.com. udp 51 false 4096" NOERROR qr,aa,rd,ra 1549 0.00065977s
[INFO] 10.85.0.1:26782 - 372 "A IN google.com. udp 51 false 4096" NOERROR qr,aa,rd,ra 1549 0.001292768s
[INFO] 10.85.0.1:62687 - 27302 "A IN google.com. udp 51 false 4096"
...
pod/coredns-64897985d-cgxmv / 10.85.0.4
# kubectl logs -n kube-system coredns-64897985d-cgxmv
.:53
[INFO] plugin/reload: Running configuration MD5 = db32ca3650231d74073ff4cf814959a7
CoreDNS-1.8.6
linux/amd64, go1.17.1, 13a9191
[INFO] Reloading
[INFO] plugin/health: Going into lameduck mode for 5s
[INFO] plugin/reload: Running configuration MD5 = 3d3f6363f05ccd60e0f885f0eca6c5ff
[INFO] Reloading complete
To narrow down the problem, I reinstalled my cluster via ansible and installed calico instead of flannel via the command below. The same problem exists there.
$ helm install calico projectcalico/tigera-operator --version v3.21.3
I used the installation guide of kubeadm to initialize the cluster. I executed the following kubeadmin init command to initialize the cluster:
$ kubeadm init \
--apiserver-advertise-address=192.168.179.101 \
--apiserver-cert-extra-sans=api.example.com \
--control-plane-endpoint=192.168.179.100 \
--cri-socket=unix:///var/run/crio/crio.sock \
--pod-network-cidr=10.244.0.0/16 \
--upload-certs
The kernel module br_netfilter and the sysctl properties are defined, but the problem still exists. I am at the end of my solution approaches and need advice from experts here. Below is a list of my kernel modules, sysctl settings and other information.
I hope someone can help me.
kernel information
$ uname -a
Linux archlinux-x86-64-000 5.10.90-1-lts #1 SMP Wed, 05 Jan 2022 13:07:40 +0000 x86_64 GNU/Linux
kernel modules
$ lsmod | sort
ac97_bus 16384 1 snd_soc_core
aesni_intel 372736 0
agpgart 53248 4 intel_agp,intel_gtt,ttm,drm
atkbd 36864 0
bpf_preload 16384 0
bridge 274432 1 br_netfilter
br_netfilter 32768 0
cec 61440 1 drm_kms_helper
cfg80211 983040 0
crc16 16384 1 ext4
crc32c_generic 16384 0
crc32c_intel 24576 3
crc32_pclmul 16384 0
crct10dif_pclmul 16384 1
cryptd 24576 2 crypto_simd,ghash_clmulni_intel
crypto_simd 16384 1 aesni_intel
drm 577536 5 drm_kms_helper,qxl,drm_ttm_helper,ttm
drm_kms_helper 278528 3 qxl
drm_ttm_helper 16384 1 qxl
ext4 933888 1
failover 16384 1 net_failover
fat 86016 1 vfat
fb_sys_fops 16384 1 drm_kms_helper
fuse 167936 1
ghash_clmulni_intel 16384 0
glue_helper 16384 1 aesni_intel
i2c_i801 36864 0
i2c_smbus 20480 1 i2c_i801
i8042 36864 0
intel_agp 24576 0
intel_gtt 24576 1 intel_agp
intel_pmc_bxt 16384 1 iTCO_wdt
intel_rapl_common 28672 1 intel_rapl_msr
intel_rapl_msr 20480 0
ip6_udp_tunnel 16384 1 vxlan
ip_set 57344 0
ip_tables 32768 0
ipt_REJECT 16384 0
ip_vs 184320 6 ip_vs_rr,ip_vs_sh,ip_vs_wrr
ip_vs_rr 16384 0
ip_vs_sh 16384 0
ip_vs_wrr 16384 0
irqbypass 16384 1 kvm
iTCO_vendor_support 16384 1 iTCO_wdt
iTCO_wdt 16384 0
jbd2 151552 1 ext4
joydev 28672 0
kvm 933888 1 kvm_intel
kvm_intel 331776 0
ledtrig_audio 16384 1 snd_hda_codec_generic
libcrc32c 16384 4 nf_conntrack,nf_nat,nf_tables,ip_vs
libps2 20480 2 atkbd,psmouse
llc 16384 2 bridge,stp
lpc_ich 28672 0
mac_hid 16384 0
mbcache 16384 1 ext4
Module Size Used by
mousedev 24576 0
net_failover 24576 1 virtio_net
nf_conntrack 172032 6 xt_conntrack,nf_nat,xt_nat,nf_conntrack_netlink,xt_MASQUERADE,ip_vs
nf_conntrack_netlink 57344 0
nf_defrag_ipv4 16384 1 nf_conntrack
nf_defrag_ipv6 24576 2 nf_conntrack,ip_vs
nf_nat 57344 3 xt_nat,nft_chain_nat,xt_MASQUERADE
nfnetlink 20480 4 nft_compat,nf_conntrack_netlink,nf_tables,ip_set
nf_reject_ipv4 16384 1 ipt_REJECT
nf_tables 249856 183 nft_compat,nft_counter,nft_chain_nat
nft_chain_nat 16384 7
nft_compat 20480 122
nft_counter 16384 84
nls_iso8859_1 16384 0
overlay 147456 18
pcspkr 16384 0
psmouse 184320 0
qemu_fw_cfg 20480 0
qxl 73728 0
rapl 16384 0
rfkill 28672 2 cfg80211
rng_core 16384 1 virtio_rng
serio 28672 6 serio_raw,atkbd,psmouse,i8042
serio_raw 20480 0
snd 114688 8 snd_hda_codec_generic,snd_hwdep,snd_hda_intel,snd_hda_codec,snd_timer,snd_compress,snd_soc_core,snd_pcm
snd_compress 32768 1 snd_soc_core
snd_hda_codec 172032 2 snd_hda_codec_generic,snd_hda_intel
snd_hda_codec_generic 98304 1
snd_hda_core 110592 3 snd_hda_codec_generic,snd_hda_intel,snd_hda_codec
snd_hda_intel 57344 0
snd_hwdep 16384 1 snd_hda_codec
snd_intel_dspcfg 28672 1 snd_hda_intel
snd_pcm 147456 7 snd_hda_intel,snd_hda_codec,soundwire_intel,snd_compress,snd_soc_core,snd_hda_core,snd_pcm_dmaengine
snd_pcm_dmaengine 16384 1 snd_soc_core
snd_soc_core 327680 1 soundwire_intel
snd_timer 49152 1 snd_pcm
soundcore 16384 1 snd
soundwire_bus 90112 3 soundwire_intel,soundwire_generic_allocation,soundwire_cadence
soundwire_cadence 36864 1 soundwire_intel
soundwire_generic_allocation 16384 1 soundwire_intel
soundwire_intel 45056 1 snd_intel_dspcfg
stp 16384 1 bridge
syscopyarea 16384 1 drm_kms_helper
sysfillrect 16384 1 drm_kms_helper
sysimgblt 16384 1 drm_kms_helper
ttm 114688 2 qxl,drm_ttm_helper
udp_tunnel 20480 1 vxlan
usbhid 65536 0
veth 32768 0
vfat 24576 0
virtio_balloon 24576 0
virtio_blk 20480 2
virtio_console 40960 0
virtio_net 61440 0
virtio_pci 28672 0
virtio_rng 16384 0
vxlan 77824 0
xhci_pci 20480 0
xhci_pci_renesas 20480 1 xhci_pci
x_tables 53248 11 xt_conntrack,xt_statistic,nft_compat,xt_tcpudp,xt_addrtype,xt_nat,xt_comment,ipt_REJECT,ip_tables,xt_MASQUERADE,xt_mark
xt_addrtype 16384 2
xt_comment 16384 64
xt_conntrack 16384 13
xt_mark 16384 12
xt_MASQUERADE 20480 6
xt_nat 16384 7
xt_statistic 16384 3
xt_tcpudp 20480 15
sysctl
sysctl properties on pastbin. Exceeds the maximum number of characters for serverfault.
