Score:1

Server

Pods on a k8s node are unaccessible, kube-proxy or CNI failed

Andy Huang

10/8/23, 5:04 PM

I have add a new node to my k8s cluster, but I found some allocated to this node cannot show logs like this:

$ kubectl logs -n xxxx xxxxx-6d5bdd7d6f-5ps6k

Unable to connect to the server: EOF

Using Lens gives error logs like this:

Failed to load logs: request to http://127.0.0.1:49271/api-kube/api/v1/namespaces/xxxxxxx/pods/xxxx34-27736483--1-hxjpv/log?tailLines=500&timestamps=true&container=xxxxxx&previous=false failed, reason: socket hang up
Reason: undefined (ECONNRESET)

I believe there's some problem in this node, when I use port-forwarding:

$ kubectl port-forward -n argocd svc/argocd-notifications-controller-metrics 9001:9001
error: error upgrading connection: error dialing backend: dial tcp 10.0.6.20:10250: i/o timeout

I think the internal IP 10.0.6.20 is wrong.

All kube-proxy pods shows running from kubectl

-> % kgp -o wide -n kube-system | grep kube-proxy
kube-proxy-7pg9d                                  1/1     Running     1 (2d20h ago)   29d     10.0.6.20        worker4     
kube-proxy-cqh2c                                  1/1     Running     1 (15d ago)     29d     10.0.6.3         worker3           
kube-proxy-lp4cd                                  1/1     Running     0               29d     10.0.6.1         worker1           
kube-proxy-r6bgw                                  1/1     Running     0               29d     10.0.6.2         worker2

But using crictl pods on each node looking for these pods

# crictl pods | grep kube-proxy
ceef94b060e56       2 days ago          Ready               kube-proxy-7pg9d                                   kube-system         1                   (default)
418bd5b46c2b9       4 weeks ago         NotReady            kube-proxy-7pg9d                                   kube-system         0                   (default)

Shows Ready or NotReady I am using Calico for CNI, in ip_vs mode. How can I fix this?

1120

1 + 12

networking

proxy

kubernetes

calico

containerd

SYN

10/9/23, 8:10 AM

There's nothing wrong with those NotReady pods, as long as you have another one that is Ready, started afterwards. What makes you think `10.0.6.20` is wrong, what's your node IP? Any chance your DNS would resolve "worker4" to wrong IP? Regardless, connection timing out to port 10250 (kubelet) indeed suggests there's no one listening on that IP. I would ssh to that node, check ip configuration,, routes, compare with a working node.

SYN

10/9/23, 8:16 AM

from crictl perspective, a pod that is NotReady may just be some leftover from a previous kubernetes pod. If you run `crictl ps`, you would see running containers, which would belong to ready pods. while `crictl ps -a` would show you exited containers, usually belonging to unready pods. usually, unready crictl pods could be removed, manually or using scripts, when kubelet leaves those behind.

Andy Huang

10/9/23, 8:24 AM

@SYN, because I do port-forwarding from my computer, which should not in the subnet where `10.0.6.20` is reachable, the node ip should be an actual IP of worker4

SYN

10/9/23, 8:27 AM

"I do port-forwarding" : please explain what do you mean here. "which should not be in the subnet where 10.0.6.20 is reachable" ; makes no sense, you realize your other workers are in the same /26? "should be an actual IP of worker4" : again, what's that node IP, then? Have you checked ip & routing configuration on that node? What about dns resolution on the other node (specifically: the api)

SYN

10/9/23, 8:31 AM

Keeping in mind the "IP" address showing on your pods is usually set by the kubelet instance running on your node, based on actual values at the time of starting the pod. Should we guess that you're using DHCP assigning addresses to your nodes, and the lease for worker4 was somehow renewed? Feel free to add details to your original post

Andy Huang

10/10/23, 3:01 AM

@SYN I use port-forwearding from kubectl from my local computer `kubectl port-forward -n argocd svc/argocd-notifications-controller-metrics 9001:9001` shows this error: `error: error upgrading connection: error dialing backend: dial tcp 10.0.6.20:10250: i/o timeout` 10.0.6.20 is impossible to reach from my local computer. I am not sure how to check routing, configurations, DNS among nodes.

SYN

10/10/23, 6:59 AM

We would use kubectl port-forward connecting SDN addresses/services. Here, we're talking about the IP of your node (unrelated to your SDN). You should have some kind of access over there.

Andy Huang

10/10/23, 11:35 AM

@SYN, I can ssh to all nodes

SYN

10/10/23, 11:45 AM

... so what is your worker4 node IP? if 10.0.6.20: check kubelet logs, as we could see there's no answer from here. Since timeout (and not refused), I suspect you have some DHCP, node changed addresses, and somehow API doesn't yet know about it.

Andy Huang

10/10/23, 1:10 PM

@SYN worker4 IP is `213.108.105.12`, the kubelet service is run by systemd, I used `journalctl -xeu kubelet` to find logs, but mostly logs are about specific container status, what specific key I should note?

Andy Huang

10/10/23, 1:54 PM

I found iptables on worker4 is dropping all packets from kubeapiserver

SYN

10/10/23, 3:19 PM

any clue why?! lacking anything better to suggest: would a reboot help, maybe?

Score:1

Server

Andy Huang

10/10/23, 4:04 PM

I solved this problem following these procedure:

Worker4

Make sure kubelet is listening on default port:

# lsof -i:10250
COMMAND PID USER   FD   TYPE DEVICE SIZE/OFF NODE NAME
kubelet 819 root   26u  IPv4  13966      0t0  TCP worker4.cluster.local:10250 (LISTEN)

Worker1 curl https://10.0.6.20:10250 Gets timeout But found curl https://10.0.6.1:10250 # worker1 from worker4 responded quick.

So this might be packets is dropped inside worker4,

This logs packet in worker4: https://www.thegeekstuff.com/2012/08/iptables-log-packets/

iptables -N LOGGING
iptables -A INPUT -j LOGGING
iptables -A LOGGING -m limit --limit 2/min -j LOG --log-prefix "IPTables-Dropped: " --log-level 4
iptables -A LOGGING -j DROP

Will save logs to /var/log/syslog

Using command to filter from logs:

tail -200 /var/log/syslog | grep IPTables-Dropped | grep 10.0.6.1
Oct 10 13:49:37 compute kernel: [637626.880648] IPTables-Dropped: IN=eth1 OUT= MAC=00:16:ce:d4:b7:01:00:16:b2:77:89:01:08:00 SRC=10.0.6.1 DST=10.0.6.20 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=29087 DF PROTO=TCP SPT=58838 DPT=10250 WINDOW=64240 RES=0x00 SYN URGP=0

So I am convinced the packet is dropped.

Adding rules:

iptables -I INPUT -s 10.0.0.0/8 -p tcp --dport 10250 -j ACCEPT

Then I can attach shell or get logs from pods on the node. I appreciate discussions with @SYN

+ 1

SYN

10/12/23, 4:53 PM

it's weird that port would not be reachable, it is indeed critical for kubernetes operations. kube-proxy and sdn components may setup rules on your nodes, but it should not interfere with this. If rebooting did not help, it could be worth investigating: where did that come from, how come you don't have it on your other nodes, ... still, nice catch. pretty weird.

Elon Musk

I sit in a Tesla and translated this thread with Ai:

EN: Pods on a k8s node are unaccessible, kube-proxy or CNI failed

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.