Score:0

Server

NodePort communication between nodes not working

mxcolin

12/13/22, 12:33 AM

I have a 4 node cluster with a NodePort service exposed from port 80 to port 30871. I can access the service on each node with that node's hostname eg:

curl http://hostname1:30871 on the node hostname1

works as you would expect. Same for

curl http://hostname2:30871 on the node hostname2

I would expect to be able to reach the service from other nodes. However, running

curl http://hostname1:30871

from node hostname2 results in a connection timed out error. Same thing when I use the IP's. I'm using flannel for networking and it's on a bare metal CentOS Cluster, Kubernetes 1.20.

I can access the service via the ClusterIP from any of the nodes. So

curl http://clusterIPaddress:80

Works like a charm for all nodes.

Firewalls are all open and tested to be working. The same thing happens for machines outside the cluster. Again I'd expect to be able to access http://hostname1:30871 from anywhere, but the same timeout issue occurs.

0 + 8

kubernetes

SYN

12/13/22, 5:07 AM

That's kind of weird ... At that stage, I would install tcpdump, and try to figure out where that missing connection gets lost. Recently (flannel+iptables), I've seen strange things with iptables rules: don't hesitate to compare rules from one nodes to the other.

mxcolin

12/13/22, 2:13 PM

It is weird, I tested the firewall and open ports by using nc on the same exact ports and it works as you would expect, which is very odd.

SYN

12/13/22, 8:44 PM

Use tcpdump, on source and destination nodes, look at the main network interface/where nodeports are listening, filter on the destination port: do you see those connections leaving your source node? Does it reach the destination node? Update your post with those tcpdumps, let us know what you see.

mxcolin

12/13/22, 9:16 PM

Here's what I see until it times out `17:15:37.090460 IP 10.120.112.226.57008 > 10.120.112.210.80: Flags [S], seq 4047977385, win 29200, options [mss 1460,sackOK,TS val 281286877 ecr 0,nop,wscale 7], length 0`

mxcolin

12/13/22, 9:33 PM

I changed k8s to allow sharing of port 80 but same behavior occurs for any port. All ports and tested between machines and working.

SYN

12/14/22, 4:46 AM

What is that? source node, dest node ? Are you running tcpdump on any interfaces? Please, EDIT your initial post, with tcpdump output on NON-SDN network interfaces (there's no reason we would see that port 80), and filter on THE DESTINATION Port (that you are connecting to: your NodePort port). Show us both SOURCE and DESTINATION nodes.

SYN

12/14/22, 4:48 AM

This `10.120.112.226.57008 > 10.120.112.210.80` should stay within your SDN. Your nodes should have a route (`ip r`), sending that subnet through your SDN. If you don't tell us where you've seen this: it's useless.

mxcolin

12/14/22, 2:50 PM

I'm not super familiar with tcpdump, trying to learn. So would this be: `sudo tcpdump --interface <physical interface name> -nn host <destination IP>` on the host and `sudo tcpdump --interface <physical interface name> -nn host <source IP> and port 30871` on the destination?

Elon Musk

I sit in a Tesla and translated this thread with Ai:

EN: NodePort communication between nodes not working

TH: การสื่อสาร NodePort ระหว่างโหนดไม่ทำงาน

RO: Comunicarea NodePort între noduri nu funcționează

RU: Связь NodePort между узлами не работает

VI: Giao tiếp NodePort giữa các nút không hoạt động

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.