Join Log in
Score:1
Marsplay avatar Server

Ubuntu 20.04 routing for just one IP (in the same subnet) ends in "dev lo" instead "dev eth0", kubernetes worker node can't connect to master node

fr flag
Marsplay

I bumped to (as it now seem to me) routing issue. I can no longer access one of my worker nodes (server) from my master node (server). AFAIK, it has nothing to do with Kubernetes, it leads to pure Linux networking issue. As the issue is with only one IP, I was troubleshooting iptables, enabled TRACE and realised that packet actualy comes accross master (eth0), gets to iptables (passes: raw > mangle >nat) but when it has to get routed from nat to filter, it just dissapears. As I understand that is the point in which kernel has to make routing decision. Checked routing and found it's not working for just that one IP (all others from same IP segment are working fine) !? As I'm with cloud provider, and can't troubleshoot networking, so I tried reinstalling OS (same Ubuntu 20.04) of master node (server). Found out that with fresh OS reinstall, the issue was not present, therefore configuration issue must be within my master Linux server (I reverted the server image back form snapshot).

root@vmi57XXXX:~# route  
Kernel IP routing table  
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface  
default         gw.provider.net 0.0.0.0         UG    0      0        0 eth0  
10.244.0.0      0.0.0.0         255.255.255.0   U     0      0        0 cni0  
10.244.1.0      10.244.1.0      255.255.255.0   UG    0      0        0 flannel.1  
172.17.0.0      0.0.0.0         255.255.0.0     U     0      0        0 docker0  
root@vmi57XXXX:~# ip route get xx.xx.xx.96  
local xx.xx.xx.96 dev lo src xx.xx.xx.96 uid 0   
    cache <local>   
root@vmi57XXXX:~# ip route get xx.xx.xx.95  
xx.xx.xx.95 via xx.xx.xx.1 dev eth0 src xx.xx.xx.95 uid 0   
    cache  
root@vmi57XXXX:~# ip route get xx.xx.xx.97  
xx.xx.xx.97 via xx.xx.xx.1 dev eth0 src xx.xx.xx.97 uid 0   
    cache   
  
root@vmi57XXXX:~# arp -v  
Address                  HWtype  HWaddress           Flags Mask            Iface  
10.244.0.60              ether   8a:94:de:43:b6:0f   C                     cni0  
10.244.0.63              ether   1e:76:6a:60:27:f3   C                     cni0  
10.244.0.62              ether   36:0b:19:5e:57:87   C                     cni0  
gw.provider.net          ether   00:c0:1d:c0:ff:ee   C                     eth0  
10.244.0.64              ether   82:03:61:c5:4d:fb   C                     cni0  
10.244.0.50                      (incomplete)                              cni0  
10.244.1.0               ether   52:3d:a5:f4:c2:2c   CM                    flannel.1  
10.244.0.61              ether   56:19:98:79:a1:3a   C                     cni0  
Entries: 8  Skipped: 0  Found: 8  

root@vmi57XXXX:~# ip netconf show dev eth0
inet eth0 forwarding on rp_filter off mc_forwarding off proxy_neigh off 
ignore_routes_with_linkdown off 
inet6 eth0 forwarding off mc_forwarding off proxy_neigh off 
ignore_routes_with_linkdown off

Any clues about whats going on over there are more than welcome!!!

Thanks

EDIT: After solving the issue it's worth mentioning that this behavior was experienced with Kubernetes 1.21.2-00 and flannel as CNI. I did the upgrade few weeks ago and this was the first restart of one worker node after the upgrade.

272
1 + 0
Score:1
Marsplay avatar Server
fr flag
Marsplay

SOLVED!

bad guy was actually Kubernetes - it set an L O C A L route on master node that can't work without Kubernetes' functional networking service (flannel - in my case). Therefore when worker node got rebooted it was no longer able to access master node's API service (6443/tcp) and couldn't present itself to API - that closed magic circle in which woker node looped with no luck.

I learned today about "local" routes maintained by kernel (all present routing tables can be found here: /etc/iproute2/rt_tables).

ip route ls table local
local xx.xx.xx.96 dev kube-ipvs0 proto kernel scope host src xx.xx.xx.96  <<< PROBLEMATIC
local xx.xx.xx.50 dev eth0 proto kernel scope host src xx.xx.xx.50  <<< i.e. OK

delete route

ip route del table local local xx.xx.xx.96 dev kube-ipvs0 proto kernel scope host src xx.xx.xx.96

and now it works

root@vmi57XXXX:~# ip route get xx.xx.xx.96
xx.xx.xx.96 via xx.xx.xx.1 dev eth0 src xx.xx.xx.50 uid 0 
    cache
0 + 2
Martin avatar
kz flag
Martin
go ahead and accept your own answer. Nice one! I never knew that there is a routing table which isn't displayed by using the "old" route command...
0
Reply
Marsplay avatar
fr flag
Marsplay
had to spend two days and one sleepless night to find that out. I wasn't sure what exactly I was looking for and googling didn't help at all. To help others I have added kubernetes to the issue title as there might be many more people with the same issue. And to accept my own answer it takes two days, that's what system rules says :)
0
Reply
ธงชาติไทย
Elon Musk
I sit in a Tesla and translated this thread with Ai:

mangohost

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.