Score:1

Calico node remains not ready when a new node with a 192.168.178.x IP joins a K8s cluster

id flag
kube-system   pod/calico-node-9czgm                                  0/1     Running     3          42d
kube-system   pod/calico-node-msfjk                                  0/1     Running     0          5m37s

...is what I get when a let "moon" to join the cluster as outlined below...

NAMESPACE   NAME        STATUS   ROLES                  AGE     VERSION
            node/moon   Ready    <none>                 5m38s   v1.20.5
            node/sky    Ready    control-plane,master   276d    v1.21.2

Whereby pod/calico-node-9czgm is sky's calico node that is running just fine (1/1 Ready) before moon joins and after moon is deleted from the k8s cluster.

❯ kubectl logs pod/calico-node-msfjk -n kube-system -f Error from server: Get "https://192.168.178.23:10250/containerLogs/kube-system/calico-node-msfjk/calico-node?follow=true": dial tcp 192.168.178.23:10250: i/o timeout

pod/calico-node-9czgm tells me:

2021-07-04 20:39:38.999 [INFO][72] felix/ipsets.go 760: Doing full IP set rewrite family="inet" numMembersInPendingReplace=2 setID="all-hosts-net"
bird: Reconfiguration requested by SIGHUP
bird: Reconfiguring
bird: device1: Reconfigured
bird: direct1: Reconfigured
bird: Reconfigured
2021-07-04 20:39:39.023 [INFO][76] confd/resource.go 277: Target config /etc/calico/confd/config/bird.cfg has been updated due to change in key: /calico/bgp/v1/host
2021-07-04 20:39:43.185 [INFO][76] confd/client.go 897: Recompute BGP peerings: HostBGPConfig(node=moon; name=ip_addr_v4) updated; HostBGPConfig(node=moon; name=network_v4) updated
bird: Reconfiguration requested by SIGHUP
bird: Reconfiguring
bird: device1: Reconfigured
bird: direct1: Reconfigured
bird: Adding protocol Mesh_192_168_178_23
bird: Mesh_192_168_178_23: Initializing
bird: Mesh_192_168_178_23: Starting
bird: Mesh_192_168_178_23: State changed to start
bird: Reconfigured
2021-07-04 20:39:43.198 [INFO][76] confd/resource.go 277: Target config /etc/calico/confd/config/bird.cfg has been updated due to change in key: /calico/bgp/v1/host
2021-07-04 20:39:53.134 [INFO][72] felix/summary.go 100: Summarising 13 dataplane reconciliation loops over 1m1.6s: avg=14ms longest=77ms ()
2021-07-04 20:40:11.444 [INFO][74] monitor-addresses/startup.go 774: Using autodetected IPv4 address on interface eth0: 88.99.99.99/32

The IP 192.168.178.23 belongs to my notebook's WiFi interface. So apparently I need to override the local' interface IP with my public IP which I try to do via --node-ip=93.93.17.11 in /etc/systemd/system/kubelet.service.d/kubeadm.conf but everything remains broken and the kubelet additionally reports:

Jul 04 22:58:43 moon kubelet[92613]: E0704 22:58:43.610441 92613 kubelet_node_status.go:586] Failed to set some node status fields: failed to validate nodeIP: node IP: "93.93.17.11" not found in the host's network interfaces

I am using K8s v1.20.5 with cri-o v1.20 on Fedora 34.

p10l avatar
us flag
If possible, please include your Calico deployment yaml files. Calico [docs](https://docs.projectcalico.org/reference/resources/node) state that "When starting a `calico/node instance`, the name supplied to the instance should match the name configured in the Node resource. By default, starting a `calico/node` instance will automatically create a node resource using the `hostname` of the compute host." If you created nodes without explicitly stating resource names, this may by an issue.
id flag
My Calico deployment is `kubectl apply -f https://docs.projectcalico.org/v3.19/manifests/calico.yaml` but thanks this is a point I can pursue further. cc @PawełGrondal
p10l avatar
us flag
have you found a solution?
moonkotte avatar
in flag
Are you able ping beetween 2 VMs? Is there any firewall?
mangohost

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.