I have been trying to install kubeadm based k8s cluster on Macbook M1. I am using VMware Fusion tech preview to visualize Ubuntu Arm64 platform. Following is the OS build info.
abhilash@lvm-master:~$ hostnamectl
Static hostname: lvm-master
Icon name: computer-vm
Chassis: vm
Machine ID: 81b2486b2a3d4f17b669067757aacdf4
Boot ID: 8fc7f05e687c47579051163b59eeec6b
Virtualization: vmware
Operating System: Ubuntu 20.04.3 LTS
Kernel: Linux 5.4.0-100-generic
Architecture: arm64
abhilash@lvm-master:~$
Usual steps to bring up the K8s cluster is to install a CNI plugin like flannel/calico after bringing up kubernetes control plane. In background this step will created multiple network interfaces as well.
I have restarted AMD64 based k8s nodes multiple times i do not see discrepancy around network interfaces post restart. However, Its not the same for ARM64 based nodes.
I have encountered missing network interfaces 10/10 times. I will be pasting "ip a" output off all nodes before and after restart. Since interfaces are missing all/most of the calico/flannel pods are either in Error/CashLoopBackOff state. Hence K8s system pods also go into Completed/CLBF.
I had taken snapshot of all vms before restart. Hence post restart when i see some interface missing and all calico pods in bad shape. I revert the snapshot . I see all interface appear again and pods are back to Running state. Hence i can deduce that CLBF/Error state of calico/kube-system pods are due to missing interfaces.
It seems like This Arm64 platform is not able to keep all flannel/calico created interfaces after reboot. I am not sure if this issue was reported or even if someone tried using ARM64 for K8s.
I am using netplan on 1 interface that i am connecting to VM. Everything is fine from that end. I get same static IP upon each restart. I guess this should not matter as I have used netplan in AMD64 as well.
I have not deleted the cluster. I can reproduce anything since I have taken snapshots at appropriate time.
