I'm on MacOS with apple silicon. I'm tring to set up a vangrant environment of two nodes which both should be able to see each other and be reachable by the host.
For that, this is the Vagrantfile I'm using:
# -*- mode: ruby -*-
# vi:set ft=ruby sw=2 ts=2 sts=2:
Vagrant.configure("2") do |config|
config.vm.box = "perk/ubuntu-2204-arm64"
config.vm.box_check_update = false
config.vm.define "kubemaster" do |node|
# Name shown in the GUI
node.vm.provider "qemu" do |vb|
vb.name = "kubemaster"
vb.memory = 2048
vb.cpus = 2
vb.ssh_port = 4028
vb.extra_qemu_args = %w(-M accel=hvf -nic vmnet-bridged,ifname=en0)
end
node.vm.hostname = "kubemaster"
end
config.vm.define "kubenode01" do |node|
node.vm.provider "qemu" do |vb|
vb.name = "kubenode01"
vb.memory = 2048
vb.cpus = 2
vb.ssh_port = 4025
vb.extra_qemu_args = %w(-M accel=hvf -nic vmnet-bridged,ifname=en0)
end
node.vm.hostname = "kubenode01"
end
end
As you can see I'm using the MacOs' hypervisor (hvf).
Then, I boot the VMs:
# in order to make it work properly
export OBJC_DISABLE_INITIALIZE_FORK_SAFETY=YES
# sudo required otherwise I got
# /Users/mbertamini/.vagrant.d/gems/2.7.6/gems/vagrant-qemu-0.3.3/lib/vagrant-qemu/driver.rb:171:in `read': Permission denied @ rb_sysopen - /Users/mbertamini/Code/Personal/tests-vagrant/1/.vagrant/machines/kubemaster/qemu/xrAIUvRZ5TU/qemu.pid (Errno::EACCES)
sudo -E vagrant up
alright, the two VMs are up. Let's see kubemaster:
vagrant@kubemaster:~$ ifconfig -a
enp0s1: flags=4098<BROADCAST,MULTICAST> mtu 1500
ether 52:54:00:12:34:56 txqueuelen 1000 (Ethernet)
RX packets 0 bytes 0 (0.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 0 bytes 0 (0.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 10.0.2.15 netmask 255.255.255.0 broadcast 10.0.2.255
inet6 fe80::5054:ff:fe12:3457 prefixlen 64 scopeid 0x20<link>
inet6 fec0::5054:ff:fe12:3457 prefixlen 64 scopeid 0x40<site>
ether 52:54:00:12:34:57 txqueuelen 1000 (Ethernet)
RX packets 43883 bytes 51365866 (51.3 MB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 11450 bytes 727113 (727.1 KB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536
inet 127.0.0.1 netmask 255.0.0.0
inet6 ::1 prefixlen 128 scopeid 0x10<host>
loop txqueuelen 1000 (Local Loopback)
RX packets 262 bytes 21495 (21.4 KB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 262 bytes 21495 (21.4 KB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
mmh, I'll try to up that interface:
vagrant@kubemaster:~$ sudo ifconfig enp0s1 up
vagrant@kubemaster:~$ sudo dhclient enp0s1
vagrant@kubemaster:~$ ifconfig
enp0s1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 192.168.1.146 netmask 255.255.255.0 broadcast 192.168.1.255
inet6 2a0c:5a82:e704:6000:5054:ff:fe12:3456 prefixlen 64 scopeid 0x0<global>
inet6 fe80::5054:ff:fe12:3456 prefixlen 64 scopeid 0x20<link>
ether 52:54:00:12:34:56 txqueuelen 1000 (Ethernet)
RX packets 309 bytes 81230 (81.2 KB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 11 bytes 1970 (1.9 KB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 10.0.2.15 netmask 255.255.255.0 broadcast 10.0.2.255
inet6 fe80::5054:ff:fe12:3457 prefixlen 64 scopeid 0x20<link>
inet6 fec0::5054:ff:fe12:3457 prefixlen 64 scopeid 0x40<site>
ether 52:54:00:12:34:57 txqueuelen 1000 (Ethernet)
RX packets 44007 bytes 51374614 (51.3 MB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 11518 bytes 733401 (733.4 KB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536
inet 127.0.0.1 netmask 255.0.0.0
inet6 ::1 prefixlen 128 scopeid 0x10<host>
loop txqueuelen 1000 (Local Loopback)
RX packets 262 bytes 21495 (21.4 KB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 262 bytes 21495 (21.4 KB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
alright! let's see if the host can see it:
$ ping 192.168.1.146 1
PING 192.168.1.146 (192.168.1.146): 56 data bytes
64 bytes from 192.168.1.146: icmp_seq=0 ttl=64 time=2.081 ms
64 bytes from 192.168.1.146: icmp_seq=1 ttl=64 time=0.360 ms
good! I'll proceed with the other node the same way:
# I'll spare you the same setup of before
vagrant@kubenode01:~$ ifconfig
enp0s1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 192.168.1.147 netmask 255.255.255.0 broadcast 192.168.1.255
let's check it:
$ ping 192.168.1.147 130
PING 192.168.1.147 (192.168.1.147): 56 data bytes
64 bytes from 192.168.1.147: icmp_seq=0 ttl=64 time=1.746 ms
64 bytes from 192.168.1.147: icmp_seq=1 ttl=64 time=0.453 ms
well, let's see if the two vms can communicate between each other:
vagrant@kubemaster:~$ ping -I enp0s1 192.168.1.147
PING 192.168.1.147 (192.168.1.147) from 192.168.1.146 enp0s1: 56(84) bytes of data.
From 192.168.1.146 icmp_seq=9 Destination Host Unreachable
From 192.168.1.146 icmp_seq=10 Destination Host Unreachable
ouch, nop. the same happens from kubenode01. So I first thought of a bad ip routing:
vagrant@kubemaster:~$ ip route
default via 192.168.1.1 dev enp0s1
default via 10.0.2.2 dev eth0 proto dhcp src 10.0.2.15 metric 100
10.0.2.0/24 dev eth0 proto kernel scope link src 10.0.2.15 metric 100
10.0.2.2 dev eth0 proto dhcp scope link src 10.0.2.15 metric 100
10.0.2.3 dev eth0 proto dhcp scope link src 10.0.2.15 metric 100
192.168.1.0/24 dev enp0s1 proto kernel scope link src 192.168.1.146
not sure, you'll tell me, but I see everything right.
The other thing that could be wrong is the host device, maybe. I'm not that an expert of MacBooks, so here is my current interface setup:
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> mtu 16384
options=1203<RXCSUM,TXCSUM,TXSTATUS,SW_TIMESTAMP>
inet 127.0.0.1 netmask 0xff000000
inet6 ::1 prefixlen 128
inet6 fe80::1%lo0 prefixlen 64 scopeid 0x1
nd6 options=201<PERFORMNUD,DAD>
gif0: flags=8010<POINTOPOINT,MULTICAST> mtu 1280
stf0: flags=0<> mtu 1280
anpi0: flags=8863<UP,BROADCAST,SMART,RUNNING,SIMPLEX,MULTICAST> mtu 1500
options=400<CHANNEL_IO>
ether c2:e8:f5:67:35:e6
inet6 fe80::c0e8:f5ff:fe67:35e6%anpi0 prefixlen 64 scopeid 0x4
nd6 options=201<PERFORMNUD,DAD>
media: none
status: inactive
anpi1: flags=8863<UP,BROADCAST,SMART,RUNNING,SIMPLEX,MULTICAST> mtu 1500
options=400<CHANNEL_IO>
ether c2:e8:f5:67:35:e7
inet6 fe80::c0e8:f5ff:fe67:35e7%anpi1 prefixlen 64 scopeid 0x5
nd6 options=201<PERFORMNUD,DAD>
media: none
status: inactive
en3: flags=8863<UP,BROADCAST,SMART,RUNNING,SIMPLEX,MULTICAST> mtu 1500
options=400<CHANNEL_IO>
ether c2:e8:f5:67:35:c6
nd6 options=201<PERFORMNUD,DAD>
media: none
status: inactive
en4: flags=8863<UP,BROADCAST,SMART,RUNNING,SIMPLEX,MULTICAST> mtu 1500
options=400<CHANNEL_IO>
ether c2:e8:f5:67:35:c7
nd6 options=201<PERFORMNUD,DAD>
media: none
status: inactive
en1: flags=8963<UP,BROADCAST,SMART,RUNNING,PROMISC,SIMPLEX,MULTICAST> mtu 1500
options=460<TSO4,TSO6,CHANNEL_IO>
ether 36:bc:75:fd:2f:00
media: autoselect <full-duplex>
status: inactive
en2: flags=8963<UP,BROADCAST,SMART,RUNNING,PROMISC,SIMPLEX,MULTICAST> mtu 1500
options=460<TSO4,TSO6,CHANNEL_IO>
ether 36:bc:75:fd:2f:04
media: autoselect <full-duplex>
status: inactive
bridge0: flags=8863<UP,BROADCAST,SMART,RUNNING,SIMPLEX,MULTICAST> mtu 1500
options=63<RXCSUM,TXCSUM,TSO4,TSO6>
ether 36:bc:75:fd:2f:00
Configuration:
id 0:0:0:0:0:0 priority 0 hellotime 0 fwddelay 0
maxage 0 holdcnt 0 proto stp maxaddr 100 timeout 1200
root id 0:0:0:0:0:0 priority 0 ifcost 0 port 0
ipfilter disabled flags 0x0
member: en1 flags=3<LEARNING,DISCOVER>
ifmaxaddr 0 port 8 priority 0 path cost 0
member: en2 flags=3<LEARNING,DISCOVER>
ifmaxaddr 0 port 9 priority 0 path cost 0
nd6 options=201<PERFORMNUD,DAD>
media: <unknown type>
status: inactive
ap1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
options=6463<RXCSUM,TXCSUM,TSO4,TSO6,CHANNEL_IO,PARTIAL_CSUM,ZEROINVERT_CSUM>
ether b2:be:83:6a:6a:23
inet6 fe80::b0be:83ff:fe6a:6a23%ap1 prefixlen 64 scopeid 0xb
nd6 options=201<PERFORMNUD,DAD>
media: autoselect (<unknown type>)
status: inactive
en0: flags=8963<UP,BROADCAST,SMART,RUNNING,PROMISC,SIMPLEX,MULTICAST> mtu 1500
options=6463<RXCSUM,TXCSUM,TSO4,TSO6,CHANNEL_IO,PARTIAL_CSUM,ZEROINVERT_CSUM>
ether b0:be:83:6a:6a:23
inet6 fe80::4a2:8e77:8e97:adf0%en0 prefixlen 64 secured scopeid 0xc
inet 192.168.1.130 netmask 0xffffff00 broadcast 192.168.1.255
inet6 2a0c:5a82:e704:6000:41c:85e6:f98e:7666 prefixlen 64 autoconf secured
inet6 2a0c:5a82:e704:6000:a0c8:2c32:8e3a:eeb8 prefixlen 64 autoconf temporary
nd6 options=201<PERFORMNUD,DAD>
media: autoselect
status: active
awdl0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
options=6463<RXCSUM,TXCSUM,TSO4,TSO6,CHANNEL_IO,PARTIAL_CSUM,ZEROINVERT_CSUM>
ether ee:fe:8f:5a:c4:ab
inet6 fe80::ecfe:8fff:fe5a:c4ab%awdl0 prefixlen 64 scopeid 0xd
nd6 options=201<PERFORMNUD,DAD>
media: autoselect
status: active
llw0: flags=8863<UP,BROADCAST,SMART,RUNNING,SIMPLEX,MULTICAST> mtu 1500
options=400<CHANNEL_IO>
ether ee:fe:8f:5a:c4:ab
inet6 fe80::ecfe:8fff:fe5a:c4ab%llw0 prefixlen 64 scopeid 0xe
nd6 options=201<PERFORMNUD,DAD>
media: autoselect
status: inactive
utun0: flags=8051<UP,POINTOPOINT,RUNNING,MULTICAST> mtu 2000
inet6 fe80::7352:ada1:763a:fc23%utun0 prefixlen 64 scopeid 0xf
nd6 options=201<PERFORMNUD,DAD>
utun1: flags=8051<UP,POINTOPOINT,RUNNING,MULTICAST> mtu 1000
inet6 fe80::ce81:b1c:bd2c:69e%utun1 prefixlen 64 scopeid 0x10
nd6 options=201<PERFORMNUD,DAD>
utun2: flags=8051<UP,POINTOPOINT,RUNNING,MULTICAST> mtu 1380
inet6 fe80::88bd:856:4539:cfeb%utun2 prefixlen 64 scopeid 0x11
nd6 options=201<PERFORMNUD,DAD>
vmenet0: flags=8b63<UP,BROADCAST,SMART,RUNNING,PROMISC,ALLMULTI,SIMPLEX,MULTICAST> mtu 1500
ether b2:cf:a1:d3:06:19
media: autoselect
status: active
vmenet1: flags=8b63<UP,BROADCAST,SMART,RUNNING,PROMISC,ALLMULTI,SIMPLEX,MULTICAST> mtu 1500
ether 9e:05:75:8a:3e:c8
media: autoselect
status: active
bridge100: flags=8a63<UP,BROADCAST,SMART,RUNNING,ALLMULTI,SIMPLEX,MULTICAST> mtu 1500
options=3<RXCSUM,TXCSUM>
ether b2:be:83:a6:85:64
inet 192.168.64.1 netmask 0xffffff00 broadcast 192.168.64.255
inet6 fe80::b0be:83ff:fea6:8564%bridge100 prefixlen 64 scopeid 0x14
inet6 fde3:1a04:ba31:1209:1077:a838:a151:ce6a prefixlen 64 autoconf secured
Configuration:
id 0:0:0:0:0:0 priority 0 hellotime 0 fwddelay 0
maxage 0 holdcnt 0 proto stp maxaddr 100 timeout 1200
root id 0:0:0:0:0:0 priority 0 ifcost 0 port 0
ipfilter disabled flags 0x0
member: vmenet0 flags=3<LEARNING,DISCOVER>
ifmaxaddr 0 port 18 priority 0 path cost 0
member: vmenet1 flags=3<LEARNING,DISCOVER>
ifmaxaddr 0 port 19 priority 0 path cost 0
nd6 options=201<PERFORMNUD,DAD>
media: autoselect
status: active
utun3: flags=8051<UP,POINTOPOINT,RUNNING,MULTICAST> mtu 1380
inet6 fe80::118:fe7d:6bf5:37bd%utun3 prefixlen 64 scopeid 0x15
nd6 options=201<PERFORMNUD,DAD>
utun4: flags=8051<UP,POINTOPOINT,RUNNING,MULTICAST> mtu 1380
inet6 fe80::2428:3e4a:81a4:4445%utun4 prefixlen 64 scopeid 0x17
nd6 options=201<PERFORMNUD,DAD>
utun5: flags=8051<UP,POINTOPOINT,RUNNING,MULTICAST> mtu 1380
inet6 fe80::7687:161a:de51:aa6d%utun5 prefixlen 64 scopeid 0x18
nd6 options=201<PERFORMNUD,DAD>
vlan0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1496
options=6063<RXCSUM,TXCSUM,TSO4,TSO6,PARTIAL_CSUM,ZEROINVERT_CSUM>
ether b0:be:83:6a:6a:23
inet 192.168.126.5 netmask 0xffffff00 broadcast 192.168.126.255
vlan: 3 parent interface: en0
media: autoselect
status: active
bridge1: flags=8863<UP,BROADCAST,SMART,RUNNING,SIMPLEX,MULTICAST> mtu 1500
options=63<RXCSUM,TXCSUM,TSO4,TSO6>
ether b2:be:83:a6:85:01
Configuration:
id 0:0:0:0:0:0 priority 0 hellotime 0 fwddelay 0
maxage 0 holdcnt 0 proto stp maxaddr 100 timeout 1200
root id 0:0:0:0:0:0 priority 0 ifcost 0 port 0
ipfilter disabled flags 0x0
nd6 options=201<PERFORMNUD,DAD>
media: <unknown type>
status: inactive
vmenet2: flags=8963<UP,BROADCAST,SMART,RUNNING,PROMISC,SIMPLEX,MULTICAST> mtu 1500
ether 9a:b6:c9:ec:46:d9
media: autoselect
status: active
bridge101: flags=8863<UP,BROADCAST,SMART,RUNNING,SIMPLEX,MULTICAST> mtu 1500
options=3<RXCSUM,TXCSUM>
ether b2:be:83:a6:85:65
Configuration:
id 0:0:0:0:0:0 priority 0 hellotime 0 fwddelay 0
maxage 0 holdcnt 0 proto stp maxaddr 100 timeout 1200
root id 0:0:0:0:0:0 priority 0 ifcost 0 port 0
ipfilter disabled flags 0x0
member: en0 flags=8003<LEARNING,DISCOVER,MACNAT>
ifmaxaddr 0 port 12 priority 0 path cost 0
member: vmenet2 flags=3<LEARNING,DISCOVER>
ifmaxaddr 0 port 22 priority 0 path cost 0
member: vmenet3 flags=3<LEARNING,DISCOVER>
ifmaxaddr 0 port 29 priority 0 path cost 0
member: vmenet4 flags=3<LEARNING,DISCOVER>
ifmaxaddr 0 port 30 priority 0 path cost 0
member: vmenet5 flags=3<LEARNING,DISCOVER>
ifmaxaddr 0 port 31 priority 0 path cost 0
media: autoselect
status: active
vmenet3: flags=8963<UP,BROADCAST,SMART,RUNNING,PROMISC,SIMPLEX,MULTICAST> mtu 1500
ether c6:cd:0d:18:c4:22
media: autoselect
status: active
vmenet4: flags=8963<UP,BROADCAST,SMART,RUNNING,PROMISC,SIMPLEX,MULTICAST> mtu 1500
ether c2:8b:0e:a0:a5:27
media: autoselect
status: active
vmenet5: flags=8963<UP,BROADCAST,SMART,RUNNING,PROMISC,SIMPLEX,MULTICAST> mtu 1500
ether aa:d4:04:94:6e:fe
media: autoselect
status: active
not sure if everything's correct and in its own place to make it work. I guess further setup is needed to make the networks communicate?
I've also followed this good guide to get what I need, but I can't use libvirt in my case. Since libvirt it should not be an obstacle, I guess I could translate this into something running ok by passing the right parameters to qemu, but I don't know well which of them.
