Score:0

passing nic to guest in qemu using iommu & vfio

gb flag

I bought a 4-port network adapter PCIe card (Intel E1G44HT 10/100/1000Mbps PCI-Express 2.0 Server Adapter I340-T4)

I am trying to install it in my hpZ600 workstation and pass it through to my guest VM

I am running Artix Linux but am mostly following directions for Arch Linux.

The issue is I am unable to pass the card to my VM.

Here is what I've done so far:

First I update my default/grub

$ sudo -E nvim /etc/default/grub
...
GRUB_CMDLINE_LINUX_DEFAULT="loglevel=3 quiet intel_iommu=on iommu=pt vfio-pci.ids=8086:150e"
...

Then I remake my grub config

sudo grub-mkconfig -o /boot/grub/grub.cfg

I reboot and make sure things are working

sudo reboot -h now

I check the output of dmesg

sudo dmesg | grep -i -e DMAR -e IOMMU
[    0.000000] Command line: BOOT_IMAGE=/vmlinuz-linux root=UUID=095a0572-cdd9-402c-80b7-425724b3cf65 rw loglevel=3 quiet intel_iommu=on iommu=pt vfio-pci.ids=8086:150e
[    0.013145] ACPI: DMAR 0x00000000DEFB8F83 000170 (v01 HPQOEM TYLSBURG 00000001      00000000)
[    0.013171] ACPI: Reserving DMAR table memory at [mem 0xdefb8f83-0xdefb90f2]
[    0.064149] Kernel command line: BOOT_IMAGE=/vmlinuz-linux root=UUID=095a0572-cdd9-402c-80b7-425724b3cf65 rw loglevel=3 quiet intel_iommu=on iommu=pt vfio-pci.ids=8086:150e
[    0.064257] DMAR: IOMMU enabled
[    0.064301] Unknown kernel command line parameters "BOOT_IMAGE=/vmlinuz-linux intel_iommu=on", will be passed to user space.
[    0.166858] DMAR-IR: This system BIOS has enabled interrupt remapping
[    0.455454] iommu: Default domain type: Passthrough (set via kernel command line)
[    0.485787] DMAR: Host address width 40
[    0.485788] DMAR: DRHD base: 0x000000fed90000 flags: 0x1
[    0.485799] DMAR: dmar0: reg_base_addr fed90000 ver 1:0 cap c90780106f0462 ecap f02076
[    0.485802] DMAR: RMRR base: 0x000000defd0000 end: 0x000000defd0fff
[    0.485804] DMAR: RMRR base: 0x000000defd1000 end: 0x000000defd1fff
[    0.485805] DMAR: RMRR base: 0x000000defd2000 end: 0x000000defd2fff
[    0.485807] DMAR: RMRR base: 0x000000defd3000 end: 0x000000defd3fff
[    0.485808] DMAR: RMRR base: 0x000000defd4000 end: 0x000000defd4fff
[    0.485809] DMAR: RMRR base: 0x000000defd5000 end: 0x000000defd5fff
[    0.485811] DMAR: RMRR base: 0x000000defd6000 end: 0x000000defd6fff
[    0.485812] DMAR: RMRR base: 0x000000defd7000 end: 0x000000defd7fff
[    0.485813] DMAR: ATSR flags: 0x0
[    0.485838] DMAR: No SATC found
[    0.485852] DMAR: dmar0: Using Queued invalidation
[    0.485953] pci 0000:00:00.0: Adding to iommu group 0
[    0.485973] pci 0000:00:01.0: Adding to iommu group 1
[    0.485990] pci 0000:00:03.0: Adding to iommu group 2
[    0.486007] pci 0000:00:07.0: Adding to iommu group 3
[    0.486043] pci 0000:00:10.0: Adding to iommu group 4
[    0.486061] pci 0000:00:10.1: Adding to iommu group 4
[    0.486096] pci 0000:00:11.0: Adding to iommu group 5
[    0.486112] pci 0000:00:11.1: Adding to iommu group 5
[    0.486130] pci 0000:00:14.0: Adding to iommu group 6
[    0.486148] pci 0000:00:14.1: Adding to iommu group 7
[    0.486167] pci 0000:00:14.2: Adding to iommu group 8
[    0.486194] pci 0000:00:15.0: Adding to iommu group 9
[    0.486248] pci 0000:00:1a.0: Adding to iommu group 10
[    0.486265] pci 0000:00:1a.1: Adding to iommu group 10
[    0.486282] pci 0000:00:1a.2: Adding to iommu group 10
[    0.486300] pci 0000:00:1a.7: Adding to iommu group 10
[    0.486317] pci 0000:00:1b.0: Adding to iommu group 11
[    0.486353] pci 0000:00:1c.0: Adding to iommu group 12
[    0.486371] pci 0000:00:1c.5: Adding to iommu group 12
[    0.486425] pci 0000:00:1d.0: Adding to iommu group 13
[    0.486444] pci 0000:00:1d.1: Adding to iommu group 13
[    0.486462] pci 0000:00:1d.2: Adding to iommu group 13
[    0.486479] pci 0000:00:1d.7: Adding to iommu group 13
[    0.486496] pci 0000:00:1e.0: Adding to iommu group 14
[    0.486531] pci 0000:00:1f.0: Adding to iommu group 15
[    0.486551] pci 0000:00:1f.2: Adding to iommu group 15
[    0.486572] pci 0000:03:00.0: Adding to iommu group 16
[    0.486591] pci 0000:03:00.1: Adding to iommu group 17
[    0.486612] pci 0000:03:00.2: Adding to iommu group 18
[    0.486630] pci 0000:03:00.3: Adding to iommu group 19
[    0.486652] pci 0000:0f:00.0: Adding to iommu group 20
[    0.486658] pci 0000:01:00.0: Adding to iommu group 12
[    0.486693] pci 0000:3f:00.0: Adding to iommu group 21
[    0.486713] pci 0000:3f:00.1: Adding to iommu group 21
[    0.486767] pci 0000:3f:02.0: Adding to iommu group 22
[    0.486787] pci 0000:3f:02.1: Adding to iommu group 22
[    0.486806] pci 0000:3f:02.4: Adding to iommu group 22
[    0.486825] pci 0000:3f:02.5: Adding to iommu group 22
[    0.486878] pci 0000:3f:03.0: Adding to iommu group 23
[    0.486897] pci 0000:3f:03.1: Adding to iommu group 23
[    0.486916] pci 0000:3f:03.2: Adding to iommu group 23
[    0.486935] pci 0000:3f:03.4: Adding to iommu group 23
[    0.486989] pci 0000:3f:04.0: Adding to iommu group 24
[    0.487009] pci 0000:3f:04.1: Adding to iommu group 24
[    0.487028] pci 0000:3f:04.2: Adding to iommu group 24
[    0.487047] pci 0000:3f:04.3: Adding to iommu group 24
[    0.487100] pci 0000:3f:05.0: Adding to iommu group 25
[    0.487120] pci 0000:3f:05.1: Adding to iommu group 25
[    0.487143] pci 0000:3f:05.2: Adding to iommu group 25
[    0.487163] pci 0000:3f:05.3: Adding to iommu group 25
[    0.487216] pci 0000:3f:06.0: Adding to iommu group 26
[    0.487236] pci 0000:3f:06.1: Adding to iommu group 26
[    0.487276] pci 0000:3f:06.2: Adding to iommu group 26
[    0.487298] pci 0000:3f:06.3: Adding to iommu group 26
[    0.487335] pci 0000:3e:00.0: Adding to iommu group 27
[    0.487356] pci 0000:3e:00.1: Adding to iommu group 27
[    0.487408] pci 0000:3e:02.0: Adding to iommu group 28
[    0.487429] pci 0000:3e:02.1: Adding to iommu group 28
[    0.487451] pci 0000:3e:02.4: Adding to iommu group 28
[    0.487472] pci 0000:3e:02.5: Adding to iommu group 28
[    0.487527] pci 0000:3e:03.0: Adding to iommu group 29
[    0.487549] pci 0000:3e:03.1: Adding to iommu group 29
[    0.487571] pci 0000:3e:03.2: Adding to iommu group 29
[    0.487593] pci 0000:3e:03.4: Adding to iommu group 29
[    0.487647] pci 0000:3e:04.0: Adding to iommu group 30
[    0.487669] pci 0000:3e:04.1: Adding to iommu group 30
[    0.487692] pci 0000:3e:04.2: Adding to iommu group 30
[    0.487716] pci 0000:3e:04.3: Adding to iommu group 30
[    0.487770] pci 0000:3e:05.0: Adding to iommu group 31
[    0.487793] pci 0000:3e:05.1: Adding to iommu group 31
[    0.487816] pci 0000:3e:05.2: Adding to iommu group 31
[    0.487838] pci 0000:3e:05.3: Adding to iommu group 31
[    0.487896] pci 0000:3e:06.0: Adding to iommu group 32
[    0.487922] pci 0000:3e:06.1: Adding to iommu group 32
[    0.487946] pci 0000:3e:06.2: Adding to iommu group 32
[    0.487969] pci 0000:3e:06.3: Adding to iommu group 32
[    0.488088] DMAR: Intel(R) Virtualization Technology for Directed I/O
[    0.520152] AMD-Vi: AMD IOMMUv2 driver by Joerg Roedel <[email protected]>
[    0.520153] AMD-Vi: AMD IOMMUv2 functionality not available on this system
[    1.253336]     intel_iommu=on
[   64.058649] vfio_iommu_type1_attach_group: No interrupt remapping support.  Use the module param "allow_unsafe_interrupts" to enable VFIO IOMMU support on this platform

I used the script from the arch wiki to ensure my groups:

...
IOMMU Group 16:
     03:00.0 Ethernet Controller [0200]: Intel Corporation 82580 Gigabit Netork Connection [8086:150e] (rev 01)
IOMMU Group 17:
     03:00.1 Ethernet Controller [0200]: Intel Corporation 82580 Gigabit Netork Connection [8086:150e] (rev 01)
IOMMU Group 18:
     03:00.2 Ethernet Controller [0200]: Intel Corporation 82580 Gigabit Netork Connection [8086:150e] (rev 01)
IOMMU Group 19:
     03:00.3 Ethernet Controller [0200]: Intel Corporation 82580 Gigabit Netork Connection [8086:150e] (rev 01)
...

And I verify my vfio groups

ls /dev/vfio
16 17 18 19 vfio

And I launch my VM

qemu-system-x86_y4 \
   -m 8G \
   -enable-kvm \
   -cpu host \
   -smp 2 \
   -net none \
   -device vfio-pci,host=03:00.0,id=net0 \
   -vga virtio -display gtk,gl=on \
   -drive format=qcow2,file=disk.cow,index=0,if=virtio \
   -drive if=pflash,format=raw,readonly=on,file=/usr/share/edk2-ovmf/x64/OVMF_CODE.fd \
   -drive if=pflash,format=raw,file=uefi_vars.fd

resulting in the following error:

qemu-system-x86_64: -device vfio-pci,host=03:00.0,id=net0: vfio 0000:03:00.0: failed to setup container for group 16: Failed to set iommu for container: Operation not permitted

I am not sure what to do next. Any thoughts?

mangohost

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.