Score:0

Intel GVT-g: Scrambled graphics on virt-manager

ng flag

enter image description here

The virt-manager video is scrambled on this VM configured to use Intel's GVT-g GPU pass-through. The screen works, I can log in on it, manipulate things on the desktop. I just can't read anything.

Any folk out there knowledgeable on GVT-g that can help me out here?

Background:

Intel's GVT-g is a feature built into 5th through 9th generation core processors that allows virtualization environments like Xen and KVM to present the Intel graphics processors resident on a hosts motherboard to multiple virtual clients, simultaneously. This in turn permits the clients to run 3D accelerated applications at near-hardware speeds.

Explanation:

I have implemented Intel's GVT-g GPU pass-through per the ArchLinux wiki on a Ubuntu 22.04 vm host (Coffee Lake/UHD Graphics 630). The VMs involved are Windows 10 and 2019, I can log into them using RDP (Remmina) despite virt-manager.

On the VMs, the hosts GPU is, indeed, passed-through to the device manager. Driver installation is pretty straight-forward. enter image description here

The really cool thing is; I get acceleration for my 3D applications even though I am in an RDP session. And I can get it on up to 8 VMs, simultaneously!

enter image description here

Everything works except the video in virt-manager!

The details:

Host configuration

inxi -Fj
System:
  Host: cview Kernel: 5.15.0-56-generic x86_64 bits: 64 Desktop: MATE 1.26.0
    Distro: Ubuntu 22.04.1 LTS (Jammy Jellyfish)
Machine:
  Type: Desktop Mobo: ASUSTeK model: PRIME B360M-A v: Rev X.0x
    serial: <superuser required> UEFI: American Megatrends v: 3202
    date: 07/10/2021
CPU:
  Info: 6-core model: Intel Core i5-9600K bits: 64 type: MCP cache:
    L2: 1.5 MiB
  Speed (MHz): avg: 800 min/max: 800/4600 cores: 1: 800 2: 800 3: 800
    4: 800 5: 800 6: 800
Graphics:
  Device-1: Intel CoffeeLake-S GT2 [UHD Graphics 630] driver: i915 v: kernel
  Device-2: NVIDIA GM107GL [Quadro K1200] driver: nouveau v: kernel
  Display: x11 server: X.Org v: 1.21.1.3 driver: X: loaded: modesetting
    unloaded: fbdev,vesa gpu: nouveau resolution: 1: 1920x1080~60Hz
    2: 1080x1920~60Hz 3: 1080x1920~60Hz 4: 1920x1080~60Hz
  OpenGL: renderer: NV117 v: 4.3 Mesa 22.0.5
Audio:
  Device-1: Intel Cannon Lake PCH cAVS driver: snd_hda_intel
  Device-2: NVIDIA GM107 High Definition Audio [GeForce 940MX]
    driver: snd_hda_intel
  Sound Server-1: ALSA v: k5.15.0-56-generic running: yes
  Sound Server-2: PulseAudio v: 15.99.1 running: yes
  Sound Server-3: PipeWire v: 0.3.48 running: yes
Network:
  Device-1: Realtek RTL8111/8168/8411 PCI Express Gigabit Ethernet
    driver: r8169
  IF: enp4s0 state: up speed: 1000 Mbps duplex: full mac: a8:5e:45:53:72:dd
  IF-ID-1: br0 state: up speed: 1000 Mbps duplex: unknown
    mac: 86:cf:5b:e2:98:aa
  IF-ID-2: br2 state: down mac: 16:e4:81:03:54:b3
  IF-ID-3: br257 state: down mac: d6:38:d7:30:2f:c5
  IF-ID-4: br258 state: down mac: ae:87:82:f8:88:d7
  IF-ID-5: br259 state: down mac: 7a:29:2d:9e:5d:ce
  IF-ID-6: br260 state: down mac: da:0b:81:1d:66:c2
  IF-ID-7: br261 state: down mac: 16:8f:e2:9c:de:43
  IF-ID-8: br262 state: down mac: fa:de:05:71:7a:2b
  IF-ID-9: virbr0 state: down mac: 52:54:00:83:4d:49
  IF-ID-10: vlan2 state: up speed: 1000 Mbps duplex: full
    mac: a8:5e:45:53:72:dd
  IF-ID-11: vlan257 state: up speed: 1000 Mbps duplex: full
    mac: a8:5e:45:53:72:dd
  IF-ID-12: vlan258 state: up speed: 1000 Mbps duplex: full
    mac: a8:5e:45:53:72:dd
  IF-ID-13: vlan259 state: up speed: 1000 Mbps duplex: full
    mac: a8:5e:45:53:72:dd
  IF-ID-14: vlan260 state: up speed: 1000 Mbps duplex: full
    mac: a8:5e:45:53:72:dd
  IF-ID-15: vlan261 state: up speed: 1000 Mbps duplex: full
    mac: a8:5e:45:53:72:dd
  IF-ID-16: vlan262 state: up speed: 1000 Mbps duplex: full
    mac: a8:5e:45:53:72:dd
  IF-ID-17: vnet13 state: unknown speed: 10 Mbps duplex: full
    mac: fe:54:00:1c:29:2c
  IF-ID-18: vnet7 state: unknown speed: 10 Mbps duplex: full
    mac: fe:54:00:ab:cb:b0
Drives:
  Local Storage: total: 5.92 TiB used: 3.16 TiB (53.3%)
  ID-1: /dev/nvme0n1 vendor: Samsung model: SSD 970 EVO Plus 1TB
    size: 931.51 GiB
  ID-2: /dev/nvme1n1 vendor: Samsung model: SSD 970 EVO Plus 1TB
    size: 931.51 GiB
  ID-3: /dev/sda vendor: Samsung model: SSD 860 PRO 512GB size: 476.94 GiB
  ID-4: /dev/sdb vendor: Western Digital model: WD4003FZEX-00Z4SA0
    size: 3.64 TiB
Partition:
  ID-1: / size: 447.02 GiB used: 82.6 GiB (18.5%) fs: btrfs dev: /dev/sda3
  ID-2: /boot/efi size: 484 MiB used: 5.2 MiB (1.1%) fs: vfat
    dev: /dev/sda1
Swap:
  ID-1: swap-1 type: partition size: 29.8 GiB used: 17.2 MiB (0.1%)
    dev: /dev/sda2
Sensors:
  System Temperatures: cpu: 27.8 C mobo: N/A gpu: nouveau temp: 47.0 C
  Fan Speeds (RPM): N/A gpu: nouveau fan: 3840
Info:
  Processes: 371 Uptime: 1d 4h 19m Memory: 31.2 GiB used: 10.24 GiB (32.8%)
  Shell: Bash inxi: 3.3.13

Grub cmdline

GRUB_CMDLINE_LINUX="ipv6.disable=1"
GRUB_CMDLINE_LINUX_DEFAULT="quiet splash intel_iommu=on i915.enable_gvt=1"

Example vm configuration ("mesa" libvirt configuration)

<domain type='kvm' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'>
  <name>win10</name>
...
  <devices>
...
   <graphics type='spice'>
      <listen type='none'/>
      <gl enable='yes' rendernode='/dev/dri/by-path/pci-0000:00:02.0-render'/>
    </graphics>
...
    <video>
      <model type='none'/>
    </video>
    <hostdev mode='subsystem' type='mdev' managed='no' model='vfio-pci' display='on'>
      <source>
        <address uuid='f056b522-1c29-4efe-8032-41affa070017'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x06' slot='0x00' function='0x0'/>
    </hostdev>
...
  </devices>
...
  <qemu:commandline>
    <qemu:arg value='-set'/>
    <qemu:arg value='device.hostdev0.x-igd-opregion=on'/>
    <qemu:arg value='-set'/>
    <qemu:arg value='device.hostdev0.ramfb=on'/>
    <qemu:arg value='-set'/>
    <qemu:arg value='device.hostdev0.driver=vfio-pci-nohotplug'/>
    <qemu:arg value='-set'/>
    <qemu:arg value='device.hostdev0.xres=1920'/>
    <qemu:arg value='-set'/>
    <qemu:arg value='device.hostdev0.yres=1080'/>
    <qemu:env name='MESA_LOADER_DRIVER_OVERRIDE' value='i965'/>
  </qemu:commandline>
</domain>
...

Libvirt hook

The virtual GPUs are created/destroyed on VM startup/shutdown using the following libvirt hook:

#!/bin/bash
#
# /etc/libvirt/hooks/qemu
# Automatically creates and destroys virtual GPUs on domain startup and shutdown
# Note: Libvirt will not pick up this file until restarted - reboot your system after creation.
#
GVT_PCI="0000:00:02.0"
MDEV_TYPE="i915-GVTg_V5_4"
fDevType="/sys/bus/pci/devices/$GVT_PCI/mdev_supported_types/$MDEV_TYPE"
domain=$1
cmd1=$2
cmd2=$3

function vGpuConfig {
    if [ -e "$fDevType/create" ]; then
        if [ "$cmd1" = "prepare" ] && [ "$cmd2" = "begin" ]; then
            if [ ! -e "$fDev" ]; then
                      echo "Starting GPU $GVT_GUID..."
                echo "$GVT_GUID" > "$fDevType/create"
            else
                echo "FAIL!! Device already running: $fDev"
            fi
        elif [ "$cmd1" = "release" ] && [ "$cmd2" = "end" ]; then
            if [ -e "$fDev/remove" ]; then
                echo "Stopping GPU $GVT_GUID..."
                echo 1 > "$fDev/remove"
            else
                echo "FAIL!! Device is not running: $fDev"
            fi
        fi
    else
        echo "FAIL!! Device type does not exist: $fDevType"
    fi
}

if [ $# -ge 3 ]; then
    if [ "$domain" = "win10" ]; then
        GVT_GUID="cb33ec6d-ad44-4702-b80f-c176f56afea1"
        fDev="/sys/bus/pci/devices/$GVT_PCI/$GVT_GUID"
        vGpuConfig
    elif [ "$domain" = "Win2k19" ]; then
        GVT_GUID="f056b522-1c29-4efe-8032-41affa070017"
        fDev="/sys/bus/pci/devices/$GVT_PCI/$GVT_GUID"
        vGpuConfig
    fi
else
          echo "usage: $0 <domain> prepare begin | release end"
fi

Summary:

This is exceeding cool technology even though Intel has moved on (later generations will implement GPU pass-through using SR-IOV).

I would like to publish my procedure on how to implement GVT-g. But virt-manager not working (plus the small matter of a SIGABRT in X when running lightdm) are holding me up. I'd appreciate any assistance I can get.

I sit in a Tesla and translated this thread with Ai:

mangohost

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.