Linux Livirt/KVM: Place all VMs in cpuset by default without reconfiguration

Joshua Boniface

11/8/23, 8:15 AM

I'm trying to set up a system whereby I have some process on a set of hosts that are in one (shielded) cpuset, while everything else (i.e. all the VMs) are in another. The goal here is to have a hyperconverged system where the VMs run on some arbitrary set of of the CPU cores, while the storage process run on the other cores, with isolation. I've selected cpuset to achieve this task as it seems like the "easiest" way to do it.

However this does not seem to work properly. VMs can be moved when first setting up the cset shield, but starting VMs later results in an error like:

libvirt: Cgroup error : Invalid value '0-31' for 'cpuset.cpus': Invalid argument

Clearly it's trying to put the VM in the root cset which is not allowed. This is with a very default configuration, just specifying a number of cores in the VM without any concern for individual CPU IDs or any such tuning (which is a requirement for this setup to work; nodes might be asymmetrical, have different core counts, topologies, etc. while VMs can live migrate around them).

But in general, I'd like to know: can Libvirt/KVM be configured to use a specific cpuset by default, without reconfiguring the VMs in any way (no manual config CPU pinning shenanigans), and without any manual tweaking on a per-process basis?

212

1 + 0

libvirt

kvm-virtualization

cpuset

Score:1

Server

DanielB

11/8/23, 8:54 AM

On systemd based hosts, all VMs are placed in a cgroup beneath /machine.slice so if you set CPU mask for /machine.slice it'll apply to all VMs. Similarly if you set CPUAffinity in /etc/systemd/system.conf, that can make all non-VM processes avoid the CPUs used for VMs.

+ 6

Joshua Boniface

11/8/23, 3:07 PM

Excellent, I had seen machine.slice but had to remove it (forcibly) to enable the default shielding. I'll experiment a bit with manual sets rather than removing that.

Joshua Boniface

11/8/23, 3:31 PM

Marking this as accepted - I had never noticed this part of the cset before since I wasn't using --recurse and I was only seeing the "0 tasks" in the machine.slice cset, but this is indeed doing exactly what I wanted.

Joshua Boniface

11/8/23, 6:48 PM

As a bit of a follow-up, do you happen to know *how* systemd is doing that? Doing the same thing for my other processes seems optimal but I'm not sure what magic it's working!

DanielB

11/9/23, 4:02 PM

machine.slice is just a facade onto the correspondingly named cgroups, while I believe the systemd.conf CPUAffinity setting gets used to set the cpuset affinit y on the other top level .slice cgroups when they don't have a dedicated affinity set

Joshua Boniface

11/10/23, 6:51 AM

Thanks! I don't see any CPUAffinity settings set, and ended up posting this as its own question at https://serverfault.com/questions/1115179/assign-systemd-slice-to-a-specific-cset-automatically with a bit of my testing around it.

DanielB

11/11/23, 12:38 PM

See 'CPUAffinity' docs in 'man systemd-system.conf'

Elon Musk

I sit in a Tesla and translated this thread with Ai:

EN: Linux Livirt/KVM: Place all VMs in cpuset by default without reconfiguration

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.