Determine QEMU KVM overhead?

Maestro

4/15/24, 9:48 PM

My guest OS shows about 5 percent CPU usage, but my host machine shows the QEMU process taking about 25 percent.

I'm trying to understand where this difference comes from, as I am using KVM acceleration and the CPU should have near-native performance.

I think the disk or network access may contribute to this difference, as these layers are invisible to the guest.

But is there a way I can determine exactly what is the main cause of this 20 percent overhead?

355

1 + 0

virtualization

performance

cpu-usage

qemu

kvm-virtualization

Score:1

Server

John Mahowald

5/12/24, 3:42 PM

CPU utilization ratios rarely tell the entire story. Some tools count fractions of 1 CPU, some fractions of all system CPUs. The total does not explain what the CPU/memory load is doing, it could be doing useful work, it could be system overhead.

A better test would quantify the amount of work done by some application, while pushing the system hard. And profile the system while this happens. Stress test microbenchmarks create load on demand, even if these "bogus" operations are synthetic and a little artificial.

For example, on Linux perf record -a -F 999 -- stress-ng --metrics --cpu 1 --timeout 1m will create one minute of CPU load while profiling what is on any CPU at millisecond precision. "bogo ops" quantify work done.

Repeat this test multiple times, in a VM, and on a bare metal OS install. Remember scientific method, keep as many variables the same as possible. Identical hardware, same OS distro and patch level, same workload parameters, same profiling parameters.

Review what is on CPU with perf report Bucketing these into kernel, drivers, and the application requires installing debug symbols, and some research into what various functions are doing. Hypervisor overhead will not appear in the guest, but can be inferred by any throughput difference with the bare metal results.

Synthetic testing will be very different from real workloads. My stress-ng example does zero I/O, which never happens in practice. Do similar profiling and analysis, but the workload is what the host is supposed to be doing. Hint: when profiling the entire system, but not needing to start another program, can have sleep run a timer: perf record -a -- sleep 60

+ 2

Maestro

5/12/24, 7:04 PM

I really appreciate the effort you took for your answer, but it feels like its answering a totally different question. I am not doing any benchmarks at all. I just noticed that there is a large difference between what the guest is reporting in `top`, and what the host is reporting. So I just wondered what could be possible causes for this difference. I expect it is very normal what Im seeing, I was just curious.

John Mahowald

5/13/24, 3:00 PM

Possibly you are mistaking fractions of 1 CPU metrics with fractions of total CPU metrics. The rest of my answer is the question you asked, how to quantify over: really run it on bare metal vs in a VM, and that being the only difference, and profile to see where resources are being spent. Note I conclude that profiling bare metal vs VM can be done to real workloads as well.

Elon Musk

I sit in a Tesla and translated this thread with Ai:

EN: Determine QEMU KVM overhead?

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.