Could anyone help me on this please?
behavior
I'm running several VMs with qemu-kvm and when creating some first few snapshots of a running VM, they are created super fast (roughly 10 seconds). However after a certain amount of snapshots, their creation performance suddenly takes a huge hit, dropping from roughly 10 seconds to 10 or 20 minutes (depending on the size of the VM's qcow2 disk).
Interestingly this increase in duration does not grow gradually at all (e.g. first one would take 10 seconds, second 60, third a few minutes), but very sharply at some point from one snapshot creation to the next. I could not figure out due to what circumstances this increase happened, and I had tried to create one snapshot right after the other where the VM would be in the very same state, to rule out any factors coming from within the VM. But while the state of the VM over various snapshots stayed mostly identical, the duration of snapshot creation exploded at some point.
One VM has roughly 50GB utilized, this drop in performance happened after 4 or 5 snapshots, while another VM with a fresh install of ubuntu in it, just a few GB utilized, this drop happened after 19 snapshots.
Also this setup had worked on my previous linux host flawlessly, which was linux mint. Now I've changed to fedora silverblue and this problem has occured.
what I've tried so far
I've created the snapshots either with the virt-manager gui or with virsh like so (with date before and after to indicate the duration), note that this is a small VM, the bigger one takes 20 minutes:
[root@fedora-silverblue steff]# date; virsh --debug=0 snapshot-create-as --domain dev_base --name s25 ; date
Wed 16 Mar 09:52:08 CET 2022
snapshot-create-as: domain(optdata): dev_base
snapshot-create-as: name(optdata): s25
snapshot-create-as: found option <domain>: dev_base
snapshot-create-as: <domain> trying as domain NAME
Domain snapshot s25 created
Wed 16 Mar 10:03:02 CET 2022
this command takes 10 minutes.
I've searched for this issue and found following discussions which indicate similar problems but were solved with solutions that didn't work for me:
This post concludes that it was fixed by qemu in another version, but that version is far older than my current one:
https://bugzilla.redhat.com/show_bug.cgi?id=988436
This post solved it by deleting all pre-existing snapshots, and that did work for me too. However deleting all snapshots before defeats the purpose for which I'm using them (being able to revert to previous functioning states), so this is not an option:
https://github.com/cuckoosandbox/cuckoo/issues/1905
unsolved and open bug:
https://bugs.launchpad.net/ubuntu/+source/qemu-kvm/+bug/741887
Since some other discussions indicate something about caching, I've also tried turning off my host and on again, with no changes whatsoever.
Some posts discuss external snapshots, but I want them to be ruled out, as their handling is more of an overhead than internal snapshots (creating profiles, defining persistence paths, etc). And since internal snapshots are the default option, I want to stay as close to vanilla setups as possible.
interpretation
Since snapshots are created fast when no other exist beforehand, the pre-existence of other snapshots must introduce some heavy overhead or conflict somewhere.
software & versions
I'm running fedora silverblue 35 with the following kvm related software:
[root@fedora-silverblue steff]# uname -r
5.16.9-200.fc35.x86_64
[root@fedora-silverblue steff]# rpm -q qemu qemu-kvm libvirt virt-manager
qemu-6.1.0-14.fc35.x86_64
qemu-kvm-6.1.0-14.fc35.x86_64
libvirt-7.6.0-5.fc35.x86_64
virt-manager-3.2.0-4.fc35.noarch
The VMs have 8GB ram and their image formats are qcow2
questions
- Any idea what could be the cause?
- Where can I find more detailed debug info?
