How to use more then 2GB of ram when using running a nested BIOS VM using KVM

nezu

3/4/24, 7:38 PM

I have a Linux host running Debian 11 and on that host, I have a Ubuntu 22.04 VM. Inside that VM I want to run a Windows 10 VM using nested virtualization. The ubuntu VM is booted using EFI and nested virtualization works well.

When the windows VM running inside of the Ubuntu VM is also booted using EFI everything works fine, all the ram I give it is detected and usable, and the VM has good performance so nested virtualization works fine.

My problem is that snapshots are not (yet) supported in vibvirt when using EFI so I have to use BIOS. However, when I change the VM to bios everything works fine except the fact that now the guest only sees 2GB of ram. I've tried booting Linux and memtest in that VM and they both only see the 2GB. I've tried creating new VMs and they also exhibit the exact same behavior (works fine when booted in EFI but not in BIOS)

I've copied the VM disk and configuration 1:1 to the host and it detects the entire 8GB (or whatever I set it to) just fine. So this issue only appears when running nested and using BIOS.

At this point, I've tried many different configurations even running super simple qemu commands, and still can't break the 2GB barrier when running nested without EFI for some reason.

Is there some limitation that I'm not aware of?

software versions (in the nested Ubuntu VM):
QEMU emulator version 7.2.0
libvirtd (libvirt) 9.0.0
Linux 5.19.0-35-generic

312

1 + 4

bios

libvirt

qemu

kvm-virtualization

anx

3/4/24, 9:09 PM

Do you actually need to care, though? Does libvirt not simply snapshot the first drive? If you configure the guest to provision NVRAM-ignorant (`ESP:/efi/boot/bootx64.efi`) loader, then tianocore/ovmf KVM guests do not need their pflash writable/snapshotted/restored (and they cannot require that, how else would "cloud images" work?)

nezu

3/4/24, 9:19 PM

@anx I have a piece of software that utilizes the snapshot functionality in libvirt. I don't care about NVRAM being writable, the default works fine. Is there a way I can configure this in libvirt so that snapshots work in EFI mode? I couldn't find anything on this last time I looked. The only thing people mentioned was making snapshots of the disk manually.

anx

3/4/24, 9:52 PM

Depends on how exactly they don't "work" right now I guess? Most missing automation around snapshottable firmware was since added, and could for your simpler case (not needing persistent firmware) already been harmless for stable versions. ([add](https://serverfault.com/posts/1125310/edit) qemu/libvirt versions, a not-so-far future answer to your question might say "upgrade X to vY.Z then both works")

nezu

3/4/24, 10:41 PM

> Depends on how exactly they don't "work" Snapshots do work when I power off the VM but don't When It's power on. `Error creating snapshot: Operation not supported: internal snapshots of a VM with pflash based firmware are not supported`.

Score:0

Server

David GG

3/4/24, 8:25 PM

Yes, there is a limitation when using nested virtualization with BIOS mode. The issue you're experiencing is likely due to the fact that the BIOS firmware in the virtual machine is not capable of exposing more than 2GB of RAM to the guest operating system.

This is a known limitation of BIOS-based virtualization and is caused by the way the BIOS firmware maps memory. In contrast, the UEFI firmware used in modern virtual machines supports memory-mapped I/O, which allows the guest operating system to access more memory.

To work around this limitation, you can either continue using nested virtualization with EFI mode, or switch to using a different virtualization technology that supports snapshots with EFI mode. Alternatively, you can try adjusting the memory settings in the BIOS firmware of the virtual machine, although this may not always be possible or effective.

In summary, the limitation of BIOS-based virtualization with nested virtualization is that the guest operating system may only be able to access up to 2GB of RAM.

+ 1

nezu

3/4/24, 9:15 PM

Do you have any references for that? Why does it only happen when running nested?

Elon Musk

I sit in a Tesla and translated this thread with Ai:

EN: How to use more then 2GB of ram when using running a nested BIOS VM using KVM

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.