What are the different versions available as Ubuntu cloud-images?...General Guide for Ubuntu Cloud Image use in Virt-manager 2021

in flag

I'm soooo thrilled to finally find prebuilt Ubuntu images. I was using osboxes, but the file size when converting is 500GBs and tedious to compact. Fresh installs are tedious, especially when 90% of people could use the default options or easily customize a from a central image!

Anyway, there are so many versions. What are the differences? Some are obvious: img, tar.gz, vmdk, ova. I'm assuming the azure vhd, vagrant, wsl are for those platforms. That squashfs might be a single file compacted version. And from the related question below, that kvm has a smaller kvm kernel. The lxd version is actually larger. What is the root version for?

Edit: I found what the root version is, see my self-reply below. I'm still having trouble setting one of these up in Virt-manager, discussion also below, so any help is appreciated. I can start a new question if someone wants to answer that separately.

EDIT2!: After much trial and searching, I was able to get the cloud-image VM working which was my primary goal. I even found and old answer to my primary question that helped me to that goal!

[   ] ubuntu-20.04-server-cloudimg-amd64-azure.vhd.manifest  2021-11-08 22:37   18K  Ubuntu Server 20.04 LTS (Focal Fossa) released builds
[   ]       2021-11-09 00:17  549M  Ubuntu Server 20.04 LTS (Focal Fossa) released builds
[   ] ubuntu-20.04-server-cloudimg-amd64-disk-kvm.img        2021-11-08 22:37  523M  Ubuntu Server 20.04 LTS (Focal Fossa) released builds
[   ] ubuntu-20.04-server-cloudimg-amd64-disk-kvm.manifest   2021-11-08 22:37   17K  Ubuntu Server 20.04 LTS (Focal Fossa) released builds
[   ] ubuntu-20.04-server-cloudimg-amd64-lxd.tar.xz          2021-11-08 22:42  832   Ubuntu Server 20.04 LTS (Focal Fossa) released builds
[   ] ubuntu-20.04-server-cloudimg-amd64-root.manifest       2021-11-08 22:40   17K  Ubuntu Server 20.04 LTS (Focal Fossa) released builds
[   ] ubuntu-20.04-server-cloudimg-amd64-root.tar.xz         2021-11-08 22:40  327M  Ubuntu Server 20.04 LTS (Focal Fossa) released builds
[   ]         2021-11-08 22:42  517M  Ubuntu Server 20.04 LTS (Focal Fossa) released builds
[   ] ubuntu-20.04-server-cloudimg-amd64-wsl.rootfs.manifest 2021-11-08 22:42   19K  Ubuntu Server 20.04 LTS (Focal Fossa) released builds
[   ] ubuntu-20.04-server-cloudimg-amd64-wsl.rootfs.tar.gz   2021-11-08 22:42  469M  Ubuntu Server 20.04 LTS (Focal Fossa) released builds
[   ] ubuntu-20.04-server-cloudimg-amd64.img                 2021-11-08 22:39  542M  Cloud image for 64-bit computers (USB image)
[   ] ubuntu-20.04-server-cloudimg-amd64.manifest            2021-11-08 22:40   17K  Cloud image for 64-bit computers (contents of live filesystem)
[   ] ubuntu-20.04-server-cloudimg-amd64.ova                 2021-11-08 22:40  525M  Cloud image for 64-bit computers (Open Virtualization Appliance)
[   ] ubuntu-20.04-server-cloudimg-amd64.squashfs            2021-11-08 22:41  371M  Ubuntu Server 20.04 LTS (Focal Fossa) released builds
[   ] ubuntu-20.04-server-cloudimg-amd64.squashfs.manifest   2021-11-08 22:41   17K  Ubuntu Server 20.04 LTS (Focal Fossa) released builds
[   ] ubuntu-20.04-server-cloudimg-amd64.tar.gz              2021-11-08 22:43  498M  Cloud image for 64-bit computers (Cloud Image/EC2 tarball)
[   ] ubuntu-20.04-server-cloudimg-amd64.vmdk                2021-11-08 22:42  525M  Ubuntu Server 20.04 LTS (Focal Fossa) released builds
br flag

I found solutions for some bugs. For some reason cloud-init doesn't start even though the cidata.iso is added.

To fix this up, the SMBIOS “serial” string parameter must be set. Source:

This example do not work for me and i found some bugreports at RH.:

$ virt-install --sysinfo type=smbios,system_serial=ds=nocloud

But this example works:

$ virt-install --qemu-commandline="-smbios type=1,serial=ds=nocloud" ...

Then cloudinit takes over as desired and sets all settings.

in flag

Okay, here's the guide I found that worked after two days of trial and error. Ubuntu would do well by their community to post some instructions for local users.

I was able to find the "vmlinuz-virtual-generic" version kernel in a buried "unpacked" folder in the cloud images releases link from above. The main page has other versions than for Focal This is the most important part because the initrd-userve.img wont recognize the root disk. You can also use a normal ubuntu kernel.

The gist of it is to use qemu-img create -b ubuntu-*-server-cloudimg-amd64.img -f qcow2 samename.img.qcow2 command to create the VM (or just use the GUI create in virt-mgr). It will give the option to configure before install and then start the VM. You'll want to add the kernel and initrd from the "unpacked" downloads to the Direct Boot section under Boot Options plus these boot args: ro init=/usr/lib/cloud-init/uncloud-init root=/dev/vda ds=nocloud ubuntu-pass=initialpassword. You have to boot twice for the password to work. Actually, just rw root=/dev/vda1 init=/bin/bash will work to create a new user and password.

Notice these instructions say vda. I had to use vda1 because there are two boot partitions vda14 for GPT/bios_boot and vda15 for EFI/ESP/boot-flagged partition. The original raw .img file uses sdaX, fyi.. converting to qcow2 or just using qemu-img is what changes that apparently.

A couple more details:

Older help pages say the kernel files and floppy disk are bundled in the tar file. This isn't the case anymore apparently and led me to more trial and error. The common errors are hanging on "GRUB_FORCE_PARTUUID attempting initrdless boot", "Error unknown command hwmatch" and "random: fast init done". Actually, the first two are when not using Direct Boot. The Ova import in VirtualBox does boot on it's own, so it's figuring out the internal kernel files somehow. I would love to know how to do that in KVM. I looked at the XML files but didnt see an obvious clue beside it setting up a floppy disk, putting me on the search to find the floppy which led me to the main guide page above. The third error I quoted is when these internal kernel files are not found, even though the fstab has correct lables for the boot drives. It doesnt even get to grub, so I cant specify disk UUIDs there, though they match in the grub file (obviously because using Direct Boot doesnt change any of that).

Here is an answer to my original question. Again based on the older tar contents with floppy, etc. It did lead me to looking deeper for the "vmlinuz-virtual" files: Where is the usage of Ubuntu cloud guest image on OpenStack documented?

Other resources:

The kernel args uncloud-init tell cloud-init it is a local VM and the nocloud arg sets up local networking (instead of having to go in and change /etc/netplan/*yaml to remove the fixed mac address, and use netplan apply and then systemctl restart networking)

Here is some more info I've learned:

Trying to set the cloud-init yaml files in an easy way with a local file led me to this solution for using cloud-init with an Ubuntu server iso: It has a few simple commands to create the files and then runs a neat little one-liner server to create the seedfrom net location. It installs and boots with Kvm. Unfortunately after using the raw image to create a VM in Virtmanager, it wouldnt boot because it couldnt find the /stillmounted/casper/initrd for some reason. So I copied the kernel files out of the mounted iso and tried using them (because I suspect the cloud image above doesnt have audio drivers in its kernel), but it also hung on the casper/initrd files.

So I went back to the VirtualBox ova. The my-seed.iso worked to change the server name. After a long search on how to pass kernel parameters (you cant beside a couple using vboxmanage setextradata), I rediscovered you can press shift while booting to edit the Grub args. After using init=/bin/bash to finally set the password and vga=792 to increase the display size, I found a bug in the image in the visudo file that the #include.. is missing a space which kills sudo. After getting apt to update, I found there is a VirtualBox bug that doesnt allow bidirectional clipboard in recent versions: But at least I finally have a way to create a basic Ubuntu VM without having to go through the whole install process. This method should work for many other distros. (Oh also, the touch /etc/cloud/cloud-init.disabled command above seems to work).

Followup tidbits:

  • The my-config.iso only works the on the first run. (unless there is a way to trigger cloud-init).
  • Hold shift to get into Grub. Use the kernel arg vga=792 for a larger terminal. Set grub timeout in /etc/default/grub and run update-grub.
  • Audio doesnt work on these cloud images without installing apt install linux-generic (about 800Mbs). And set audio output enabled in Vbox.

Last Update: I finally figured out the key to getting the cloud-image to boot without using the Direct Boot files. Basically, either install the non-kvm, generic kernel version using apt install linux-image-$(uname -r) and change kvm to generic (~90Mb), or use the kvm kernel and ssh into the VM using systemctl enable serial-getty@ttyS0.service && systemctl start serial-getty@ttyS0.service, then virtsh console <vm-name>. To get in initially to make a user use root=/dev/vda1 init=/bin/bash rw for the kernel args. Then delete or move two files /etc/grub.d/01.. and /etc/default/grub.d/50.., and comment out the first line of /etc/default/grub.d/40-force-partuuid.cfg. Change /etc/default/grub hidden to menu, adjust timeout, etc if desired. Then update-grub and it should work.

in flag

EDIT: See my other answer. The below is somewhat irrelevant now I have a working VM, beside the permissions issue I didnt solve when using virt-v2v. There is an Ubuntu specific version of that tool called virt-convert, but I ended up using qemu-img which succeeded without permissions issues.

The root version is literally the whole linux filesystem, ie boot, dev, etc..

/boot doesnt have any files, like grub. So I'm looking for the easiest version to create a Virt-manager VM from. Am I supposed to boot with direct boot and add rw init/=bin/bash to the root=/dev/sda1 kernel args to get a shell in initrd, and then somehow use grub-update to install grub? I could mount and copy over a /boot directory from my host linux, but it doesn't seem like this is the recommended way. I've also played with tagging the sda1 volume as boot on the img version using a Gparted live iso attached. I suppose I could use fdisk or cfdisk to do that. I wonder if it has an MBR.

I also tried to convert the ova version. I've tried the img version of Ubuntu Minimal and had to use direct kernel boot. I tried to get Virt-manager SeaBIOS to recognize the MBR, but there were 3 partitions (separate boot and esp/efi) and it errors with grub_force_partuuid attempting initrdless boot. Is it better to try connecting to the UEFI partition?

I just tried converting the ova using virt-v2v which succeeded, but I get this error Could not open '/var/lib/libvirt/images/ubuntu-focal-20.04-cloudimg-20211108-sda': Permission denied. So I tried chmodding it to 777 and tried running as root from the CLI virsh start ubuntu-focal-20.04-cloudimg-20211108 with the same error. The other images in that same directory run just fine, but I didnt use virt-v2v on them.

muru avatar
us flag
Why do you have two separate answer posts, when you know you can edit answers?
alchemy avatar
in flag
@muru, why do you care? no, offense, but do you have the time to edit all this? I provided a message in this first answer that it is somewhat irrelevant, meaning some is relevant, and it contains keywords of my journey that other may get search results on, as well as an unanswered question which I continue to ask in my second partial answer.. relax.. and/or do me a favor and upvote my second answer so it comes first.
Zanna avatar
kr flag
This seems like it would make more sense as either an extension to the question, or a separate question. It seems to me like your question has evolved and you've ended up writing more of a how to in your other answer - I wouldn't expect the title of the question (if I searched for it) to lead to the information you've ended up providing. The whole Q&A could be quite a bit clearer and might work better split into two... I'd be happy to help with editing but I don't want to tread on your toes... any thoughts?
alchemy avatar
in flag
@Zanna, thanks for your constructive tone :).. Yes, this is probably a common issue you 'editor' users run into. By the way, do you guys get monetary rewards or just increased rep for more power in return? Or are you independently wealthy 'angels'? ..I thought about renaming the question or starting another, but this is the first question many would have when trying to solve a larger problem. I've long advocated that all the SE pages create a topic to group similar issues. Tags kind of do that, but each individual question can usually fit into a larger topic. All link I listed could have..
alchemy avatar
in flag
..been grouped into that topic and it would have saved me from having to explain them. Although, the combined work is it's own guide. I actually left all of the 'journey' of the answer on the page to assist people in finding it from whatever starting point. Again, a topic tree would solve that. Wikipedia is a model of that. It would cut down on repeat questions. I think until that real solution happens, it's a waste of time to bandaid the problem by separating information which reduces it's usefulness. If you want to create a new more clear version, more power to ya! Thanks for all you do
alchemy avatar
in flag
A good way to institute a topic tree would be to make the tags organized into a tree. Then the placing of a question would follow naturally. The tags could be given a weighting that could determine if a topic was broad or specific overall. Then secondary tags could keep the information in labeling of the opposite scope (specific or broad), or related topics. Ontology trees like Wikidata, Freebase, Cyc or DBpedia could be used as trees. Tags could even be voted on for weighting to reduce centralize curation. Absolute titles become less relevant, as they can probably never convey as much info.
Zanna avatar
kr flag
I would not be able to write any version of your question / answer as I know nothing about the subject! There have been suggestions about tag hierarchies in the past... the main issue with making things *even more complicated* than they already are is that for the most part the SE model relies on a whole lot of un/self trained people to figure it all out and implement it. Also, tags are only useful for searching internally. Most people will come from search engines. Misleading titles hurt my brain because there's just no point having information if people can't find it. Thanks for fixing that.
Zanna avatar
kr flag
Monetary rewards for editing as if. People edit because they care about making the site useful. See also Users with less than 2000 reputation need to "suggest" edits and have them approved. If the suggestion is approved, one gets 2 reputation. Users with more than 2k can edit freely and don't gain reputation for it. I didn't edit much before I reached 2k as I hadn't learned markdown well. Most beginners don't know markdown so most editing is just making code readable by fixing formatting!
alchemy avatar
in flag
@Zanna, I see your contributions around often, so a big thank you! I only wish I had time to clean up the interwebs! It is appreciated. ...If the tags were visually present like in wikipedia, people would use them regardless of where they started searching. But I take your point about titles from that perspective. Where was the last progress made for tag hierarchy? Maybe I can move the ball forward a bit. (Oh, also I wanted to state the fact that I hate that people can edit my hardwork without having to ask. It should be flagged, not altered if it is problematic, unless it is grossly offensive

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.