Score:0

Changing the kernel for 20.04 in AWS appears to break boot sequence (may be a grub config issue?)

tr flag

Using Ubuntu 20.04 on AWS (ec2), I wanted to change my kernel from AWS to a generic one. When I try it the suggested way with changing the Grub Config, it stops the machine.

This is because Jibri (Jitsi Video Recorder) requires the use of ALSA and to modprobe snd-aloop , but anything that would want a lowlatency or different kernel would have the same need - wanting to use a different kernel than the AWS one.

What I tried:

Grab the image:

sudo apt install linux-image-extra-virtual

You’ll see it reference something like Linux 5.4.0-84-generic in the install packages, it helps later on. Now, list your boot entries that you have for that new generic kernel

grep -A200 submenu /boot/grub/grub.cfg |grep -P '^(?=.*menuentry)(?=.*generic)'

I see:

menuentry 'Ubuntu, with Linux 5.4.0-84-generic' --class ubuntu --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-5.4.0-84-generic-advanced-e8070c31-bfee-4314-a151-d1332dc23486' { menuentry 'Ubuntu, with Linux 5.4.0-84-generic (recovery mode)' --class ubuntu --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-5.4.0-84-generic-recovery-e8070c31-bfee-4314-a151-d1332dc23486' {

In what is likely the top line, and not the one that mentions recovery mode, copy it’s menuentry at the start of the line. On mine looks like Ubuntu, with Linux 5.4.0-84-generic , but it will be different for everyone.

Now, I will edit /etc/default/grub ,

Comment out the GRUB_DEFAULT=0 (loads the first entry in your grub bootlist), and change it to the string we found earlier. For me, this section of the file now looks like:

#GRUB_DEFAULT=0
GRUB_DEFAULT="Ubuntu, with Linux 5.4.0-84-generic"

This is not a proper setup yet - we are using it to let grub check and tell us the best way to do it.

At this point, we can sudo update-grub, which will give me a friendly warning like this (this is good!)

Warning: Please don't use old title `Ubuntu, with Linux 5.4.0-84-generic' for GRUB_DEFAULT, use `Advanced options for Ubuntu>Ubuntu, with Linux 5.4.0-84-generic' (for versions before 2.00) or `gnulinux-advanced-e8070c31-bfee-4314-a151-d1332dc23486>gnulinux-5.4.0-84-generic-advanced-e8070c31-bfee-4314-a151-d1332dc23486' (for 2.00 or later)

Again, this is good! We know that we’ve specified something it recognises. Everyone is running grub newer than 2.00 these days, so we now edit /etc/default/grub again and change it to it’s final value:

(my values will be different to yours)

#GRUB_DEFAULT=0
#GRUB_DEFAULT="Ubuntu, with Linux 5.4.0-84-generic"
GRUB_DEFAULT="gnulinux-advanced-e8070c31-bfee-4314-a151-d1332dc23486>gnulinux-5.4.0-84-generic-advanced-e8070c31-bfee-4314-a151-d1332dc23486"

sudo update-grub , and I see the warning is gone now.

Now I do a sudo reboot now

At this point, grub has given me a config that should try and boot the generic kernel, falling back to the normal one.

Instead it gets stuck in a boot loop.

In the serial console, I see this during it's boot loop repeatedly.

[    0.000000] Linux version 5.4.0-84-generic (buildd@lgw01-amd64-050) (gcc version 9.3.0 (Ubuntu 9.3.0-17ubuntu1~20.04)) #94-Ubuntu SMP Thu Aug 26 20:27:37 UTC 2021 
(Ubuntu 5.4.0-84.94-generic 5.4.133)
[    0.000000] Command line: BOOT_IMAGE=/boot/vmlinuz-5.4.0-84-generic root=PARTUUID=5198cbc0-01 ro console=tty1 console=ttyS0 nvme_core.io_timeout=4294967295 panic=-
1
[    0.000000] KERNEL supported cpus:
[    0.000000]   Intel GenuineIntel
[    0.000000]   AMD AuthenticAMD
[    0.000000]   Hygon HygonGenuine
[    0.000000]   Centaur CentaurHauls
[    0.000000]   zhaoxin   Shanghai  
... lots of cpu init that's hard to copy ..
[   10.366218] rtc_cmos 00:00: setting system clock to 2021-09-21T11:25:30 UTC (1632223530)
[   10.373185] md: Waiting for all devices to be available before autodetect
[   10.460173] md: If you don't use raid, use raid=noautodetect
[   10.464642] md: Autodetecting RAID arrays.
[   10.468024] md: autorun ...
[   10.470867] md: ... autorun DONE.
[   10.473906] VFS: Cannot open root device "PARTUUID=5198cbc0-01" or unknown-block(0,0): error -6
[   10.562806] Please append a correct "root=" boot option; here are the available partitions:
[   10.569825] Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(0,0)
[   10.674637] CPU: 1 PID: 1 Comm: swapper/0 Not tainted 5.4.0-84-generic #94-Ubuntu
[   10.682527] Hardware name: Amazon EC2 t3.small/, BIOS 1.0 10/16/2017
[   10.686735] Call Trace:
[   10.691100]  dump_stack+0x6d/0x8b
[   10.759061]  panic+0x101/0x2e3
[   10.761974]  mount_block_root+0x23f/0x2e8
[   10.765568]  mount_root+0x38/0x3a
[   10.768584]  prepare_namespace+0x13f/0x194
[   10.771873]  kernel_init_freeable+0x23f/0x263
[   10.775368]  ? rest_init+0xb0/0xb0
[   10.859141]  kernel_init+0xe/0x110
[   10.862101]  ret_from_fork+0x35/0x40
[   10.865843] Kernel Offset: 0x36a00000 from 0xffffffff81000000 (relocation range: 0xffffffff80000000-0xffffffffbfffffff)

It's telling me that it can't find the boot device or partuuid according to it. At this point, I pretty much have to terminate the instance and try again, as I can't recover it easily.

Is there some better way to change my kernel to the generic one? I'd be happy with getting to use the generic image in anyway possible at this point if it boots.

cn flag
I doubt you are allowed to it like this. The correct method is to create a NEW instance and attach your personal disk to the new instance. Kernel, cpu and memory are all tied to settings you create the instance with.
tr flag
That seems a bit circular doesn't it though? you would need to be able to create the instance from an existing AMI (ie start with a kernel of Ubuntu 20.04, and then be able to boot that kernel) before you could create an AMI out of it with changing the kernel to boot with..
cn flag
Nope. AWS provides the instances from their consoles with a custom kernel. I doubt you are allowed and doubt it is possible to do what you want. The monthly fee you pay for using AWS is tied with the cpu, gpu, kernel and OS version. I -am- making some assumptions though (otherwise I'd make it an answer ;-) ); I use mostly GCEand have only 3 AWS and those 3 are a bit dated.
tr flag
That isn't the case, there are numerous guides in the past that have changed the kernel, including on AWS. To link to an example: https://github.com/jitsi/jibri/blob/master/README.md#alsa-and-loopback-device (linked from an AWS install guide, it's still relevant for ubuntu 18.04 and 20.04). The strange part is I have even done this back in Feburary (running a generic kernel), but can't appear to do it again. I get left with a non booting instance.
Score:0
cn flag

As stated in log trace :

FS: Cannot open root device "PARTUUID=5198cbc0-01" or unknown-block(0,0): error -6
[   10.562806] Please append a correct "root=" boot option; here are the available partitions:
[   10.569825] Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(0,0)

The problem mainly lies in forced PARTUUID set in /etc/default/grub.d/40-force-partuuid.cfg

The solution is to :

  1. Install specific linux-generic kernel (sudo apt install linux-generic for the latest). You can check what you have available with command linux-version list
  2. change GRUB_DEFAULT="1>2" or accordingly to your grub.cfg menu order list.

("1>2") Explanation: Number 1 is for "Advanced Ubuntu Options" (second menuentry starting from zero) and number 2 is third submenu option in "Advanced Ubuntu Options" (starting from zero) which is generic kernel that should be used in my case.

  1. Comment out line GRUB_FORCE_PARTUUID in /etc/default/grub.d/40-force-partuuid.cfg
  2. sudo update-grub && sudo update-grub2
  3. Reboot
ru flag
Nice. Saved my bacon.
mangohost

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.