Raspberry Pi4B / (64bit 21.04): 5.11 kernel tree's -1014-raspi / 1015-raspi apt update kills the OS

I've now got two of my RPi4B installations (Xubuntu, 64bit) of 21.04 into a non-bootable state after a normal apt update with the kernel tree update to 5.11.0-1014-raspi or 5.11.0-1015-raspi. I made the first error report on that thing a week or so back when the first one of my RPi4B builds booted into the Rainbow of Death and I warned the devs that there is an issue with that. I thought it'd been safe to update on another RPi4B running the same OS, now it's booting to the underscore of doom.

How can I fix the bootloader, if at all? Any help is appreciated. I have no clue whatsoever what goes into the whole uboot segment and so forth, and I sure as hell wouldn't want to reinstall the entire OS from scratch.

The first unbootable unit shows absolutely nothing going wrong in /var/log/ , here's the last of the logs from that one: https://paste.ubuntu.com/p/SXnzTTsNtS/

My main question is; can I somehow bring the bootloader back to life, since that is what seems to be the problem here? When the first unit got zapped because of an apt-update, I did copy the boot partition data from the then-still-working one and tried rsyncing the data, to no avail.

There is absolutely no documentation on how Ubuntu's Uboot works on RPi4B's, at least I haven't found any. If someone can point me to the Raspberry Pi 4B U-boot documentation, I would be very happy to read it. Now I'm just out of luck with two non-functional OS builds that worked perfectly well before the routine kernel update.

Please advice. Thanks.

UPDATE 1: I got a fresh USB stick, flashed Ubuntu Server 64-bit for the RPi4B with rpi-imagerand booted into a fresh system. After running apt updates and upgrades, it too updated to the -1015-raspi, after which I rebooted and it went itno the "rainbow of death" with four blinks in a row (which would indicate that a bootloader was not found).

So, another one bites the dust. I was hoping to maybe be able to get a working bootloader that way, but no such luck.

UPDATE 2: There's now a bugreport at https://bugs.launchpad.net/ubuntu/+source/linux-meta-raspi/+bug/1937924

UPDATE 3: SOLVED I solved it on my own without having to reinstall by waiting for a new apt update on the kernel packages and then just copied the boot partition data from a fresh install to the broken OS's one. (See the answer below.)

Ubuntu boots from fat32 partition of sd-card. During flashing the new kernel, the old one's files are saved with .bak extension. So, just rename them using another Linux.

1. Mount your boot partition (/dev/sda1 in my case)

sudo mount /dev/sda1 /mnt

2. Rename all files with .bak suffix.

sudo rename -f -v 's/.bak//g' /mnt/*.bak
sudo rename -f -v 's/.bak//g' /mnt/overlays/*.bak

Done!

UPD: If the reason for your issue is too small boot partition this approach may not helps you.

I got this fixed on my own by just waiting for a new kernel update to be out on the 64bit Ubuntu on RPi4B. I wrote the latest image of Ubuntu Server 64bit on to a USB stick using rpi-imager and then ran the initial setup on the RPi4B on it. I then ran an install on all the available apt update / apt upgrade packages, and made sure it was booting without problems after these updates.

I then copied all the boot-partition's files from the healthy new Ubuntu Server to the damaged OS's boot partition, excluding meta-data, network-config and user-data files as well as the cmdline.txt and config.txt.

Plugged in and prayed, after a while of fscking in the four-dotted (. . . .) Ubuntu screen, both of the OS's came back to life! I am extremely happy now.

So yep, it was a bootloader problem like I thought. Just a word of advice to keep an eye out on this potential pitfall, lest you end up in the same type of limbo after a kernel-related apt update. Back up often! :o)