Score:0

Kernel not updating with atp-get update/upgrade - Ubuntu 21.04 - TFTP boot NFS root

ru flag

I am gradually getting a Pi 4 cluster up and running, and have hit an issue I have not been able to resolve with google. It is my understanding of the Ubuntu boot process that is lacking really.

I have 5 nodes. Each is running Ubuntu 21.04. The first is a storage node, and runs from an SD card. It then has a couple of SSDs in RAID 1 split into 8 partitions. The first 4 partitions contain the root file systems for each of the other 4 nodes and are exported via NFS. Each node has a directory /srv/nfs/{RpiID}>/ where the partition is mounted. This storage node is also running TFTP and has a boot directory for each of the other 4 nodes at /srv/tftpboot/{RpiID}/. These directories are then bind mounted to /srv/nfs/{RpiID}/boot/firmware/ on the storage server.

NFS:

mark@rgd-strg-01:~$ ls /srv/nfs/ -la
total 24
drwxrwxrwx  6 root root 4096 Sep 27 14:11 .
drwxr-xr-x  5 root root 4096 Sep 28 12:30 ..
drwxr-xr-x 20 root root 4096 Sep 21 08:59 637d46a8
drwxr-xr-x 20 root root 4096 Sep 21 08:53 68fe97e5
drwxr-xr-x 20 root root 4096 Sep 21 09:04 727e3e34
drwxr-xr-x 20 root root 4096 Sep 21 09:11 ba061f16

/etc/exports:

mark@rgd-strg-01:~$ cat /etc/exports 
/srv/nfs/637d46a8/ *(insecure,rw,async,no_subtree_check,no_root_squash)
/srv/nfs/68fe97e5/ *(insecure,rw,async,no_subtree_check,no_root_squash)
/srv/nfs/727e3e34/ *(insecure,rw,async,no_subtree_check,no_root_squash)
/srv/nfs/ba061f16/ *(insecure,rw,async,no_subtree_check,no_root_squash)
/srv/tftpboot/ *(insecure,rw,async,no_subtree_check,no_root_squash)

TFTP:

mark@rgd-strg-01:~$ ls /srv/tftpboot/ -la
total 24
drwxr-xr-x 6 root root 4096 Sep 28 12:11 .
drwxr-xr-x 5 root root 4096 Sep 28 12:30 ..
drwxr-xr-x 3 root root 4096 Sep 28 15:24 637d46a8
drwxr-xr-x 3 root root 4096 Sep 29 09:04 68fe97e5
drwxr-xr-x 3 root root 4096 Sep 28 13:22 727e3e34
drwxr-xr-x 3 root root 4096 Sep 28 13:22 ba061f16

/etc/fstab:

mark@rgd-strg-01:~$ cat /etc/fstab  
LABEL=writable  /        ext4   discard,errors=remount-ro       0 1
LABEL=system-boot       /boot/firmware  vfat    defaults        0       1
/srv/tftpboot/637d46a8 /srv/nfs/637d46a8/boot/firmware  none    defaults,bind 0 0 
/srv/tftpboot/68fe97e5 /srv/nfs/68fe97e5/boot/firmware  none    defaults,bind 0 0
/srv/tftpboot/727e3e34 /srv/nfs/727e3e34/boot/firmware  none    defaults,bind 0 0
/srv/tftpboot/ba061f16 /srv/nfs/ba061f16/boot/firmware  none    defaults,bind 0 0

Here is examples of the cmdline.txt and other important files from a node:

cmdline.txt:

mark@rgd-strg-01:~$ cat /srv/tftpboot/68fe97e5/cmdline.txt 
net.ifnames=0 dwc_otg.lpm_enable=0 console=serial0,115200 console=tty1 root=/dev/nfs 
nfsroot=10.1.0.20:/srv/nfs/68fe97e5,tcp ip=dhcp elevator=deadline rootwait fixrtc rw 
cgroup_enable=cpuset cgroup_memory=1 cgroup_enable=memory

fstab:

mark@rgd-strg-01:~$ cat /srv/nfs/68fe97e5/etc/fstab 
10.1.0.20:/srv/nfs/68fe97e5   /       nfs     defaults,_netdev   0  0 
tmpfs   /tmp    tmpfs   defaults    0   0
tmpfs   /var/tmp    tmpfs   defaults    0   0
tmpfs   /var/run    tmpfs   defaults    0   0

My issue is that in this configuration (having played with the boot process that I do not fully understand to enable the network booting), everything seems to be working fine, but I am having some trouble with kernel updates on the 4 network nodes. After running apt-get upgrade I get the following message

Message screen shot

Yet after a reboot, I can see that the kernel has not been updated:

mark@rdg-clust-01:~$ uname -r
5.11.0-1007-raspi

How can I either correct this properly, or at least do it manually? Can anyone help me understand the boot process a little better? Here is the contents of the boot and firmware directories for a node

Boot:

mark@rgd-strg-01:~$ ls /srv/nfs/68fe97e5/boot/ -la
total 133404
drwxr-xr-x  4 root root     4096 Sep 30 06:19 .
drwxr-xr-x 20 root root     4096 Sep 21 08:53 ..
-rw-------  1 root root  5099714 Jul 28 12:42 System.map-5.11.0- 
1007-raspi
-rw-------  1 root root  5115999 Aug 23 07:05 System.map-5.11.0- 
1017-raspi
-rw-------  1 root root  5115843 Sep 21 14:56 System.map-5.11.0- 
1019-raspi
-rw-r--r--  1 root root   233406 Jul 28 12:42 config-5.11.0-1007- 
raspi
-rw-r--r--  1 root root   233767 Aug 23 07:05 config-5.11.0-1017- 
raspi
-rw-r--r--  1 root root   233790 Sep 21 14:56 config-5.11.0-1019- 
raspi
lrwxrwxrwx  1 root root       44 Sep 30 06:14 dtb -> dtbs/5.11.0- 
1019-raspi/./bcm2711-rpi-4-b.dtb
lrwxrwxrwx  1 root root       44 Sep 20 16:07 dtb-5.11.0-1007- 
raspi -> dtbs/5.11.0-1007-raspi/./bcm2711-rpi-4-b.dtb
lrwxrwxrwx  1 root root       44 Sep 21 08:56 dtb-5.11.0-1017- 
raspi -> dtbs/5.11.0-1017-raspi/./bcm2711-rpi-4-b.dtb
lrwxrwxrwx  1 root root       44 Sep 30 06:14 dtb-5.11.0-1019- 
raspi -> dtbs/5.11.0-1019-raspi/./bcm2711-rpi-4-b.dtb
drwxr-xr-x  7 root root     4096 Sep 30 06:14 dtbs
drwxr-xr-x  3 root root     4096 Sep 29 09:04 firmware
lrwxrwxrwx  1 root root       28 Sep 30 06:10 initrd.img -> 
initrd.img-5.11.0-1019-raspi
-rw-r--r--  1 root root 28854277 Sep 20 16:07 initrd.img-5.11.0- 
1007-raspi
-rw-r--r--  1 root root 31566190 Sep 21 08:56 initrd.img-5.11.0- 
1017-raspi
-rw-r--r--  1 root root 31593753 Sep 30 06:14 initrd.img-5.11.0- 
1019-raspi
lrwxrwxrwx  1 root root       28 Sep 30 06:10 initrd.img.old -> 
initrd.img-5.11.0-1017-raspi
lrwxrwxrwx  1 root root       25 Sep 30 06:10 vmlinuz -> vmlinuz- 
5.11.0-1019-raspi
-rw-------  1 root root  9464117 Jul 28 12:42 vmlinuz-5.11.0- 
1007-raspi
-rw-------  1 root root  9525235 Aug 23 07:05 vmlinuz-5.11.0- 
1017-raspi
-rw-------  1 root root  9526813 Sep 21 14:56 vmlinuz-5.11.0- 
1019-raspi
lrwxrwxrwx  1 root root       25 Sep 30 06:10 vmlinuz.old -> 
vmlinuz-5.11.0-1017-raspi

Firmware:

mark@rgd-strg-01:~$ ls /srv/nfs/68fe97e5/boot/firmware/ -la
total 79968
drwxr-xr-x 3 root root     4096 Sep 29 09:04 .
drwxr-xr-x 4 root root     4096 Sep 30 06:19 ..
-rw-r--r-- 1 root root     1024 Sep 16 13:46 README
-rw-r--r-- 1 root root    26914 Sep 16 13:46 bcm2710-rpi-2-b.dtb
-rw-r--r-- 1 root root    29031 Sep 16 13:46 bcm2710-rpi-3-b-plus.dtb
-rw-r--r-- 1 root root    28412 Sep 16 13:46 bcm2710-rpi-3-b.dtb
-rw-r--r-- 1 root root    26910 Sep 16 13:46 bcm2710-rpi-cm3.dtb
-rw-r--r-- 1 root root    49254 Sep 16 13:46 bcm2711-rpi-4-b.dtb
-rw-r--r-- 1 root root    48910 Sep 16 13:46 bcm2711-rpi-400.dtb
-rw-r--r-- 1 root root    49318 Sep 16 13:46 bcm2711-rpi-cm4.dtb
-rw-r--r-- 1 root root    20140 Sep 16 13:46 bcm2837-rpi-3-a-plus.dtb
-rw-r--r-- 1 root root    21009 Sep 16 13:46 bcm2837-rpi-3-b-plus.dtb
-rw-r--r-- 1 root root    20545 Sep 16 13:46 bcm2837-rpi-3-b.dtb
-rw-r--r-- 1 root root    19872 Sep 16 13:46 bcm2837-rpi-cm3-io3.dtb
-rw-r--r-- 1 root root     4638 Sep 16 13:46 boot.scr
-rw-r--r-- 1 root root    52456 Sep 16 13:46 bootcode.bin
-rw-r--r-- 1 root root      228 Sep 28 13:22 cmdline.txt
-rw-r--r-- 1 root root     1142 Sep 16 13:46 config.txt
-rw-r--r-- 1 root root     7314 Sep 16 13:46 fixup.dat
-rw-r--r-- 1 root root     5448 Sep 16 13:46 fixup4.dat
-rw-r--r-- 1 root root     3187 Sep 16 13:46 fixup4cd.dat
-rw-r--r-- 1 root root     8452 Sep 16 13:46 fixup4db.dat
-rw-r--r-- 1 root root     8454 Sep 16 13:46 fixup4x.dat
-rw-r--r-- 1 root root     3187 Sep 16 13:46 fixup_cd.dat
-rw-r--r-- 1 root root    10298 Sep 16 13:46 fixup_db.dat
-rw-r--r-- 1 root root    10298 Sep 16 13:46 fixup_x.dat
-rw-r--r-- 1 root root 22666147 Sep 16 13:46 initrd.img
-rw-r--r-- 1 root root     1559 Sep 16 13:46 overlay_map.dtb
drwxr-xr-x 2 root root    12288 Sep 16 13:46 overlays
-rw-r--r-- 1 root root  2952960 Sep 16 13:46 start.elf
-rw-r--r-- 1 root root  2228800 Sep 16 13:46 start4.elf
-rw-r--r-- 1 root root   793116 Sep 16 13:46 start4cd.elf
-rw-r--r-- 1 root root  3722504 Sep 16 13:46 start4db.elf
-rw-r--r-- 1 root root  2981192 Sep 16 13:46 start4x.elf
-rw-r--r-- 1 root root   793116 Sep 16 13:46 start_cd.elf
-rw-r--r-- 1 root root  4794472 Sep 16 13:46 start_db.elf
-rw-r--r-- 1 root root  3704808 Sep 16 13:46 start_x.elf
-rw-r--r-- 1 root root   515920 Sep 16 13:46 uboot_rpi_3.bin
-rw-r--r-- 1 root root   571832 Sep 16 13:46 uboot_rpi_4.bin
-rw-r--r-- 1 root root   558216 Sep 16 13:46 uboot_rpi_arm64.bin
-rw-r--r-- 1 root root 25580032 Sep 16 13:46 vmlinux
-rw-r--r-- 1 root root  9464117 Sep 16 13:46 vmlinuz

From this I can see that vmlinux and vmlinuz in the firmware directory are old, and I presume that these represent the kernel loaded at boot time. So I removed them and created a symbolic link on the node pointing /boot/firmware/vmlinuz to /boot/vmlinuz then I updated /boot/firmware/config.txt and set kernel=vmlinuz. after a reboot it still loads the old kernel. What do I need to do?

ru flag
cross posted to the raspberry Pi stack exchange https://raspberrypi.stackexchange.com/questions/132509/kernel-not-updating-with-atp-get-update-upgrade-ubuntu-21-04-tftp-boot-nfs-r?noredirect=1#comment225340_132509
mangohost

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.