NFS root file system

I am trying to get a remote root (and boot) NFS filesystem working correctly and I am hitting the limits of my knowledge. I wonder if someone can point me in the right direction.

Server

Both TFTP and NFS are served from a Linksys WRT 1900 ACS router running OpenWRT 19.07. This device also has an external SSD which is used as an overlay root file system.

https://openwrt.org/docs/guide-user/additional-software/extroot_configuration.

Each client has a root system file in this directory (Ubuntu 21.04)

/nfs/exports/tftp/

and a boot directory under

/tftp/tftpboot/

And here is the contents of /etc/exports:

/overlay/upper/nfs/exports/ *(rw,sync,no_subtree_check,no_root_squash) /overlay/upper/tftp/tftpboot/ *(rw,sync,no_subtree_check,no_root_squash)

If i do not add the /overlay/upper/ prefix then I can't mount the exported directory

Clients

Clients successfully PXE boot and load the operating system (Ubuntu 21.04), I can log in, SSH etc. However this is not without seeing errors during boot up, or if I try to do an apt-get upgrade.

The first error I see during boot up is this:

[Failed] Failed to start Remount Root and Kernel File System

then later I see

[Failed] Failed to mount /boot

yet once logged in /boot appears to be mounted.

Here is the fstab from a client

10.1.0.1:/overlay/upper/nfs/exports/tftp/68fe97e5 / nfs defaults,_netdev,nfsvers=4.1,proto=tcp 0 0

10.1.0.1:/overlay/upper/tftp/tftpboot/68fe97e5 /boot nfs defaults,_netdev,nfsvers=4.1,proto=tcp 0 0

tmpfs /tmp tmpfs defaults 0 0

tmpfs /var/tmp tmpfs defaults 0 0

tmpfs /var/run tmpfs defaults 0 0

(sorry about formatting)

And /boot/cmdline.txt

net.ifnames=0 dwc_otg.lpm_enable=0 console=serial0,115200 console=tty1 root=/dev/nfs nfsroot=10.1.0.1:/overlay/upper/nfs/exports/tftp/68fe97e5,tcp ip=dhcp elevator=deadline rootwait fixrtc rw cgroup_enable=cpuset cgroup_memory=1 cgroup_enable=memory

When I look at systemctl systemd-remount-fs.service I see this line

May 27 15:10:08 rdg-clust-01 systemd-remount-fs[317]: mount.nfs: an incorrect mount option was specified

Can anyone point out which option I am getting wrong?

Look at how this works in KickStart, JumpStart or the old RoboInstaller. The miniroot kernel image is served by TFTP and the file system is served by NFS. This is also how diskless UNIX clients worked in the old days. I'm pretty sure you can't serve an entire filesystem with TFTP. It's just not how the protocol is designed.

There are a number of online documents that can help guide you to a successful diskless system configuration:

https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/html/managing_storage_devices/setting-up-a-remote-diskless-system_managing-storage-devices

https://www.xmodulo.com/diskless-boot-linux-machine.html

http://mirror.internode.on.net/pub/ldp/en/Diskless-root-NFS-HOWTO/Diskless-root-NFS-HOWTO.pdf