Join Log in
Score:0
avatar Ubuntu

Systemd fails to mount some disks during boot

do flag
tormentum

I have a system with a lot of disks (roughly 96 data drives, formatted xfs) running Ubuntu 20.04 LTS. During boot, systemd manages to mount most of the drives, but fails to fully last 20 or so mounts listed in the /etc/fstab.

The system has multiple 24 bay SAS shelves chained together and I mount using /mnt/shelf/shelf#/disk# referencing a device by label (eg: /dev/disk/by-label/disk-###) in the fstab file. The fstab file is very long but all the data disk mount lines are the same, looking something like this:

*snip*
/dev/disk/by-label/disk-501 /mnt/shelf/5/501 xfs defaults 0 0
/dev/disk/by-label/disk-502 /mnt/shelf/5/502 xfs defaults 0 0
/dev/disk/by-label/disk-503 /mnt/shelf/5/503 xfs defaults 0 0
*snip*

Strangely, it's always the disks toward the bottom of the fstab that fail. When I look at the syslog, I have the following entries that appear relevant:

Nov 26 09:13:16 h2 systemd[1]: Found device HUS724040ALS640 disk-092.
Nov 26 09:13:16 h2 systemd[1]: Found device HUS724040ALS640 disk-096.
Nov 26 09:13:16 h2 systemd[1]: Mounting /mnt/shelf/4/092...
Nov 26 09:13:16 h2 systemd[1]: Mounting /mnt/shelf/4/096...
Nov 26 09:13:16 h2 systemd[1]: Mounted /mnt/shelf/1/014.
Nov 26 09:13:16 h2 systemd[1]: Mounted /mnt/shelf/3/069.
Nov 26 09:13:16 h2 systemd[1]: Found device HUS724040ALS640 disk-088.
Nov 26 09:13:16 h2 systemd[1]: Mounting /mnt/shelf/4/088...
Nov 26 09:13:16 h2 systemd[1]: Mounted /mnt/shelf/4/092.
Nov 26 09:13:16 h2 systemd[1]: Mounted /mnt/shelf/4/096.
*snip*
Nov 26 09:13:16 h2 systemd[1]: mnt-shelf-4-073.mount: Job mnt-shelf-4-073.mount/start failed with result 'dependency'.
Nov 26 09:13:16 h2 systemd[1]: dev-disk-by\x2dlabel-disk\x2d073.device: Job dev-disk-by\x2dlabel-disk\x2d073.device/start failed with result 'timeout'.
Nov 26 09:13:16 h2 systemd[1]: dev-disk-by\x2dlabel-disk\x2d074.device: Job dev-disk-by\x2dlabel-disk\x2d074.device/start timed out.
Nov 26 09:13:16 h2 systemd[1]: Timed out waiting for device /dev/disk/by-label/disk-074.
Nov 26 09:13:16 h2 systemd[1]: Dependency failed for /mnt/shelf/4/074.
Nov 26 09:13:16 h2 systemd[1]: mnt-shelf-4-074.mount: Job mnt-shelf-4-074.mount/start failed with result 'dependency'.

Once booted, I get around this issue by manually mounting the disks that failed during boot.

I believe there is some sort of systemd startup ordering issue (maybe a missing fstab option I can use?) or dependency issue that is causing systemd to timeout waiting for the disk devices to properly enumerate fully.

Any ideas on where I can begin to look here?

109
0 + 0
user10489 avatar
in flag
user10489
does `mount -a` mount the missing drives? It might just be timing out mounting so many. It should be possible to find the systemd mount target and increase the timeout.
0
Reply
do flag
tormentum
It actually doesn't. I have to comment out the affected drives in the fstab, run `systemctl daemon-reload`, uncomment the lines and then `mount -a`
0
Reply
user10489 avatar
in flag
user10489
That makes me wonder if there's a hard coded max number of mounts in systemd, although it still could be a timeout.
0
Reply
ธงชาติไทย
Elon Musk
I sit in a Tesla and translated this thread with Ai:

mangohost

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.