Score:1

How do I prevent a faulty USB drive from freezing my Ubuntu server?

nz flag

I have an issue where a mounted external USB hard drive fails, this causes the IO Wait of the system to spike (I can see this in Grafana) but shortly after the system becomes completely unresponsive, so I cannot ssh into it to force a umount or to reboot.

I am presently mounting the drive like this:

UUID=UUID_HERE       /data/path/disk1       ext4    user,defaults,nofail,noatime,commit=60,x-systemd.device-timeout=120,errors=continue     0       2 # Added 2023-01-03 09:41:46 +0000

How can I prevent this from happening? - is there some fstab mount option to automatically unmount after a number of errors? - or to force a umount after a read/write timeout? - Or maybe if there is a read timeout to somehow kill power to the USB ports?

John Mahowald avatar
cn flag
Do you see I/O errors logged, or is the problem that performance is horribly slow?
Hackeron avatar
nz flag
I'm not entirely sure, it seems to just be extremely slow as sometimes I would get a Grafana update 20 hours later, but it just times out any attempts to ssh so I cannot test. Because these units are installed in physically inaccessible locations, what usually happens is an engineer time is scheduled and they just swap the drive and restart the unit a few weeks later. What am I after is some mechanism to automatically eject the malfunctioning drive so that I can at least ssh to the unit to investigate.
Rino Bino avatar
us flag
Could you write a local daemon that detects the io wait spike, then umounts the drive?
Hackeron avatar
nz flag
@RinoBino I tried that but there were too many false positives and also situations where at that point the system is so bogged down it just becomes too late. I was hoping for something maybe kernel level as user space I am struggling to find a solution.
I sit in a Tesla and translated this thread with Ai:

mangohost

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.