I have a ZFS dataset that exists in /proc/mounts and in /etc/mtab, but the folder doesn't exist in the filesystem, zfs umount reports the dataset doesn't exist, and zfs mount reports that the dataset is already mounted.
This is on RHEL 8.8 (fully updated) running zfs dkms 2.1.11-1
Is it possible to umount this dataset without rebooting the server? How would I even begin to troubleshoot this sort of problem?
I believe this occurred when it had the sharenfs setting enabled on the dataset and then a zfs umount was attempted while the nfs daemon was sharing. I have tried setting sharenfs=off (which succeeds, but doesn't fix the problem). The nfs services have been shut down and the problem still occurs. I can zfs send the datasets to another system and verify they're intact, but I just need them mounted on this server.
I am not using containers or additional mount namespaces (that I'm aware of).
The mounts are listed in /etc/mtab and /proc/mounts
topher@nzxt:/$ grep Backups -H /etc/mtab /proc/mounts
/etc/mtab:array/Backups /array/Backups zfs rw,seclabel,nosuid,nodev,noexec,noatime,xattr,noacl 0 0
/etc/mtab:array/Backups/gwen /array/Backups/gwen zfs rw,seclabel,nosuid,nodev,noexec,noatime,xattr,noacl 0 0
/proc/mounts:array/Backups /array/Backups zfs rw,seclabel,nosuid,nodev,noexec,noatime,xattr,noacl 0 0
/proc/mounts:array/Backups/gwen /array/Backups/gwen zfs rw,seclabel,nosuid,nodev,noexec,noatime,xattr,noacl 0 0
The directory doesn't exist in the filesystem
topher@nzxt:/$ ls /array/Backups
ls: cannot access '/array/Backups': No such file or directory
The dataset do exist in zfs (pool is array)
topher@nzxt:/$ zfs list array/Backups array/Backups/gwen
NAME USED AVAIL REFER MOUNTPOINT
array/Backups 2.62T 1.59T 1.71T /array/Backups
array/Backups/gwen 364G 1.59T 364G /array/Backups/gwen
Umounting (lazy, forced, or otherwise) fails.
topher@nzxt:/$ sudo zfs umount array/Backups
cannot unmount '/array/Backups/gwen': no such pool or dataset
topher@nzxt:/$ sudo zfs umount array/Backups/gwen
cannot unmount '/array/Backups/gwen': no such pool or dataset
topher@nzxt:/$ sudo umount /array/Backups/gwen
umount: /array/Backups/gwen: no mount point specified.
topher@nzxt:/$ sudo umount /array/Backups
umount: /array/Backups: no mount point specified.
I'm unable to export the zfs pool either.
topher@nzxt:~$ sudo zpool export array
cannot unmount '/array/Backups/gwen': unmount failed
Attempts to remount the directory report that it's already mounted
topher@nzxt:/$ sudo zfs mount array/Backups
cannot mount 'array/Backups': filesystem already mounted
topher@nzxt:/$ sudo zfs mount array/Backups/gwen
cannot mount 'array/Backups/gwen': filesystem already mounted
lsof throws errors attempting to walk the affected directory and does not return any open filehandles in those paths
topher@nzxt:/$ sudo lsof | grep Backups
lsof: WARNING: can't stat() zfs file system /array/Backups
Output information may be incomplete.
lsof: WARNING: can't stat() zfs file system /array/Backups/gwen
Output information may be incomplete.
fuser doesn't think the mount exists
topher@nzxt:~$ sudo fuser -vm /array/Backups
Specified filename /array/Backups does not exist.
topher@nzxt:~$ sudo fuser -vm /array/Backups/gwen
Specified filename /array/Backups/gwen does not exist.
