4

I have a ZFS dataset that exists in /proc/mounts and in /etc/mtab, but the folder doesn't exist in the filesystem, zfs umount reports the dataset doesn't exist, and zfs mount reports that the dataset is already mounted.

This is on RHEL 8.8 (fully updated) running zfs dkms 2.1.11-1

Is it possible to umount this dataset without rebooting the server? How would I even begin to troubleshoot this sort of problem?

I believe this occurred when it had the sharenfs setting enabled on the dataset and then a zfs umount was attempted while the nfs daemon was sharing. I have tried setting sharenfs=off (which succeeds, but doesn't fix the problem). The nfs services have been shut down and the problem still occurs. I can zfs send the datasets to another system and verify they're intact, but I just need them mounted on this server.

I am not using containers or additional mount namespaces (that I'm aware of).

The mounts are listed in /etc/mtab and /proc/mounts

topher@nzxt:/$ grep Backups -H /etc/mtab /proc/mounts /etc/mtab:array/Backups /array/Backups zfs rw,seclabel,nosuid,nodev,noexec,noatime,xattr,noacl 0 0 /etc/mtab:array/Backups/gwen /array/Backups/gwen zfs rw,seclabel,nosuid,nodev,noexec,noatime,xattr,noacl 0 0 /proc/mounts:array/Backups /array/Backups zfs rw,seclabel,nosuid,nodev,noexec,noatime,xattr,noacl 0 0 /proc/mounts:array/Backups/gwen /array/Backups/gwen zfs rw,seclabel,nosuid,nodev,noexec,noatime,xattr,noacl 0 0

The directory doesn't exist in the filesystem

topher@nzxt:/$ ls /array/Backups ls: cannot access '/array/Backups': No such file or directory

The dataset do exist in zfs (pool is array)

topher@nzxt:/$ zfs list array/Backups array/Backups/gwen NAME USED AVAIL REFER MOUNTPOINT array/Backups 2.62T 1.59T 1.71T /array/Backups array/Backups/gwen 364G 1.59T 364G /array/Backups/gwen

Umounting (lazy, forced, or otherwise) fails.

topher@nzxt:/$ sudo zfs umount array/Backups cannot unmount '/array/Backups/gwen': no such pool or dataset topher@nzxt:/$ sudo zfs umount array/Backups/gwen cannot unmount '/array/Backups/gwen': no such pool or dataset topher@nzxt:/$ sudo umount /array/Backups/gwen umount: /array/Backups/gwen: no mount point specified. topher@nzxt:/$ sudo umount /array/Backups umount: /array/Backups: no mount point specified.

I'm unable to export the zfs pool either.

topher@nzxt:~$ sudo zpool export array cannot unmount '/array/Backups/gwen': unmount failed

Attempts to remount the directory report that it's already mounted

topher@nzxt:/$ sudo zfs mount array/Backups cannot mount 'array/Backups': filesystem already mounted topher@nzxt:/$ sudo zfs mount array/Backups/gwen cannot mount 'array/Backups/gwen': filesystem already mounted

lsof throws errors attempting to walk the affected directory and does not return any open filehandles in those paths

topher@nzxt:/$ sudo lsof | grep Backups lsof: WARNING: can't stat() zfs file system /array/Backups Output information may be incomplete. lsof: WARNING: can't stat() zfs file system /array/Backups/gwen Output information may be incomplete.

fuser doesn't think the mount exists

topher@nzxt:~$ sudo fuser -vm /array/Backups Specified filename /array/Backups does not exist. topher@nzxt:~$ sudo fuser -vm /array/Backups/gwen Specified filename /array/Backups/gwen does not exist.
Improve this question
6
  • 1
    Can you export the zpool? Or are there are other, still-mounted, filesystems on it that you need to leave mounted? Commented Jun 3, 2023 at 18:32
  • Yes, there are other datasets in use. Let me see if I can move those services elsewhere to test exporting. Commented Jun 3, 2023 at 21:47
  • Export fails with umount error. :/ Updating question. Commented Jun 3, 2023 at 22:03
  • What happens if you try to mount the filesystem? If it succeeds, can you then unmount it successfully? Commented Jun 4, 2023 at 3:20
  • Attempts to mount say that it's already mounted... D: Commented Jun 4, 2023 at 6:16

1 Answer 1

Reset to default
3

Try a lazy or forced umount and see what happens?

umount -l or umount -f

See: https://github.com/openzfs/zfs/issues/2435 and Linux mountpoint can not be unmounted no matter what?

Improve this answer
6
  • 1
    Sorry I didn't mention that neither of these worked. I'll amend my question. Commented Jun 3, 2023 at 16:23
  • Can you just reboot? Commented Jun 3, 2023 at 20:02
  • 1
    If there are stale handles, what does showmount or netstat -anp | grep 2049 show? Commented Jun 3, 2023 at 20:03
  • samba and nfsd are off, nothing listening on those ports. showmount predictably times out after a minute with the message clnt_create: RPC: Timed out I might be able to reboot, but this is a production server, I'd like to know how to troubleshoot this issue (so that I can prevent/fix it) as rebooting is not always convenient. I suspect the mount is busy, but I don't know how to find out what is keeping open file handles and/or why zfs and the kernel can't agree on the current state. Commented Jun 3, 2023 at 22:00
  • 1
    Sometimes it's not worth troubleshooting... This is not a normal condition, so efforts to prevent it may not be fruitful. You can examine zpool history and zpool events for your own records. But take the downtime hit if this is important. Commented Jun 4, 2023 at 2:13

You must log in to answer this question.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.