I have an NFS volume in /etc/fstab that I expect to mount:
10.0.1.1:/data/col1/os_backup_hp /hp-ux-backup nfs ro,vers=3
It mounts with no problem via systemd, as it should. Given that there is no concept of "boot time" vs. "fully up and running" with systemd, I know that any volume in /etc/fstab will automatically mount any time that it is made available. However, the oddity here (and I've seen this on other systems in my environment using NFS volumes), is that with no action of my own, or any automation I'm aware of, there is a subdirectory in that volume that automatically mounts on its own after a period of time. I'm not sure when it happens, or how as I've not been able to find anything in the kern.log or journalctl refering to the mount. Here is my df output:
root@bp01:/var/log# df -t nfs -h
Filesystem Size Used Avail Use% Mounted on
10.0.1.1:/data/col1/os_backup_hp 336T 160T 177T 48% /hp-ux-backup
10.0.1.1:/data/col1/os_backup_hp/.snapshot 336T 160T 177T 48% /hp-ux-backup/.snapshot
The .snapshot is the unexpected mount. I'm not certain that it's systemd doing this, but with no mention in the logging, I suspect no. The .snapshot directory is a part of the NFS volume presented by a Dell/EMC filer. But it would have already been visible in the original NFS volume mount. So this is kind of redundant. I am planning on checking with the group that handles automation for my team to see if they set anything up that might wind up causing this, but I'm not sure if that is it either. Has anyone else seen anything like this?
Deck
My two kopeks ... I believe (think) that snapshot 'mounts' are part of NFS behaviour
check out https://community.netapp.com/t5/Network-and-Storage-Protocols/unwanted-automount-snapshot-Directories-Linux/m-p/113167
You may want to consult a local sysadmin / get on to a NFS related forum for better information.
2 Likes
Thank you very much. That is more info than my other searches turned up. But that sounds exactly like what I am seeing. I don't think that I ever actually did an 'ls' against anything in the .snapshot directory, and I shouldn't run into any stale file handle issues since these directories are always present.
I'll see if I can do some more digging on the NFS behavior in Linux kernels higher than 2.6 (we're on 5.15). I have to imagine there is a reason for it. I have not ever seen this on our Redhat boxes.
1 Like
ack, i'll do likewise when time allows.
Or could it be there's an 'automount' configured I wonder? Automounts mount automatically when a user accesses that pathname. Worth a little research?
I could well be completely wrong but thought this after you say it mounts all on its own.
Others on here will have greater knowledge of 'automount'.
Thank you for the idea. I can confirm we have not installed or configured autofs. But I am not sure if there are other kinds of automount installed by default. I will check.
Well I would search for 'Ubuntu Automount' (I haven't done it), see where that config lives on the system, and check that it's not configured.
I just ran through a few searches for general information on Ubuntu and automounting. Everything points back to the 'autofs' package for true systemwide automounting. Running a 'dkpg-query -l' I confirmed that autofs and the automount commands are not listed. Now that I've tested more based on the NetApp article linked by munkeHoller and also found a Redhat article about the same kind of behavior, it appears it has something to do with the NFS server side (A Dell/EMC Data Domain).
1 Like
I found a Redhat article that outlined the same behavior, but it was in reference to a NetApp filer. However, I unmounted the .snapshot redundant mapping, confirmed it was gone. Then I did an 'ls' against /hp-ux-backup/.snapshot and... the mount came back. So, identical behavior to what is mentioned in the NetApp article you linked.
I think what the root cause could be is some automation we have that might try to check something about each mounted file system. I have also reproduced the issue if I do a 'du' against that path. The .snapshot mount comes back. So I will check with our automation team.
Thank you for the assistance. Hopefully I can find out what is trying to access /hp-ux-backup/.snapshot and causing the redundant NFS mount.
2 Likes
Please share back to the forum when you've completed your sleuthing 
Linux identifies the .snapshot directory as a different file system. This avoids duplicate inodes (that could confuse tools like find, maybe even the cache).
Expired snapshots can cause annoying "stale nfs handles", so I recommend to hide the .snapshot on the NFS server. It can be still accessed by fully typing .snapshot
I have determined what is happening. Thanks to the info provided by munkeHoller and MadeInGermany, it appears that the Dell/EMC filer has the export option 'crossmnt' enabled by default and they provide no way of disabling it within their web UI (still have to dig around and see if I can get access to a CLI via the admin).
These are the simple actions that cause the crossmnt to happen:
du -h /hp-ux-backup
ls -ltra /hp-ux-backup/
It is definitely Linux kernel related. Our older, legacy systems that are on kernel 2.6.32 don't exhibit this issue, but our newer systems above that kernel version do. If there is a way to add the 'nocrossmnt' option to the export on an NFS server, that could prevent this from happening if .snapshot is a separate file system (and it likely is). It might also be possible to use the 'hide' option on the export to do the same, but that, I believe is more directory based than file system based. And again, without a way to make any changes to the export on the Dell/EMC filer, we might just have to live with these showing up on our newer systems. Also, this does not occur on our HP-UX hosts where this volume is used the most. Thanks again for the help!
2 Likes
@deckard , nice work,
https://www.man7.org/linux/man-pages/man5/exports.5.html
check out the nohide option , think setting that may prevent .snapshot being ghostmounted (i just made that term up btw, may copyleft it 
)
NB: my knowledge of NFS should probably be flagged as 'deprecated' as its been many years since i (was obljiged) to dabble.
2 Likes
On NetApp NAS my best practice was to hide .snapshot so ls -a, du, find won't discover it.
I just googled a NetApp doc, the respective chapter states
Users can access Snapshot copies to restore individual files from NFS clients. In some workloads
however, applications might list the contents of the NFS mount. In some cases, the .snapshot directory
gets included, which can add considerable time to these scans – especially in high file count
environments. In those cases, the Snapshot directory can be hidden from NFS mounts using the NFS
server option -v3-hide-snapshot.
As evidenced by the command, this only impacts NFSv3 clients, as the .snapshot directory is already
hidden for NFSv4.x.
Even when the .snapshot directory is hidden, it is still accessible. To remove access to the .snapshot
directory to clients, use the volume option -snapdir-access.
Note: These options don’t take effect until the export is remounted.
While this addresses all NFS clients, on Linux it also reduces the extra mounts and "stale NFS" errors when snapshots are deleted.
1 Like