Automount issue

ce9888 · July 21, 2015, 8:54am

Hi there,

I have a strange problem, I have a NFS server running AIX 7.1 TL3 SP3, let's call it server A. I have another AIX 7.1 TL3 SP3 server, let's call it server B, that's automounting a filesystem from server A.

When server B is automounting the filesystem from server A, I can't see any files or directories. I tried to manually mount the filesystem on server B from server A and I see all the files and directories. On server B I tried to create a file on the automount path, this is working. But when I look on the server A the file I just created is not there. I umounted the exported filesystem on server A and guess what, the file I created on server B was there. It's looks like the automount service from server B is mounting the mount point from server A not the actual filesystem...

Anyone has seen that ?

agent.kgb · July 21, 2015, 9:07am

As my friend writes, crystall ball is being repaired right now. Can you explain the same, but with commands and configuration files. E.g. I make mount: # mount nfs-server:/fs-share /mnt , the result is mount | grep /mnt and df -m /mnt . I can write to FS: cd /mnt ; echo Hello world >hello.txt ; ls -l /mnt/hello.txt . If I do the same: excerpt from automount config , I receive: mount | grep ; df -m and can't do anything: echo Good-bye world >/mnt/bye.txt ; ls -l /mnt/bye.txt

ce9888 · July 21, 2015, 10:21am

Ok here's the details

Server A configuration

p02s:root# cat /etc/exports
/home                -access=sdt-net
/home/s/JourEceArch  -access=sdt-net
/home/s/web          -access=sdt-net
/home/s/rapports_cme -access=sdt-net
/home/cores          -access=sdt-net

p02s:root# cat /etc/netgroup
sdt-net (a01s,,) (a01s1,,) (a02s,,) (a02s1,,) (a03s,,) (c01s,,) (c01s1,,) (c02s,,) (c02s1,,) (m01s,,) (m01s1,,) (m02s,,) (m05s,,) (m
06s,,) (m07s,,) (m08s,,) (m09s,,) (m10s,,) (m14s,,) (m17s,,) (p01s,,) (p01s1,,) (p02s,,) (p02s1,,) (r01s,,) (r01s1,,) (u01s,,) (u01s
1,,) (u02s,,) (u02s1,,) (va02s1,,) (x01s,,) (x01s1,,) (x02s,,) (x02s1,,) (x03s,,) (x03s1,,) (x04s,,) (x04s1,,)
sdt-root
sdt-user

p02s:root# showmount -e
export list for p02s:
/home                sdt-net
/home/s/JourEceArch  sdt-net
/home/s/web          sdt-net
/home/s/rapports_cme sdt-net
/home/cores          sdt-net

p02s:root# ls -altr /home/s/rapports_cme
total 24
drwxrwx---    2 spsy     spec            256 Sep 24 2013  lost+found/
drwxrwsr-x    2 spsy     spec            256 Sep 24 2013  RAPPORTS_ECHANGES/
drwxrwsrwx    2 spsy     spec            256 May 12 14:17 PROG_INT/
drwxrwsr-x    2 spsy     spec            256 May 12 14:17 PLAN_RET/
drwxrwsr-x    2 spsy     spec            256 May 12 14:17 BilanCME/
drwxrwsr-x    2 spsy     spec            256 May 28 13:55 PLAN_ANNUEL/
drwxrwsrwx    2 spsy     spec            256 Jun 01 08:02 data_bilancme/
drwxrwxr-x   12 spsy     spec           4096 Jun 01 08:03 ./
lrwxrwxrwx    1 root     spec              8 Jun 05 11:23 PROGRAMME_INT�GR�@ -> PROG_INT/
lrwxrwxrwx    1 root     spec              5 Jun 05 11:23 BILAN_PR�VISIONNEL@ -> GEODE/
lrwxrwxrwx    1 root     spec              8 Jun 05 11:23 BILAN_CM�@ -> BilanCME/
drwxrwsr-x   31 spsy     spec           4096 Jun 25 08:20 ../
drwxrwsr-x    2 spsy     spec            256 Jul 09 00:10 GEODE/

p02s:root# df /home/s/rapports_cme
Filesystem    512-blocks      Free %Used    Iused %Iused Mounted on
/dev/fslv08      3932160   3905928    1%       36     1% /home/s/rapports_cme

p02s:root# mount
  node       mounted        mounted over    vfs       date        options
-------- ---------------  ---------------  ------ ------------ ---------------
         /dev/hd4         /                jfs2   May 01 15:24 rw,log=/dev/hd8
         /dev/hd2         /usr             jfs2   May 01 15:24 rw,log=/dev/hd8
         /dev/hd9var      /var             jfs2   May 01 15:24 rw,log=/dev/hd8
         /dev/hd3         /tmp             jfs2   May 01 15:24 rw,log=/dev/hd8
         /dev/fwdump      /var/adm/ras/platform jfs2   May 01 15:26 rw,log=/dev/hd8
         /dev/hd1         /users           jfs2   May 01 15:26 rw,log=/dev/hd8
         /dev/hd11admin   /admin           jfs2   May 01 15:26 rw,log=/dev/hd8
         /proc            /proc            procfs May 01 15:26 rw
         /dev/hd10opt     /opt             jfs2   May 01 15:26 rw,log=/dev/hd8
         /dev/livedump    /var/adm/ras/livedump jfs2   May 01 15:26 rw,log=/dev/hd8
         /dev/fslv03      /home            jfs2   May 01 15:26 rw,log=/dev/hd8
         /dev/fslv04      /u01/app         jfs2   May 01 15:26 rw,log=/dev/hd8
         /dev/fslv05      /home/cores      jfs2   May 01 15:26 rw,log=/dev/hd8
         /dev/fslv00      /var/adm/messback jfs2   May 01 15:26 rw,log=/dev/hd8
         -hosts           /vhe             autofs May 01 15:26 rw,soft,nobrowse,ignore
         -hosts           /net             autofs May 01 15:26 ro,soft,nobrowse,ignore
         /dev/fslv06      /home/s/web      jfs2   Jun 05 12:01 rw,log=INLINE
         /dev/fslv09      /home/s/JourEceArch jfs2   Jun 05 12:01 rw,log=INLINE
         /dev/fslv08      /home/s/rapports_cme jfs2   Jul 20 15:01 rw,log=INLINE

Server B

Let's check what is mounted

a01s:root# mount
  node       mounted        mounted over    vfs       date        options
-------- ---------------  ---------------  ------ ------------ ---------------
         /dev/hd4         /                jfs2   Jul 21 08:25 rw,log=/dev/hd8
         /dev/hd2         /usr             jfs2   Jul 21 08:25 rw,log=/dev/hd8
         /dev/hd9var      /var             jfs2   Jul 21 08:25 rw,log=/dev/hd8
         /dev/hd3         /tmp             jfs2   Jul 21 08:25 rw,log=/dev/hd8
         /dev/hd1         /users           jfs2   Jul 21 08:25 rw,log=/dev/hd8
         /dev/hd11admin   /admin           jfs2   Jul 21 08:25 rw,log=/dev/hd8
         /proc            /proc            procfs Jul 21 08:25 rw
         /dev/hd10opt     /opt             jfs2   Jul 21 08:25 rw,log=/dev/hd8
         /dev/livedump    /var/adm/ras/livedump jfs2   Jul 21 08:25 rw,log=/dev/hd8
         /dev/fslv00      /home            jfs2   Jul 21 08:25 rw,log=INLINE
         /dev/fslv01      /u01/app         jfs2   Jul 21 08:25 rw,log=INLINE
         /dev/lv_backuprootvg /backuprootvg    jfs2   Jul 21 08:25 rw,log=INLINE
         /dev/fslv02      /var/adm/messback jfs2   Jul 21 08:25 rw,log=INLINE
         /dev/fslv03      /home/cores      jfs2   Jul 21 08:25 rw,log=/dev/loglv00
         /dev/fslv04      /home/s/FEATURES jfs2   Jul 21 08:25 rw,log=INLINE
         /dev/fslv05      /home/s/release  jfs2   Jul 21 08:25 rw,log=INLINE
         /dev/fslv06      /www             jfs2   Jul 21 08:25 rw,log=INLINE
         /dev/fslv07      /home/saves      jfs2   Jul 21 08:25 rw,log=INLINE
         /dev/fslv08      /home/sdbr       jfs2   Jul 21 08:25 rw,log=INLINE
         /dev/fslv09      /home/sdbc       jfs2   Jul 21 08:25 rw,log=INLINE
         /dev/lv_cme      /www/docs/Rapports-CME jfs2   Jul 21 08:25 rw,log=INLINE
         -hosts           /vhe             autofs Jul 21 08:25 rw,soft,nobrowse,ignore
         -hosts           /net             autofs Jul 21 08:25 ro,soft,nobrowse,ignore

Switch to my user to check the automount content

a01s:root# su - spsy
a01s:spsy$ cd /vhe/p02s/home/s/rapports_cme
a01s:spsy$ ls -altr
total 8
drwxrwsr-x   31 spsy     spec           4096 Jun 25 08:20 ../
drwxrwsr-x    2 spsy     spec            256 Jul 20 14:01 ./
a01s:spsy$ exit

The automount is responding but is showing no files or directories. Let's try it manually

a01s:root# mount p02s:/home/s/rapports_cme /mnt
a01s:root# su - spsy
a01s:spsy$ cd /mnt
a01s:spsy$ ls -altr
total 24
drwxrwx---    2 spsy     spec            256 Sep 24 2013  lost+found/
drwxrwsr-x    2 spsy     spec            256 Sep 24 2013  RAPPORTS_ECHANGES/
drwxrwsrwx    2 spsy     spec            256 May 12 14:17 PROG_INT/
drwxrwsr-x    2 spsy     spec            256 May 12 14:17 PLAN_RET/
drwxrwsr-x    2 spsy     spec            256 May 12 14:17 BilanCME/
drwxrwsr-x    2 spsy     spec            256 May 28 13:55 PLAN_ANNUEL/
drwxrwsrwx    2 spsy     spec            256 Jun 01 08:02 data_bilancme/
drwxrwxr-x   12 spsy     spec           4096 Jun 01 08:03 ./
lrwxrwxrwx    1 root     spec              8 Jun 05 11:23 PROGRAMME_INT�GR�@ -> PROG_INT/
lrwxrwxrwx    1 root     spec              5 Jun 05 11:23 BILAN_PR�VISIONNEL@ -> GEODE/
lrwxrwxrwx    1 root     spec              8 Jun 05 11:23 BILAN_CM�@ -> BilanCME/
drwxrwsr-x    2 spsy     spec            256 Jul 09 00:10 GEODE/
dr-xr-xr-x   30 root     sys            4096 Jul 21 08:25 ../

It's working this way.

Let's check what is mounted

a01s:spsy$ mount
  node       mounted        mounted over    vfs       date        options
-------- ---------------  ---------------  ------ ------------ ---------------
         /dev/hd4         /                jfs2   Jul 21 08:25 rw,log=/dev/hd8
         /dev/hd2         /usr             jfs2   Jul 21 08:25 rw,log=/dev/hd8
         /dev/hd9var      /var             jfs2   Jul 21 08:25 rw,log=/dev/hd8
         /dev/hd3         /tmp             jfs2   Jul 21 08:25 rw,log=/dev/hd8
         /dev/hd1         /users           jfs2   Jul 21 08:25 rw,log=/dev/hd8
         /dev/hd11admin   /admin           jfs2   Jul 21 08:25 rw,log=/dev/hd8
         /proc            /proc            procfs Jul 21 08:25 rw
         /dev/hd10opt     /opt             jfs2   Jul 21 08:25 rw,log=/dev/hd8
         /dev/livedump    /var/adm/ras/livedump jfs2   Jul 21 08:25 rw,log=/dev/hd8
         /dev/fslv00      /home            jfs2   Jul 21 08:25 rw,log=INLINE
         /dev/fslv01      /u01/app         jfs2   Jul 21 08:25 rw,log=INLINE
         /dev/lv_backuprootvg /backuprootvg    jfs2   Jul 21 08:25 rw,log=INLINE
         /dev/fslv02      /var/adm/messback jfs2   Jul 21 08:25 rw,log=INLINE
         /dev/fslv03      /home/cores      jfs2   Jul 21 08:25 rw,log=/dev/loglv00
         /dev/fslv04      /home/s/FEATURES jfs2   Jul 21 08:25 rw,log=INLINE
         /dev/fslv05      /home/s/release  jfs2   Jul 21 08:25 rw,log=INLINE
         /dev/fslv06      /www             jfs2   Jul 21 08:25 rw,log=INLINE
         /dev/fslv07      /home/saves      jfs2   Jul 21 08:25 rw,log=INLINE
         /dev/fslv08      /home/sdbr       jfs2   Jul 21 08:25 rw,log=INLINE
         /dev/fslv09      /home/sdbc       jfs2   Jul 21 08:25 rw,log=INLINE
         /dev/lv_cme      /www/docs/Rapports-CME jfs2   Jul 21 08:25 rw,log=INLINE
         -hosts           /vhe             autofs Jul 21 08:25 rw,soft,nobrowse,ignore
         -hosts           /net             autofs Jul 21 08:25 ro,soft,nobrowse,ignore
         -hosts           /vhe/p02s/home   autofs Jul 21 09:34 rw,soft,nobrowse,nest,ignore
p02s     /home            /vhe/p02s/home   nfs3   Jul 21 09:34 rw,soft,nest
         -hosts           /vhe/p02s/home/cores autofs Jul 21 09:34 rw,soft,nobrowse,nest,ignore
p02s     /home/s/rapports_cme /mnt             nfs3   Jul 21 09:35

I would expect entries for my /vhe/p02s/home/s/rapports_cme in the mount output but they are obviously not there.

Let's try to create a file

a01s:spsy$ cd /vhe/p02s/home/s/rapports_cme
a01s:spsy$ echo "test automount path" > test_file.txt
a01s:spsy$ ls -altr
total 16
drwxrwsr-x   31 spsy     spec           4096 Jun 25 08:20 ../
-rw-rw-r--    1 spsy     spec             20 Jul 21 09:48 test_file.txt
drwxrwsr-x    2 spsy     spec            256 Jul 21 09:48 ./

Let's check on /mnt if it's there

a01s:spsy$ cd /mnt
a01s:spsy$ ls -altr
total 24
drwxrwx---    2 spsy     spec            256 Sep 24 2013  lost+found/
drwxrwsr-x    2 spsy     spec            256 Sep 24 2013  RAPPORTS_ECHANGES/
drwxrwsrwx    2 spsy     spec            256 May 12 14:17 PROG_INT/
drwxrwsr-x    2 spsy     spec            256 May 12 14:17 PLAN_RET/
drwxrwsr-x    2 spsy     spec            256 May 12 14:17 BilanCME/
drwxrwsr-x    2 spsy     spec            256 May 28 13:55 PLAN_ANNUEL/
drwxrwsrwx    2 spsy     spec            256 Jun 01 08:02 data_bilancme/
lrwxrwxrwx    1 root     spec              8 Jun 05 11:23 PROGRAMME_INT�GR�@ -> PROG_INT/
lrwxrwxrwx    1 root     spec              5 Jun 05 11:23 BILAN_PR�VISIONNEL@ -> GEODE/
lrwxrwxrwx    1 root     spec              8 Jun 05 11:23 BILAN_CM�@ -> BilanCME/
drwxrwsr-x    2 spsy     spec            256 Jul 09 00:10 GEODE/
dr-xr-xr-x   30 root     sys            4096 Jul 21 08:25 ../
drwxrwxr-x   12 spsy     spec           4096 Jul 21 09:47 ./

Thought luck...

Umounting everything

a01s:root# cd /
a01s:root# umount /mnt
a01s:root# umount /vhe/p02s/home/cores
a01s:root# umount  /vhe/p02s/home
a01s:root# umount  /vhe/p02s/home
a01s:root# mount
  node       mounted        mounted over    vfs       date        options
-------- ---------------  ---------------  ------ ------------ ---------------
         /dev/hd4         /                jfs2   Jul 21 08:25 rw,log=/dev/hd8
         /dev/hd2         /usr             jfs2   Jul 21 08:25 rw,log=/dev/hd8
         /dev/hd9var      /var             jfs2   Jul 21 08:25 rw,log=/dev/hd8
         /dev/hd3         /tmp             jfs2   Jul 21 08:25 rw,log=/dev/hd8
         /dev/hd1         /users           jfs2   Jul 21 08:25 rw,log=/dev/hd8
         /dev/hd11admin   /admin           jfs2   Jul 21 08:25 rw,log=/dev/hd8
         /proc            /proc            procfs Jul 21 08:25 rw
         /dev/hd10opt     /opt             jfs2   Jul 21 08:25 rw,log=/dev/hd8
         /dev/livedump    /var/adm/ras/livedump jfs2   Jul 21 08:25 rw,log=/dev/hd8
         /dev/fslv00      /home            jfs2   Jul 21 08:25 rw,log=INLINE
         /dev/fslv01      /u01/app         jfs2   Jul 21 08:25 rw,log=INLINE
         /dev/lv_backuprootvg /backuprootvg    jfs2   Jul 21 08:25 rw,log=INLINE
         /dev/fslv02      /var/adm/messback jfs2   Jul 21 08:25 rw,log=INLINE
         /dev/fslv03      /home/cores      jfs2   Jul 21 08:25 rw,log=/dev/loglv00
         /dev/fslv04      /home/s/FEATURES jfs2   Jul 21 08:25 rw,log=INLINE
         /dev/fslv05      /home/s/release  jfs2   Jul 21 08:25 rw,log=INLINE
         /dev/fslv06      /www             jfs2   Jul 21 08:25 rw,log=INLINE
         /dev/fslv07      /home/saves      jfs2   Jul 21 08:25 rw,log=INLINE
         /dev/fslv08      /home/sdbr       jfs2   Jul 21 08:25 rw,log=INLINE
         /dev/fslv09      /home/sdbc       jfs2   Jul 21 08:25 rw,log=INLINE
         /dev/lv_cme      /www/docs/Rapports-CME jfs2   Jul 21 08:25 rw,log=INLINE
         -hosts           /vhe             autofs Jul 21 08:25 rw,soft,nobrowse,ignore
         -hosts           /net             autofs Jul 21 08:25 ro,soft,nobrowse,ignore

Let's go back on Server A to check what's happening

p02s:root# cd /home/s/rapports_cme
p02s:root# ls -altr
total 24
drwxrwx---    2 spsy     spec            256 Sep 24 2013  lost+found/
drwxrwsr-x    2 spsy     spec            256 Sep 24 2013  RAPPORTS_ECHANGES/
drwxrwsrwx    2 spsy     spec            256 May 12 14:17 PROG_INT/
drwxrwsr-x    2 spsy     spec            256 May 12 14:17 PLAN_RET/
drwxrwsr-x    2 spsy     spec            256 May 12 14:17 BilanCME/
drwxrwsr-x    2 spsy     spec            256 May 28 13:55 PLAN_ANNUEL/
drwxrwsrwx    2 spsy     spec            256 Jun 01 08:02 data_bilancme/
lrwxrwxrwx    1 root     spec              8 Jun 05 11:23 PROGRAMME_INT�GR�@ -> PROG_INT/
lrwxrwxrwx    1 root     spec              5 Jun 05 11:23 BILAN_PR�VISIONNEL@ -> GEODE/
lrwxrwxrwx    1 root     spec              8 Jun 05 11:23 BILAN_CM�@ -> BilanCME/
drwxrwsr-x   31 spsy     spec           4096 Jun 25 08:20 ../
drwxrwsr-x    2 spsy     spec            256 Jul 09 00:10 GEODE/
drwxrwxr-x   12 spsy     spec           4096 Jul 21 09:47 ./

File is not there either...

Let's try to umount the filesystem and see what lies underneath

p02s:root# cd /
p02s:root# umount /home/s/rapports_cme
p02s:root# cd /home/s/rapports_cme
p02s:root# ls -altr
total 16
drwxrwsr-x   31 spsy     spec           4096 Jun 25 08:20 ../
-rw-rw-r--    1 spsy     spec             20 Jul 21 09:48 test_file.txt
drwxrwsr-x    2 spsy     spec            256 Jul 21 09:48 ./
p02s:root# cat test_file.txt
test automount path

Here's the file

What I can understand, is the /home/s/rapports_cme filesystem is not mounted correctly by the automount on server B. Is it because server A is already exporting it's parent directory ? I guess. Is it a bug with automount or is it suppose to act like that ?

agent.kgb · July 21, 2015, 3:48pm

The first difference I see, you try to automount the share using your user account, but when you do it manually you mount the share using the root account.

Second, you didn't show automount configuration (or did I miss something?). But what I see and what is strange for me is "automounter on top of automounter" configuration. You have /vhe directory, which is defined in automounter. And you have at the same the directory /vhe/p02s/home which also seems to be defined in automounter. You receive several layers of mounts. When automounter starts it reserves the directory /vhe/p02s/home for the future mount, but when you go into /vhe it mounts the filesystem from the NFS server into /vhe and with this action it over-mounts the directory /vhe/p02s/home. That's why when you go into /vhe/p02s/home as user nothing happens - these are 2 different directories.

I think, you don't need such complex configuration. You can export just /home from you server. All other sub-directories are exported automagically with it and because you use NFS v3 you can mount sub-directories if you want to do it.