I've run into a issue where I can not delete directories under a user directory.
There are several of these dir now. I can do a 'mv' of the dir to another dir. ie 'mv sp2456 old/' and it will move the directory, but I can not delete them. Here is all the information I hope you will need to help me out. Thank you.
I have done a chmod -R 777 sp2f941 to allow all access
I also did a chown -R root:sys sp2f941.
Previous to these two commands the files were owned by spark:usr and were rwxr-xr-w
(ll is alias for "ls -la" )
bash-3.00# cd /home/spark/sp2f941/BUILD/
bash-3.00# ll
total 7
drwxrwxrwx 6 root sys 3072 Mar 25 14:02 .
drwxrwxrwx 3 root sys 512 Mar 25 14:30 ..
bash-3.00# rm -f *
bash-3.00# ll
total 7
drwxrwxrwx 6 root sys 3072 Mar 25 14:02 .
drwxrwxrwx 3 root sys 512 Mar 25 14:30 ..
bash-3.00# cd ..
bash-3.00# ls
BUILD
bash-3.00# rmdir BUILD
rmdir: 0653-611 Directory BUILD is not empty.
bash-3.00# whoami
root
bash-3.00# uname -ax
AIX cehazel 102983244 3 5 000623664C00
bash-3.00#
Mount shows:
/dev/hd1 /home jfs Feb 07 16:56 rw,log=/dev/hd8
I am able to partially re-create this problem, but not consistently. All other files and dirs in the user dir I can delete and act normally.
If you need anymore information, please let me know. Thank you for your help.
Is this on NFS? Happened to me the other day on a Linux box, but a Solaris box on the same cluster could delete the files just fine. (The admin on the Linux box even tried unmounting /home and restarting the daemon, but that didn't help at all.)
The rmdir command is never going to remove a directory that is not empty, even if you are root. Try to cd to your directory and run "ls -la". My bet is you've got some files that aren't showing up in your "ll" output(.profile & .sh_history come to mind).
Anyway, you can also try "rm -rf /home/spark/sp2f941/BUILD"
Tried that, here is the output:
bash-3.00# ls -la BUILD/
total 7
drwxrwxrwx 6 root sys 3072 Mar 25 14:02 .
drwxrwxrwx 3 root sys 512 Mar 25 14:30 ..
bash-3.00# rm -Rf BUILD
rm: 0653-611 Directory BUILD is not empty.
bash-3.00#
One possibility is that the directory is NFS-mounted, as era has already mentioned. Another possiblity would be a process that writes to this directory. Suppose a process that writes more or less constantly to a file, something like "vmstat 1 > /some/file" if you now remove "/some/file" ls will not show it, but it will still be there as long as the process (vmstat in this example) runs. You can see that by running "df" against the filesystem holding the file.
(btw. this is one excellent method to drive a Sysadmin crazy: Fill up the /tmp filesystem with such a mechanism and use a sufficiently innocent processname. No, do NOT try this on your colleague. ;-)) )
Use a tool like "fuser" to determine processes which still write to the filesystem/directory to remedy such a situation. Once you kill the process the filespace becomes available at once as the respective i-node is released automatically.
The /home directory is mounted as JFS. It a partition on a local drive.
I tried fuser with some flags but did not get any useful information back.
bash-3.00# pwd
/home/spark/old/sp2f941
bash-3.00# fuser -d *
BUILD:
bash-3.00# cd BUILD
bash-3.00# fuser -d *
*: A file or directory in the path name does not exist.
bash-3.00# fuser -dV *
*: A file or directory in the path name does not exist.
bash-3.00# cd ..
bash-3.00# fuser -dV *
BUILD:
bash-3.00# fuser -u /home
/home:
bash-3.00#
Did I do the correct parameters?
here is the full results from mount.
bash-3.00# mount
node mounted mounted over vfs date options
-------- --------------- --------------- ------ ------------ ---------------
/dev/hd4 / jfs Feb 07 16:55 rw,log=/dev/hd8
/dev/hd2 /usr jfs Feb 07 16:55 rw,log=/dev/hd8
/dev/hd9var /var jfs Feb 07 16:55 rw,log=/dev/hd8
/dev/hd3 /tmp jfs Feb 07 16:55 rw,log=/dev/hd8
/dev/hd1 /home jfs Feb 07 16:56 rw,log=/dev/hd8
/proc /proc procfs Feb 07 16:56 rw
/dev/hd10opt /opt jfs Feb 07 16:56 rw,log=/dev/hd8
bash-3.00#
I tried a fuser on the device and got the following results:
bash-3.00# fuser -uV /dev/hd1
/dev/hd1:
rawdev 15084c(root)
rawdev 19652c(root)
rawdev 20182c(root)
hmmm, this is awkward. To be honest, I'm left to guessing.
Can you umount the file system? Have you already tried to reboot the system and delte the directory afterwards? I d not think that the reboot itself (that is: the reloading of the kernel image) will resolve the problem, but maybe stopping all the processes and umounting all the filesystems might help.
I umounted the partition, no luck. I will try and reboot it next, but I have my doubts as I am pretty sure the system has been rebooted since I first saw this problem.
bash-3.00# fuser /dev/hd1
/dev/hd1:
bash-3.00# umount /dev/hd1
bash-3.00# mount /dev/hd1
bash-3.00# cd /home/spark
bash-3.00# cd old
bash-3.00# ls
sp2f941 sp37c21 sp37c41 sp37c61 sp50061
bash-3.00# rm -Rf *
rm: 0653-611 Directory sp2f941/BUILD is not empty.
rm: 0653-611 Directory sp2f941 is not empty.
rm: 0653-611 Directory sp37c21/BUILD is not empty.
rm: 0653-611 Directory sp37c21 is not empty.
rm: 0653-611 Directory sp37c41/BUILD is not empty.
rm: 0653-611 Directory sp37c41 is not empty.
rm: 0653-611 Directory sp37c61/BUILD is not empty.
rm: 0653-611 Directory sp37c61 is not empty.
rm: 0653-611 Directory sp50061/BUILD is not empty.
rm: 0653-611 Directory sp50061 is not empty.
bash-3.00#
(yes, all of those dirs are empty)
I came back to this problem and found what was causing the empty directories to not be deletable.
A build script was running "dos2unix BUILD/*"
this caused the directories under the BUILD directory to be converted somehow. I guess dos2unix does something to the directory file/header that corrupts it. The directories disappear from BUILD, but are still being tracked somehow.
The only solution I have so far is to recreate the partition.
So, on AIX5.3, don't run dos2unix on directory names.