Can anyone help me in cleaning /opt filesystem..
i have checked all the options and i have cleared all the logs and the total size of the files in /opt is shown as 1.8GB were as the size of /opt is 4.8GB
but wen i run the command
# df -h /opt
it gives
capacity
99%
Please help me..its urgent
run the command:
du -sh /opt/*
and have a look at the output...
i tried " du -sh /opt/* " and the total size comes to only 1.8 GB
but when i check for the total file system usage it shows as 4.7GB used
can you reboot the server? if not, check for open files in /opt which are used by a running application...
i cant reboot the server as its a critical backup server.
i tried using " lsof | grep /opt " , but that gives a big list.
Those files would be devices , there would be a dodgy process which would be occyping the space, try top and prstat -a to identify the process.
lsof is your friend.
Use
lsof | grep /opt | grep deleted
or
lsof +L1
Both of the above will show the process which has the deleted files opened. So until you stop the process, the space will not be freed.
$ lsof | grep /tmp | grep deleted
less 22073 root 4r REG 8,1 1948 1504213 /tmp/file (deleted)
$ lsof +L1
COMMAND PID USER FD TYPE DEVICE SIZE NLINK NODE NAME
less 22073 root 4r REG 8,1 1948 0 1504213 /tmp/file (deleted)
From man page:
A specification of the form ``+L1'' will select open files that have been unlinked.
Find out which is the process, and do the required stuff.
here is what i get..
#lsof +L1
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NLINK NODE NAME
gstd 21811 root 0w VREG 85,3 3146831005 0 12684 /opt (/dev/md/dsk/d3)
gstd 21811 root 1w VREG 85,3 3146831005 0 12684 /opt (/dev/md/dsk/d3)
gstd 21811 root 2w VREG 85,3 3146831005 0 12684 /opt (/dev/md/dsk/d3)
Any help on fixing this is appreciated.
Is there a symbolic link in your system that points to /opt? Maybe some software application that loads in other directories but links itself to /opt?
In desperation there are likely to be applications in /opt that are not constantly running that could therefore be moved to a different filesystem and then linked to from /opt to make them appear to be still in /opt. e.g. if you had Adobe Acrobat (highly unlikely on a backup server but it will serve for this example) then you could do the following to move Adobe Acrobat to the root file system:
# mv /opt/Adobe /opt_Adobe
# ln -s /opt_Adobe /opt/Adobe
No symbolic liink pointing to /opt ..
No I'm suggesting you move something out of /opt to free up some space and then use a symbolic link to make appear to be still in /opt.
Are we sure that some log file isn't leaking or ran crazy and caused this, or perhaps some temp file or what not?
What does du display of the /opt directory? You can do it at a directory level and if you know certain directories should not be that large then you can narrow the problem down, or are you really just out of space on that volume due to having too much installed in /opt?
if there is a big difference between the output of du and df, there are
probably open files (by a running process), that are deleted or moved.
you can find them, as already reported, with
lsof -a +L1
or something like this:
find /proc/ -type f -links 0 -exec ls -l {} \;
should have a similar effect
(find(1) - find files (man pages section 1: User Commands) - Sun Microsystems)
a reboot would probably be the easiest solution.
If a log or core file has caused /opt to be filled up then it will have a recent date and time stamp so you can do something like:
# find /opt -mount -type f -mtime -7 -ls
And you will get a long listing of all file modified in the last 7 days.
HTH
Sorry for the late reply. Thanks for the Help
I rebooted the server and things are back to normal now.
as mentioned this issue was due to OPEN FILES. and reeboot was the easiest way to sort it out.
the reboot was mentioned 5 days ago... so, yes that was because it is easy and can safe a lot of work!
In the first post you "cleared all the logs". Time to acquaint yourself with which processes hold log files open and the correct procedure to maintain those logs. Where there is no procedure supplied it is often better to copy and trunctate active logs rather than just deleting them.
As with all Systems Administration issues it helps to know which Operating System.