---------- Post updated at 09:53 AM ---------- Previous update was at 09:44 AM ----------
Have already deleted as much log file can like syslog.log and mail.log from /var/adm/syslog
Am wondering if can do any thing for this dir
/var/ios
/var/stm
/var/vx
Anything can do with /var/adm/wtmp (As I know can do this with single user mode only am not sure )
/var/adm/sw
/var/opt/sanmgr
/var/opt/perf
/var/opt/gcc
/var/opt/wbem
Excuse if am asking silly Que ... as am new on hp-ux and not aware of use of these dir/files
Deleting logfiles usually makes the situation worse -- they're probably still in use so won't be deleted, and will exist on disk until the logger closes them. But they're no longer in the directory after you delete them, preventing you from doing anything to them!
Try restarting your logger to release the deleted files, and don't delete logfiles thereafter -- truncate them. Having it open won't let the logger stop you from doing that.
Thanks !!
have release about 20Mb space by doing so ... it's abruptly increases am not sure why before day it was about 60MB am not sure what happen ...
Have much space free on /usr if can mv and create symlink for any of directory mentioned above that is non critical as per OS term ..
--Shirish
---------- Post updated at 11:19 AM ---------- Previous update was at 10:36 AM ----------
Thanks have got the temporary solution have moved 30 MB of SAN archived logs
Your "bdf" and your "du -sk" do not agree. There's about 4 Gigabytes difference. That's ignoring the fact that "du -sk" follows links and can be much higher than "bdf".
You should be using "du -skx" to stop "du" traversing into other filesystems.
If you have been deleting (not nulling) active logs you probably need a reboot.
Beware that some system logging processes stop working completely if you delete their logs. Be prepared to create an empty log with exactly the same permissions as before if you find that a logging process stops working.
Methyl , Thanks for this usefull info about du cmd .
Can I do any thing with my /var/adm/sw dir as near about 2.5 GB utilization by this ..
was wondering if can move it to /usr and create a symlink over here as below
/var/adm/sw -->> /usr/backup-var/sw
or with any of below dirs ..
/var/ios
/var/stm
/var/vx
Which is less critical as per OS term ..am not aware ..
Not a good idea...Unless you like fighting with funky issues....
That said in there (/var/adm/sw) you will find all the patches installed (and obsoleted, and uninstalled etc...) You may gain space by intellingently doing some cleanup, beware of the risk of not being able to "rollback" removed patches afterward... but very first superseeded patchs are of little risk...
What about /var/adm/syslog? what size are the logs there?
I trim them regularly ( customized by me though - important info in beginning syslog (at boot time) is always kept till next reboot...)
...
There are logs in /var/stm you can clean also and save 10-400 MB...
I have an old 10.20 that makes me sweat regularly (vital old legacy stuff nobody wanted to port : forms 3 with oracle 7.2.3...)
civ:/var/stm/logs/sys $ ll
total 434840
-rw-r--r-- 1 root root 220525916 Feb 24 11:29 activity_log
-rw-rw-r-- 1 root sys 878 May 5 2010 config.stm
-rw-r--r-- 1 root root 4584 Feb 2 09:02 diaglogd_activity_log
-rw-r--r-- 1 root root 5480 Feb 2 09:02 memlogd_activity_log
-rw-r--r-- 1 root root 5872 Feb 2 09:02 scan_hw_log
civ:/var/stm/logs/sys $ bdf /var
Filesystem kbytes used avail %used Mounted on
/dev/vg00/lvol8 921600 555664 344185 62% /var
I only have this unique volume group vg00 in mirror on old SCSI 9GB (all space saving is important...) with all on it.
When I forget to look (thanks for reminding me Hehe...) the system crashes... and this box is not on site but in the countryside (with snow lately and Im on motorcycle...)
I just zero activity_log periodically, So Do it today:
civ:/var/stm/logs/sys $ ll
total 434840
-rw-r--r-- 1 root root 220578080 Feb 24 11:37 activity_log
-rw-rw-r-- 1 root sys 878 May 5 2010 config.stm
-rw-r--r-- 1 root root 4584 Feb 2 09:02 diaglogd_activity_log
-rw-r--r-- 1 root root 5480 Feb 2 09:02 memlogd_activity_log
-rw-r--r-- 1 root root 5872 Feb 2 09:02 scan_hw_log
civ:/var/stm/logs/sys $ >activity_log
civ:/var/stm/logs/sys $ r bdf
bdf /var
Filesystem kbytes used avail %used Mounted on
/dev/vg00/lvol8 921600 338265 547997 38% /var
Look at how fast its growing though:
-rw-r--r-- 1 root root 220525916 Feb 24 11:29 activity_log
-rw-r--r-- 1 root root 220578080 Feb 24 11:37 activity_log
Be vary careful when moving things in system partitions because you still need an intact system when you are up single-user.
Regarding /var/adm/sw . The directory /var/adm/sw/save is all the undo information for the patches applied on the system and can be huge. I believe that there is a formal process to "commit" the patches which also means that you cannot roll back those patches. I've never needed to do it.
Still concerned about your /var partition itself. Have you been able to reboot? If it is still like this after a reboot I wonder if there has been a faulty attempt to extend the filesystem.
Also you don't seem to have a /var/opt in your "du" list.
What I really ment was more : When things grow abnormally fast in /var, look first in stm !
the logs hardly grow till something make them go bezeurk...
Here you see after I passed on the box and checked throughly:
civ:/var/stm/logs/sys $ date
Fri Feb 24 18:47:36 MET 2012
civ:/var/stm/logs/sys $ ps -ef|grep iagm|grep -v grep
root 9732 1 0 11:55:07 ? 0:07 /usr/sbin/stm/uut/bin/sys/diagmond
civ:/var/stm/logs/sys $ ll
total 178
-rw-r--r-- 1 root root 64208 Feb 24 11:56 activity_log
-rw-rw-r-- 1 root sys 878 May 5 2010 config.stm
-rw-r--r-- 1 root root 4648 Feb 24 11:55 diaglogd_activity_log
-rw-r--r-- 1 root root 5608 Feb 24 11:55 memlogd_activity_log
-rw-r--r-- 1 root sys 5872 Feb 24 11:55 scan_hw_log
Dead calm since...
Then there is /var/tombstones, with all sort of HW dumps when things go wrong...or systrem crashes, beware the system will increment each time so once your system is settled ( no bugs - HW issues) cleanup all of them to start as "new"
@Methyl,
Was archived some sanmgr log files and some of syslog.log and mail.log and also all coda and pm logs .
Have restarted the system some 3 day back successfully .
Here's the o/p you asked above .
# bdf
Filesystem kbytes used avail %used Mounted on
/dev/vg00/lvol3 1540096 269472 1260704 18% /
/dev/vg00/lvol1 497584 60880 386944 14% /stand
/dev/vg00/lvol9 6144000 6133600 10400 100% /var
/dev/vgignite/lvdata
46080000 42112000 3939816 91% /var/opt/ignite/recovery/archives
/dev/vg00/lvol6 6144000 2017720 4094352 33% /usr
/dev/vg00/lvol5 1024000 533375 460016 54% /u01_local
/dev/vg00/lvol4 1228800 560808 662936 46% /tmp
/dev/vgignite/software
5120000 3466942 1565621 69% /software
/dev/vg00/lvol8 7905280 6064768 1826440 77% /opt
/dev/vg00/lvol7 1024000 530200 490008 52% /home
/dev/vg00/lvol10 10240000 8616296 1576150 85% /depot
<root@pb>/var/opt/ignite # du -skx /var
6121776 /var
<root@pb>/var/stm/logs/sys #
<root@pb>/var # du -skx /var/* | sort -n | egrep -v ^0
8 /var/aa
8 /var/aaa
8 /var/ovsuf
8 /var/ovsuf.orig
16 /var/statmon
24 /var/run
32 /var/config.txt
192 /var/dt
392 /var/preserve
408 /var/asx
416 /var/yp
472 /var/spool
648 /var/sam
760 /var/tombstones
2440 /var/tmp
10528 /var/ios
24312 /var/stm
74776 /var/vx
1892024 /var/adm
4113808 /var/opt
<root@pb>/var #
below cmd o/p is too big have SNIP FOR LAST 15 DAYS ...
/var/ -xdev -type f -size +10000000c -exec ls -lad {} \;
got f�w files which got suddenly increase in last 15 days .. are as below
<root@pb>/var/opt/ignite # find /var/ -xdev -type f -mtime -15 -size +10000000c -exec ls -lad {} \; | sort -nr
-rw-r--r-- 1 root sys 25427968 Feb 24 03:41 /var/opt/omni/db40/datafiles/cdb/fn1.ext
-rw-r--r-- 1 root sys 22544384 Feb 18 19:11 /var/opt/omni/db40/datafiles/cdb/fn3.ext
-rw-r--r-- 1 root sys 207421440 Feb 24 03:41 /var/opt/omni/db40/datafiles/cdb/fnames.dat
-rw-r--r-- 1 root sys 16809984 Feb 18 18:43 /var/opt/omni/db40/datafiles/cdb/fn4.ext
-rw-r--r-- 1 root sys 15368192 Feb 24 03:41 /var/opt/omni/db40/datafiles/cdb/dirs.dat
-rw-r--r-- 1 root root 10000158 Feb 24 16:40 /var/opt/sanmgr/commandview/server/logs/OldServerTrace.txt
-rw-r--r-- 1 bin sys 10004881 Feb 22 01:00 /var/opt/ignite/clients/0x00306E2C31A7/recovery/2012-02-22,01:00/flist
-rw-r--r-- 1 bin sys 10004763 Feb 15 01:00 /var/opt/ignite/clients/0x00306E2C31A7/recovery/2012-02-15,01:00/flist
-rw------- 1 root root 34117764 Feb 18 19:26 /var/opt/omni/db40/dcbf/ca3d090e_4b84d566_4aae_0001_4F083C18.dat
ar
As I see some files /var/opt/omni/db40/datafiles/cdb/fnames.dat is increased suddenly to about 40+MB in last 10 days ..
Was looking for instead of doing something with application files if can move any non critical OS dir to some other partitions and create symlink in /var
You can try to look in /var/opt to see if you can move some non-system stuff elsewhere and use simlinks, better still : I did this on one box...
Check (especially oracle...) what is /var/tmp if symbolic link forget about it if not, move its content to /tmp and create a symlink /var/tmp-->/tmp
Before check that no one has open files ( well if you are moving its less important...) and more important no one new is going to use it the time of the modification...
If you have oracle installed be sure you have enough space in /tmp (make it 1.5 GB to be safe and keep it 1GB free when using orainstaller...)
Well we've found the missing 4 Gigabytes in /var now your post includes /var/opt !
Looks like you are running a gigantic Omniback Cell Manager on a server with inadequate disc space.
Consider planning to move /var/opt/omni to another substantially larger filesystem and get it out of /var - or even moving Omniback to a brand new dedicated backup server . You don't seem to have any suitable mounted spare disc space at the moment.
The file /var/opt/omni/db40/datafiles/cdb/fnames.dat is the master directory of filenames for all your non-expired backups. Your backup policy needs review because this server has inadequate disc space to store the Omniback directory files.
Imho. This server is grossly undersized and needs more disc drives fitted.
Ps. I suspect that the reason your Ignite backups are so large is because you have so much in vg00 .
Can you drill down to find the really big directories in /var/opt just in case there is another one as well as /var/opt/omni?
Looks like so much services so little disk scenario
You should check if you can purge DP database (carefully and planed )
See if you need files like 'OldServerTrace.txt'.
I have similar situation on inherited DP cell manager, with grossly oversized DP database, and no window to do proper purge (constant backups, tape replications).
To properly fix this situation you need some downtime and a good plan.
I would present some disk space (if possible) and make a symlink on most used /var part as an emergency measure.
Also, implement some kind of log rotation ( logrotate is avalible on HPUX porting centre ). Porting And Archive Centre For HP-UX
@Peasant
As the O/P has not posed the backup strategy (and most importantly the number of files involved) you post is not valid. A properly set up Omniback will include a routine purge of records of expired backups.
Ps. I wouldn't use a symbolic link for /var/opt/omni. If you need to move it, it needs to be a mounted filesystem. This is due to errors in Omniback software.
I wasn't thinking about omni filesystem (since it still has space).
Symlink is only a 'put out the fire' measure until proper planing has been done.
I wouldn't advise mixing SAN / local disk in rootvg, hence the symlink.
---------- Post updated at 02:00 AM ---------- Previous update was at 02:00 AM ----------
I wasn't thinking about omni filesystem (since it still has space).
Symlink is only a 'put out the fire' measure until proper planing has been done.
I wouldn't advise mixing SAN / local disk in rootvg, hence the symlink.
