HP UX 10.20 E9000 System Full?

Hi

We have an old GIS system running on unix 10.20. Our IT department have kinda washed their hands of it I'm having a problem where the system seems to but full. It had been working find for the last few years but now the drive seems to be full. It is only used for viewing old information so nothing has been added to it in years. I take it some sort of temp files are filling up the drive?. I'm a bit of a novice but have root access so should be able to try some solution if anyone has some?.

Cheers

Sorry probably should of put this on the Unix for dummies section

What kind of box is this a 9000/700 or 9000/800 (wkstation or server)? or is the E part of the server type ( E35 E55 ?)

I dont remember if 10.20 had model command
could you show us the output of bdf ?
and vgdisplay to start with?
Even if nothing is added, system logs continue to grow you know...

And the output of uname -a

1 Like

It's a E9000/813, which I'm told is on its last legs but that will be another thread(looking to back it up or move it to a newer system if that's even possible). I'm not in the same office as the box(using mobaxterm to access it) so used uname -a (HP-UX gis B.10.20 E9000/813) to get the details on it.

Yeah I was thinking it is some sort of log file filling up just don't know what to remove. I'm back in work in the morning will post up the bdf and vgdisplay outputs

Thanks vbe

Find recent big files

find / -fstype nfs -prune -o -mtime -30 -size +10000 -print
1 Like
model
 
9000/813/D330
 
Bdf
 
Filesystem          kbytes    used   avail %used Mounted on
/dev/vg00/lvol3     151552   62818   83194   43% /
/dev/vg00/lvol1      47829   17647   25399   41% /stand
/dev/vg00/lvol8     512000  512000       0  100% /var
/dev/vg00/lvol7     405504  273394  123857   69% /usr
/dev/vg00/lvol14   5218304 4611798  568719   89% /users
/dev/vg00/lvol13   2097152  942446 1082603   47% /u04
/dev/vg00/lvol12   2097152 1746199  329024   84% /u03
/dev/vg00/lvol11   2097152  815131 1201937   40% /u02
/dev/vg00/lvol10   2051553  642418 1203979   35% /u01
/dev/vg00/lvol6      32768   25390    7378   77% /tmp
/dev/vg00/lvol15    307200  183965  116551   61% /tell
/dev/vg00/lvol5     360448  180957  168383   52% /opt
/dev/vg00/lvol4      20480   20480       0  100% /home
 
--- Volume groups ---
VG Name                     /dev/vg00
VG Write Access             read/write     
VG Status                   available                 
Max LV                      255    
Cur LV                      15     
Open LV                     15     
Max PV                      16     
Cur PV                      7      
Act PV                      7      
Max PE per PV               2200         
VGDA                        14  
PE Size (Mbytes)            4               
Total PE                    4585    
Alloc PE                    4451    
Free PE                     134     
Total PVG                   0
 
 
 
find / -fstype nfs -prune -o -mtime -30 -size +10000 -print
/var/mail/root
/u01/app/oracle/product/8.0.3./network/log/listener.log
/u02/oradata/oi1p/rbs01.dbf
/u02/oradata/oi1p/system01.dbf
/u02/oradata/oi1p/temp01.dbf
/u02/oradata/oi1p/tools01.dbf
/u03/oradata/oi1p/gis02.dbf
/u03/oradata/oi1p/gis01.dbf
/u04/oradata/oi1p/gis_index01.dbf
/u04/oradata/oi1p/rbs02.dbf
/u04/oradata/oi1p/repository01.dbf
 
 
This is the error message I get when using the GIS program
 
vxfs: mesg 001: vx_nospace - /dev/vg00/lvol8 file system full (1 block extent)

so looks like vol8 is full?
[/FONT]

Hi,

This could be several things, I'd be tempted to start with /var/adm. /var/log, /var/spool or /var/mail

Having had a brief re-read of the post - I'd say that the most likely culprit here is the mail file for the root user, the most common problem in my experience is output from cron jobs. In addition, you seem to have an issue with /home, so you should have a quick look in there.

Files to lookout for are wtmp, wmtpx and standard log files also worth looking in t mail directories for users with large mail files.

Regards

Dave

1 Like

Yes lvol8 is full... and you have a D330 server... that means with optional 2 disk on FW-SCSI board or max 7 disks on standard internal SCSI... which seems to be your case...
My concern is that all is in the same VG, not cool at all in case of crash...
As I have no more HP boxes, I have to try to remember what 10.20 had and usabel on a D class, I would start by

swlist -l product |grep ONLINE*JFS 

but I will have to try uppercase, lowercase or mix.. We are looking for the onlineJFS product that if you have will save you... having to go single user to do weird manipulations...
Only this can be done only when you have freeded some space, for you have some unused blocks of disks: Free PE 134 That a bit more than 400MB...
Your /var is too small even for a 10.20 D class, I always allocated a minimum of 750 and that was short... Because is /var there is all the patches also, and a 10.20 fully patched thats more than 750MB...
Your case its oracle I suppose that writes in /var/tmp ( or /var/opt/oracle )
My suggestion as I dont what you know about HP-UX and how familiar you are with it ( I staterd on HP-UX 8.04...that was a long time ago on a 822 and a 855...) is you use SAM:
type sam -> there is somewhere where it says in one menu "trim logs? " Go there and trim all the logs If it complains about nop space ( you must be root though...)
go to /var/adm/syslog/and do a

 rm OLD* 

And see if that helps
then go in /var/tmp and try to remove all unwanted old files you find
but /var/tmp might be a sybolink link to /tmp ...in which case it will not help...
Have a look in /var/spool also
But most probably /var/adm will be the highest consumer for here lies also all the patches...
So the idea is to birng /var to about 90 % so that you can extaned it to about 750-780 MB so you keep about 100-150MB spare for any other space realted issue where that little amount can make all the difference

1 Like

I looked around and D3x0: five hot-swap trays for a Fast-Wide 68-pin SCSI-2 high-voltage differential hard drive !
So I wonder if you are not in RAID5...

I just noticed:

/dev/vg00/lvol4      20480   20480       0  100% /home

Your /home needs some cleanup too...
How many users are defined here?
-> Go in /home
then:

ll; du -sk * 

And show us the output!

And I need the output of

 vgdisplay -v vg00 

because depending of your disks configuration we may not have all that much free space finally... (depending if you are in mirror or RAIDX...)

1 Like

Thanks lots to digest.

Found a large file var/adm/syslog/mail.log 80Mb deleted that and got it back down to 85%

One other big file I found in var/mail/root is 107Mb I take it is OK to delete this?

I trimed the log files too so not looking too bad now

Hi,

It should be OK to delete /var/mail/root - but you might want to have a look at it's contents first and check what has generated it.

Just type "mail" as the root user and you can have a quick look at the contents of the mails. If you are not interested in the contents then just type;

root@sys # >/var/mail/root

or

cp /dev/null /var/mail/root

As I said earlier it is most probably output from cron or other system utilities.

Regards

Dave

1 Like
# ll; du -sk *
total 1650
-rw-------   1 root       sys            209 Oct 10 15:32 .TTauthority
-rw-------   1 root       sys           1031 Oct 10 15:32 .Xauthority
drwxr-xr-x  12 root       sys           1024 Oct 10 16:15 .dt
-rwxr-xr-x   1 root       sys           3948 Oct 15  1998 .dtprofile
drwx------   2 root       sys             96 Dec 11  1998 .elm
-rw-rw-rw-   1 root       sys             56 Nov  7  2005 .infrtool.ini
drwx------   3 root       users         1024 Jan 22  2000 .netscape
-r--r--r--   1 bin        bin           2715 Apr 10  2008 .profile
-r--------   1 root       sys             49 Mar 28  2008 .rhosts
-r--------   1 root       sys             43 Mar 28  2008 .rhosts.old
-rw-------   1 root       sys           2438 Oct 25  2013 .sh_history
drwxr-xr-x   3 root       root            96 Sep 15  1998 .sw
drwxrwxrwx   3 root       development     96 Dec 18  1997 MIS
drwx------   2 root       sys             96 Oct 23  1998 Mail
drwxr-xr-x   2 root       root            96 Sep 15  1998 SD_CDROM
drwxrwxrwx   2 root       sys             96 May 28  2003 SRM_CENTRAL
drwxrwxr-x   2 root       root          1024 Oct  9 16:30 TT_DB
-rw-------   1 root       sys          24623 Jul  5  2006 a.out
-rw-rw-rw-   1 root       sys              0 Sep 30  1998 adviser.out
lr-xr-xr-t   1 root       sys              8 Sep 15  1998 bin -> /usr/bin
drwxrwxrwx   2 root       sys             96 Oct 11  2000 cdrom
drwxrwxrwx   2 root       root            96 Nov  7  2000 core
-rw-rw-rw-   1 root       sys           2241 Dec 31  1999 correct_cron
drwxrwxrwx   2 root       sys             96 Jan  4  2006 database
-rw-------   1 root       mail            61 Aug 12  2005 dead.letter
dr-xr-xr-x  13 bin        bin           3072 Aug 11 14:02 dev
dr-xr-xr-x  25 bin        bin           6144 Oct 10 16:22 etc
-rw-rw-rw-   1 root       sys           1365 Oct 16  2000 file
drwxrwxr-x   2 root       sys           1024 Apr 18  2011 frans
drwxrwxrwx   2 root       sys             96 Feb 29  2000 frans_new
drwxrwxr-x   2 root       users           96 Feb 27  2001 gisdev
-rw-rw-r--   1 root       sys              0 Sep 30  1998 glance.err
drwxr-xr-x  56 root       root          2048 Apr 14  2008 home
drwxrwxrwx   2 root       sys           1024 Oct 11  2000 hpscripts
drwxrwxrwx   2 root       sys             96 Nov  7  2005 infrtool
-rwx------   1 root       sys            390 Jan 14  2005 kpi_script
lr-xr-xr-t   1 root       sys              8 Sep 15  1998 lib -> /usr/lib
drwxr-xr-x   2 root       root            96 Sep 15  1998 lost+found
drwxrwxrwx   2 root       sys             96 Jan 16  1998 main_maps
dr-xr-xr-x   1 root       root           512 Aug 11 14:02 net
-rw-r-----   1 root       sys         215040 Apr 18  2011 net_lj8150.tar
-rw-------   1 root       sys              0 Sep 15  1998 nohup.out
-rw-rw-rw-   1 root       sys           2242 Dec 31  1999 nye_cron
dr-xr-xr-x  25 bin        bin           1024 Apr 10  2008 opt
-rw-r-----   1 root       sys          17623 Feb 24  2012 oracle_audit
drwx------   2 root       sys             96 Mar 28  2012 oralce_audit
-rwxr--r--   1 root       sys           3056 Jan  6  2005 privgroups1.22.pl
-rwxr--r--   1 root       sys            967 Jan  6  2005 privusers.ksh
dr-xr-xr-x  12 bin        bin           3072 Jan 23  2003 sbin
-rw-rw-rw-   1 root       sys         204800 Sep 19  2002 secng-1.51.tar
-rw-rw-rw-   1 root       sys              0 Nov 26  1998 sh.history
drwxrwxr-x  82 root       users         3072 Aug 10  1999 soft_gis
dr-xr-xr-x   4 bin        bin           1024 Jan 23  2003 stand
drwxr-xr-x   8 root       root          1024 Apr 24  2003 tell
lrwxrwxr-x   1 root       users            5 Sep 23  1999 tell1 -> /tell
drwxrwxrwt  17 bin        bin         149504 Oct 10 16:50 tmp
-rw-r--r--   1 root       sys              4 Dec 30  2013 tmp.ftp.1004
dr-xr-xr-x   3 root       root            96 Mar  6  2000 tmp_mnt
drwxrwxrwx   2 root       sys           1024 Mar 25  1998 tools
drwxr-xr-x   5 root       root          1024 Apr  1  2008 u01
drwxr-xr-x   8 root       root          1024 Apr 10  2008 u02
drwxr-xr-x   4 root       root            96 Mar 27  2000 u03
drwxr-xr-x   6 root       root          1024 Mar 23  2000 u04
drwxr-xr-x  96 i94883     users         3072 Oct  1  2013 users
dr-xr-xr-x  26 bin        bin           1024 Aug 27  1999 usr
drwxr-xr-x  19 bin        bin           1024 Sep 15  1998 var
-r-xr-xr-x   1 bin        bin         155648 Feb 14  2003 xfs.1020
0       MIS
0       Mail
0       SD_CDROM
0       SRM_CENTRAL
39      TT_DB
25      a.out
0       adviser.out
0       bin
0       cdrom
0       core
3       correct_cron
0       database
1       dead.letter
19      dev
2657    etc
2       file
4430    frans
0       frans_new
26      gisdev
0       glance.err
18937   home
328     hpscripts
1489    infrtool
1       kpi_script
0       lib
0       lost+found
0       main_maps
1       net
210     net_lj8150.tar
0       nohup.out
3       nye_cron
179048  opt
18      oracle_audit
117     oralce_audit
3       privgroups1.22.pl
1       privusers.ksh
14122   sbin
200     secng-1.51.tar
0       sh.history
35272   soft_gis
17647   stand
179700  tell
0       tell1
23304   tmp
1       tmp.ftp.1004
0       tmp_mnt
8       tools
642418  u01
813261  u02
1744585 u03
940768  u04
4580789 users
268658  usr
310941  var
152     xfs.1020
  
vgdisplay -v vg00
 Current LE                  99        
   Allocated PE                198         
   Used PV                     2       
   LV Name                     /dev/vg00/lvol5
   LV Status                   available/syncd           
   LV Size (Mbytes)            352             
   Current LE                  88        
   Allocated PE                176         
   Used PV                     2       
   LV Name                     /dev/vg00/lvol8
   LV Status                   available/syncd           
   LV Size (Mbytes)            500             
   Current LE                  125       
   Allocated PE                250         
   Used PV                     2       
   LV Name                     /dev/vg00/lvol6
   LV Status                   available/syncd           
   LV Size (Mbytes)            32              
   Current LE                  8         
   Allocated PE                16          
   Used PV                     2       
   LV Name                     /dev/vg00/lvol4
   LV Status                   available/syncd           
   LV Size (Mbytes)            20              
   Current LE                  5         
   Allocated PE                10          
   Used PV                     2       
   LV Name                     /dev/vg00/lvol10
   LV Status                   available/syncd           
   LV Size (Mbytes)            2048            
   Current LE                  512       
   Allocated PE                512         
   Used PV                     4       
   LV Name                     /dev/vg00/lvol11
   LV Status                   available/syncd           
   LV Size (Mbytes)            2048            
   Current LE                  512       
   Allocated PE                512         
   Used PV                     2       
   LV Name                     /dev/vg00/lvol12
   LV Status                   available/syncd           
   LV Size (Mbytes)            2048            
   Current LE                  512       
   Allocated PE                512         
   Used PV                     2       
   LV Name                     /dev/vg00/lvol13
   LV Status                   available/syncd           
   LV Size (Mbytes)            2048            
   Current LE                  512       
   Allocated PE                512         
   Used PV                     2       
   LV Name                     /dev/vg00/lvol14
   LV Status                   available/syncd           
   LV Size (Mbytes)            5096            
   Current LE                  1274      
   Allocated PE                1274        
   Used PV                     3       
   LV Name                     /dev/vg00/lvol9
   LV Status                   available/syncd           
   LV Size (Mbytes)            200             
   Current LE                  50        
   Allocated PE                50          
   Used PV                     1       
   LV Name                     /dev/vg00/lvol15
   LV Status                   available/syncd           
   LV Size (Mbytes)            300             
   Current LE                  75        
   Allocated PE                75          
   Used PV                     1       

   --- Physical volumes ---
   PV Name                     /dev/dsk/c0t5d0
   PV Status                   available    
   Total PE                    507     
   Free PE                     0       
   PV Name                     /dev/dsk/c0t8d0
   PV Status                   available    
   Total PE                    508     
   Free PE                     0       
   PV Name                     /dev/dsk/c0t9d0
   PV Status                   available    
   Total PE                    508     
   Free PE                     0       
   PV Name                     /dev/dsk/c0t10d0
   PV Status                   available    
   Total PE                    508     
   Free PE                     0       
   PV Name                     /dev/dsk/c0t11d0
   PV Status                   available    
   Total PE                    1023    
   Free PE                     0       
   PV Name                     /dev/dsk/c1t14d0
   PV Status                   available    
   Total PE                    508     
   Free PE                     0       
   PV Name                     /dev/dsk/c1t15d0
   PV Status                   available    
   Total PE                    1023    
   Free PE                     134   

found alot of log files home/sysadm/performance/gis

Things look to be in better shape.

/dev/vg00/lvol3     151552   61477   84555   42% /
/dev/vg00/lvol1      47829   17647   25399   41% /stand
/dev/vg00/lvol8     512000  318624  181482   64% /var
/dev/vg00/lvol7     405504  273395  123856   69% /usr
/dev/vg00/lvol14   5218304 4592606  586713   89% /users
/dev/vg00/lvol13   2097152  942446 1082603   47% /u04
/dev/vg00/lvol12   2097152 1746199  329024   84% /u03
/dev/vg00/lvol11   2097152  815131 1201937   40% /u02
/dev/vg00/lvol10   2051553  642418 1203979   35% /u01
/dev/vg00/lvol6      32768   25390    7378   77% /tmp
/dev/vg00/lvol15    307200  183965  116551   61% /tell
/dev/vg00/lvol5     360448  180957  168383   52% /opt
/dev/vg00/lvol4      20480   11031    9164   55% /home

Thanks again

So as expected lvol8 is in mirror so you need 400MB to add 200...
I see your output is not complete since its missing lvol1 lvol2 lvol3 etc...
Which could be interesting, to know if they are mirrored too, for you have some FS that arent...
I see you cleaned up /home and I think you could do some cleaning in /tmp

What output do you get from swapinfo -tam ?
(never worked on a D3X0 but had to administer a lot of D220,D230,D270... I had a D350 to administer but never found it...
I loved those little boxes... never had serious issues( a disk from time to time but since mirrored never had downtime...) )

/dev/vg00/lvol1      47829   17647   25399   41% /stand 
#  /dev/vg00/lvol2 is the default swap in HP-UX and we will know by swapinfo its usage..

/dev/vg00/lvol3     151552   61477   84555   42% /
/dev/vg00/lvol4      20480   11031    9164   55% /home
/dev/vg00/lvol5     360448  180957  168383   52% /opt
/dev/vg00/lvol6      32768   25390    7378   77% /tmp
/dev/vg00/lvol7     405504  273395  123856   69% /usr
/dev/vg00/lvol8     512000  318624  181482   64% /var

/dev/vg00/lvol14   5218304 4592606  586713   89% /users
/dev/vg00/lvol13   2097152  942446 1082603   47% /u04
/dev/vg00/lvol12   2097152 1746199  329024   84% /u03
/dev/vg00/lvol11   2097152  815131 1201937   40% /u02
/dev/vg00/lvol10   2051553  642418 1203979   35% /u01
/dev/vg00/lvol15    307200  183965  116551   61% /tell

Where is /dev/vg00/lvol9 ? What was it if not used anymore? and if not used it can be recycled into free PE

Do you have a DDS tape device on it ?

 ioscan -funC tape 

All the tar_files in / if you want to keep them there can be gzipped:

 gzip *.tar 
# swapinfo -tam
             Mb      Mb      Mb   PCT  START/      Mb
TYPE      AVAIL    USED    FREE  USED   LIMIT RESERVE  PRI  NAME
dev         512      19     493    4%       0       -    1  /dev/vg00/lvol2
dev         200      18     182    9%       0       -    1  /dev/vg00/lvol9
reserve       -      99     -99
memory      289     164     125   57%
total      1001     300     701   30%       -       0    -
 
# ioscan -funC tape
Class     I  H/W Path    Driver      S/W State H/W Type  Description
=====================================================================
tape      0  8/16/5.0.0  stape       CLAIMED   DEVICE    HP      C1533A
                        /dev/rmt/0m            /dev/rmt/c2t0d0BEST  
                        /dev/rmt/0mb           /dev/rmt/c2t0d0BESTb 
                        /dev/rmt/0mn           /dev/rmt/c2t0d0BESTn 
                        /dev/rmt/0mnb          /dev/rmt/c2t0d0BESTnb

AFAIK it does have DDS tape device but I have never actually seen the unix box it is in another office on the other side of the city. I know they had some issue with drive failure but no one want to spend any money on the system but its great for old GIS records

Haha We got lvol9 back! Its a swap device! Hummmm

I ask you that because I hope you have a complete Ignite-UX installed! This is to make a bootable OS tape, you can use to reconfigure or recover the system, then you use ordinary backup for the rest... Thats of course up to you...

The interesting thing about what you produced so far if not complete shows:

 --- Physical volumes ---
   PV Name                     /dev/dsk/c0t5d0
   PV Status                   available    
   Total PE                    507     
   Free PE                     0       
   PV Name                     /dev/dsk/c0t8d0
   PV Status                   available    
   Total PE                    508     
   Free PE                     0       
   PV Name                     /dev/dsk/c0t9d0
   PV Status                   available    
   Total PE                    508     
   Free PE                     0       
#----------------------------------
AND!
   PV Name                     /dev/dsk/c0t10d0
   PV Status                   available    
   Total PE                    508     
   Free PE                     0       
   PV Name                     /dev/dsk/c0t11d0
   PV Status                   available    
   Total PE                    1023    
   Free PE                     0       
   PV Name                     /dev/dsk/c1t14d0
   PV Status                   available    
   Total PE                    508     
   Free PE                     0       
   PV Name                     /dev/dsk/c1t15d0
   PV Status                   available    
   Total PE                    1023    
   Free PE                     134

You seem to have 2 controllers and I wonder if these 2 sets of disks are not the ones in mirror...

Why are you swapping? Can you see how much RAM you have?

grep hysica /var/adm/syslog/syslog.log 

Though it might return nothing since you trimmed the logs...

The system died a few years ago and they struggled to find a working backup tape. The had to get the old system admin out of retirement. I'm just worried it will happen again and we lose all the information.I am only learning as I go with unix but it been fun. It was a pretty advanced GIS system Mountaintop not far off what we use today Smallworld.

# grep hysica /var/adm/syslog/syslog.log
Aug 11 14:02:45 gis vmunix:     physical page size = 4096 bytes, logical page size = 4096 bytes
Aug 11 14:02:45 gis vmunix:     Physical: 393216 Kbytes, lockable: 294292 Kbytes, available: 340484 Kbytes

No idea why they are swapping I'm only a novice.