Filesystem Full

I noticed that whenever something is printed from my workstation, the available disk space in the /dev/dsk/c0t0d0s0 decreases considerably. Hence, after using my workstation for sometime, I encounter an error message: "Filesystem Full" that prevents me from printing any further.
Is there a way to increase or release the available disk space without the need of reformatting my hard disk? During the Solaris installation, 741263 Kbytes is the prompted size for /dev/dsk/c0t0d0s0 and it was set as it is.

The following are the number of free disk blocks and files that I have tracked and saved for the past three consecutive days:

June 25, 2005 (morning):
hci408:~ >df -k
Filesystem kbytes used avail capacity Mounted on
/dev/dsk/c0t0d0s0 741263 560393 121569 83% /
/proc 0 0 0 0% /proc
fd 0 0 0 0% /dev/fd
mnttab 0 0 0 0% /etc/mnttab
swap 791888 16 791872 1% /var/run
swap 793088 1216 791872 1% /tmp
/dev/dsk/c0t0d0s7 36592677 793017 35433734 3% /export/home
/dev/dsk/c0t0d0s4 61407 9 55258 1% /var/dump
/dev/dsk/c0t0d0s3 493527 20 444155 1% /var/tmp

June 25, 2005 (evening):
hci408:~ >df -k
Filesystem kbytes used avail capacity Mounted on
/dev/dsk/c0t0d0s0 741263 560033 121929 83% /
/proc 0 0 0 0% /proc
fd 0 0 0 0% /dev/fd
mnttab 0 0 0 0% /etc/mnttab
swap 790600 16 790584 1% /var/run
swap 792624 2040 790584 1% /tmp
/dev/dsk/c0t0d0s7 36592677 798861 35427890 3% /export/home
/dev/dsk/c0t0d0s4 61407 9 55258 1% /var/dump
/dev/dsk/c0t0d0s3 493527 20 444155 1% /var/tmp

June 26, 2005 (morning):
hci408:~ >df -k
Filesystem kbytes used avail capacity Mounted on
/dev/dsk/c0t0d0s0 741263 559239 122723 83% /
/proc 0 0 0 0% /proc
fd 0 0 0 0% /dev/fd
mnttab 0 0 0 0% /etc/mnttab
swap 776016 16 776000 1% /var/run
swap 778040 2040 776000 1% /tmp
/dev/dsk/c0t0d0s7 36592677 783881 35442870 3% /export/home
/dev/dsk/c0t0d0s4 61407 9 55258 1% /var/dump
/dev/dsk/c0t0d0s3 493527 43 444132 1% /var/tmp

June 26, 2005 (evening):
hci408:~ >df -k
Filesystem kbytes used avail capacity Mounted on
/dev/dsk/c0t0d0s0 741263 559302 122660 83% /
/proc 0 0 0 0% /proc
fd 0 0 0 0% /dev/fd
mnttab 0 0 0 0% /etc/mnttab
swap 772480 16 772464 1% /var/run
swap 774504 2040 772464 1% /tmp
/dev/dsk/c0t0d0s7 36592677 791427 35435324 3% /export/home
/dev/dsk/c0t0d0s4 61407 9 55258 1% /var/dump
/dev/dsk/c0t0d0s3 493527 51 444124 1% /var/tmp

June 27, 2005 (morning):
hci408:~ >df -k
Filesystem kbytes used avail capacity Mounted on
/dev/dsk/c0t0d0s0 741263 559365 122597 83% /
/proc 0 0 0 0% /proc
fd 0 0 0 0% /dev/fd
mnttab 0 0 0 0% /etc/mnttab
swap 754224 16 754208 1% /var/run
swap 756256 2048 754208 1% /tmp
/dev/dsk/c0t0d0s7 36592677 759331 35467420 3% /export/home
/dev/dsk/c0t0d0s4 61407 9 55258 1% /var/dump
/dev/dsk/c0t0d0s3 493527 51 444124 1% /var/tmp

June 27, 2005 (evening):
hci408:~ >df -k
Filesystem kbytes used avail capacity Mounted on
/dev/dsk/c0t0d0s0 741263 559414 122548 83% /
/proc 0 0 0 0% /proc
fd 0 0 0 0% /dev/fd
mnttab 0 0 0 0% /etc/mnttab
swap 744952 16 744936 1% /var/run
swap 746984 2048 744936 1% /tmp
/dev/dsk/c0t0d0s7 36592677 764606 35462145 3% /export/home
/dev/dsk/c0t0d0s4 61407 9 55258 1% /var/dump
/dev/dsk/c0t0d0s3 493527 51 444124 1% /var/tmp

Below are some information regarding my workstation:

hci408:~ >wsinfo
Workstation Name: hci408
Workstation Type: SUNW,Sun-Blade-100;sparc;sun4u

          Host Id: 830ee018
 Internet Address: 150.10.128.1
   Network Domain: [Couldn't Ascertain]

Physical Memory (RAM): 512 Megabytes
Virtual Memory (RAM): 882 Megabytes
Virtual Memory in Use: approx. 23%

 Operating System: SunOS Release 5.8 Generic_108528-18
    Window System: OpenWindows Version 6.4.1

hci408:~ >more /etc/release
Solaris 8 HW 12/02 s28s_hw1wos_06a SPARC
Copyright 2002 Sun Microsystems, Inc. All Rights Reserved.
Assembled 12 December 2002

Thank you.

Excuse me but is there anyone of you out there who could get me an input about my query regarding the dilemma that I'm facing right now concerning the "Filesystem Full" error message that I always encounter in my workstation?

Thanks in advance!

Run this command to find what files have changed over the past day:
find / -mount -type f -mtime -1 -print | xargs ls -l

Dear Mr. Perderabo,

I'm sorry for the late response. Please accept my apology.

The following are the information you told me to check in my workstation:

Date Checked: 12 July 2005 (evening)

hci408:~ >find / -mount -type f -mtime -1 -print |xargs ls -l
find: cannot read dir /lost+found: Permission denied
find: cannot read dir /var/sadm/patch/: Permission denied
find: cannot read dir /var/spool/mqueue: Permission denied
find: cannot read dir /var/spool/lp/fifos/private: Permission denied
find: cannot read dir /var/spool/lp/fifos/public: Permission denied
find: cannot read dir /var/spool/lp/requests/hci408: Permission denied
find: cannot read dir /var/spool/lp/tmp: Permission denied
find: cannot read dir /var/dt/sdtlogin: Permission denied
find: cannot read dir /var/crash/hci408: Permission denied
-rw-rw-r-- 1 lp lp 19 Jul 12 08:13 /etc/lp/printers/lp408/faultMessage
-r--r--r-- 1 root other 10892 Jul 12 20:08 /var/adm/lastlog
-rw-r--r-- 1 root other 14341 Jul 12 08:12 /var/adm/messages
-rw-r--r-- 1 root bin 3720 Jul 12 19:59 /var/adm/utmpx
-rw-r--r-- 1 adm adm 64225800 Jul 12 20:08 /var/adm/wtmpx
-rw------- 1 root root 24945 Jul 12 03:30 /var/cron/log
-rw-r--r-- 1 root other 86916 Jul 12 08:16 /var/log/syslog
-rw-r----- 1 lp lp 198 Jul 12 08:12 /var/lp/logs/lpsched
-rw-r----- 1 root sn408 180 Jul 12 08:13 /var/lp/logs/requests
-rw-rw---- 1 user408 mail 0 Jul 12 08:33 /var/mail/user408

hci408:~ >df -k
Filesystem kbytes used avail capacity Mounted on
/dev/dsk/c0t0d0s0 741263 580136 101826 86% /
/proc 0 0 0 0% /proc
fd 0 0 0 0% /dev/fd
mnttab 0 0 0 0% /etc/mnttab
swap 737264 16 737248 1% /var/run
swap 737272 24 737248 1% /tmp
/dev/dsk/c0t0d0s3 493527 124 444051 1% /var/tmp
/dev/dsk/c0t0d0s7 36592677 916917 35309834 3% /export/home
/dev/dsk/c0t0d0s4 61407 9 55258 1% /var/dump

Date Checked: 13 July 2005 (morning)

hci408:~ >find / -mount -type f -mtime -1 -print |xargs ls -l
find: cannot read dir /lost+found: Permission denied
find: cannot read dir /var/sadm/patch/: Permission denied
find: cannot read dir /var/spool/mqueue: Permission denied
find: cannot read dir /var/spool/lp/fifos/private: Permission denied
find: cannot read dir /var/spool/lp/fifos/public: Permission denied
find: cannot read dir /var/spool/lp/requests/hci408: Permission denied
find: cannot read dir /var/spool/lp/tmp: Permission denied
find: cannot read dir /var/dt/sdtlogin: Permission denied
find: cannot read dir /var/crash/hci408: Permission denied
-rw-rw-r-- 1 lp lp 19 Jul 12 08:13 /etc/lp/printers/lp408/faultMessage
-r--r--r-- 1 root other 10892 Jul 13 04:47 /var/adm/lastlog
-rw-r--r-- 1 root other 14341 Jul 12 08:12 /var/adm/messages
-rw-r--r-- 1 root bin 4092 Jul 12 23:12 /var/adm/utmpx
-rw-r--r-- 1 adm adm 64263000 Jul 13 04:47 /var/adm/wtmpx
-rw------- 1 root root 25246 Jul 13 03:30 /var/cron/log
-rw-r--r-- 1 root other 86916 Jul 12 08:16 /var/log/syslog
-rw-r----- 1 lp lp 198 Jul 12 08:12 /var/lp/logs/lpsched
-rw-r----- 1 root sn408 362 Jul 13 03:29 /var/lp/logs/requests
-rw-rw---- 1 user408 mail 0 Jul 12 08:33 /var/mail/user408

hci408:~ >df -k
Filesystem kbytes used avail capacity Mounted on
/dev/dsk/c0t0d0s0 741263 580168 101794 86% /
/proc 0 0 0 0% /proc
fd 0 0 0 0% /dev/fd
mnttab 0 0 0 0% /etc/mnttab
swap 726624 16 726608 1% /var/run
swap 727832 1224 726608 1% /tmp
/dev/dsk/c0t0d0s3 493527 140 444035 1% /var/tmp
/dev/dsk/c0t0d0s7 36592677 885313 35341438 3% /export/home
/dev/dsk/c0t0d0s4 61407 9 55258 1% /var/dump

You're not running that command as root. If you're not the administrator of that box, you need to refer this problem to the person that is. Any admin looking at that output will instantly see a major problem: a 64 MB wtmpx file.

Dear Mr. Perderabo,

Below is the information that I have saved last night:

hci408:~ >pwd
/export/home/user408
hci408:~ >cd /
hci408:/ >su
# pwd
/
# find / -mount -type f -mtime -1 -print | xargs ls -l
-rw-r--r-- 1 root root 4 Jul 13 06:07 /dev/.devfsadm_dev.lock
-rw-r--r-- 1 root root 57344 Jul 13 06:07 /dev/.devlink_db
-rw-r--r-- 1 root root 2236 Jul 13 06:07 /etc/.cpr_config
-rw-r--r-- 1 root other 314 Jul 13 06:07 /etc/coreadm.conf
-rw-r--r-- 1 root root 55 Jul 13 06:07 /etc/dfs/sharetab
-rw-r--r-- 1 root other 239 Jul 13 06:07 /etc/dumpadm.conf
-rw-r--r-- 1 root sys 620 Jul 13 06:07 /etc/rmtab
-rw-r--r-- 1 root root 4 Jul 13 06:07 /etc/saf/zsmon/_pid
-rw-r--r-- 1 root root 1024 Jul 13 06:09 /TT_DB/property_table.ind
-rw-r--r-- 1 root root 3072 Jul 13 06:09 /TT_DB/property_table.rec
-rw-r--r-- 1 root root 1024 Jul 13 06:09 /TT_DB/property_table.var
-r--r--r-- 1 root other 10892 Jul 13 19:23 /var/adm/lastlog
-rw-r--r-- 1 root other 21519 Jul 13 09:10 /var/adm/messages
-rw------- 1 root root 216 Jul 13 19:27 /var/adm/sulog
-rw-r--r-- 1 root bin 3720 Jul 13 19:26 /var/adm/utmpx
-rw-r--r-- 1 adm adm 68356488 Jul 13 19:23 /var/adm/wtmpx
-rw------- 1 root root 25398 Jul 13 06:07 /var/cron/log
-rw-r--r-- 1 root root 8609 Jul 13 06:07 /var/dmi/db/1l.comp
-rw-r--r-- 1 root root 5285 Jul 13 06:07 /var/dmi/db/1l.tbl
-rw------- 1 root root 44 Jul 13 06:07 /var/dt/A:0-8uaaIa
-rw-r--r-- 1 root root 0 Jul 13 06:07 /var/dt/Xerrors
-rw-r--r-- 1 root root 4 Jul 13 06:07 /var/dt/Xpid
-rw-r--r-- 1 root other 88727 Jul 13 11:26 /var/log/syslog
-rw-r----- 1 root sn408 0 Jul 13 06:09 /var/lp/logs/lpNet
-rw-r----- 1 lp lp 0 Jul 13 06:09 /var/lp/logs/lpsched
-rw-r----- 1 root sn408 540 Jul 13 08:36 /var/lp/logs/requests
-rw-rw---- 1 user408 mail 1238 Jul 13 11:26 /var/mail/user408
-rw------- 1 root root 4 Jul 13 06:07 /var/sadm/wbem/logr/cimbootserver.pid
-rw-r--r-- 1 root root 6456 Jul 13 06:07 /var/saf/_log
-rw-r--r-- 1 root sys 21373 Jul 13 06:07 /var/saf/zsmon/log
-rw-r--r-- 1 root root 22 Jul 13 06:07 /var/snmp/snmpdx.st
-rw-rw-r-- 1 lp lp 0 Jul 13 06:07 /var/spool/lp/SCHEDLOCK
-rw-r--r-- 1 lp lp 14 Jul 13 08:35 /var/spool/lp/tmp/hci408/.SEQF
-rw-r----- 1 root root 4 Jul 13 06:07 /var/spool/print/.printd.lock
-rw-r--r-- 1 daemon daemon 10 Jul 13 06:07 /var/statmon/stat

Unless you really need that wtmpx file for something, you should trim it. If nothing else:
cp /dev/null /var/adm/wtmpx
and then watch it. It shouldn't be growing that fast.

I don't exactly know what wtmpx file does in my system. I am quite skeptical whether I should trim this file or not because I don't know how to do that. Could you please reiterate further what does cp /dev/null /var/adm/wtmpx actually does? Thanks!

/var/adm/wtmpx contains the history of user access and adminstrative/accounting information for the utmpx database.
Yoiu can use 'last' command looks in the /var/adm/wtmpx file, which
records all logins and logouts, for information about a
user, a terminal, or any group of users and terminals.

( If accounting isn't enabled, clean up wtmp and wtmpx )
----------------------
You can refer to man pages for wtmpx and last.

cp /dev/null file
will leave you with an empty file. Empty the file or let system run out of disk space and crash. Whether you have accounting enabled on not, when was the last time you used wtmpx? The system doesn't need that data. It is saving it in case you want to see it. You have allowed the file to get too big to be useful. I have frequently copied /dev/null to /var/adm/wtmpx on solaris systems. It won't cause a system problem. If you let the system run out of space and crash, do you have the expertise to deal with that?

Besides you need to determine why it is growing so fast. It's too hard to do that while it's that large.

Sorry but I don't know how to check if the accounting is enabled or not. Could you please let me know? Before using the command cp /dev/null /var/adm/wtmpx, I still have to refer this matter to the customer support of the software that I am using right now whether this wtmpx file is being used by their software or not. This is just to confirm that our field operation won't be affected by such step. Moreover, I don't know if I am accessing wtmpx files or not and why is it growing that fast. For your info, I have temporarily stopped printing from my workstation until this problem is solved.

Thanks a lot for your help guys. I really appreciate it.

I ran the command �last -a -f /var/adm/wtmpx | more� to investigate what causes wtmpx file to grow fast and I found out that whenever we transfer a file via ftp from our workstation to pc and vice-versa, the size of this file increases at the same time. For your information, we do an ftp transfer every now and then to do a regular check to the file that is being generated during our field operation.

Could you please let me know how to disable the accounting in case we need to clean up the wtmpx file later on?

I am still waiting for the response from the Customer Support of the Software that we are using whether this wtmpx file is being utilized by their software to monitor any incongruities during our operation or not. I will let you know as soon as I get a reply from them. In the meantime, thanks a lot for all the help!

Accounting is a bunch of different things. They all write to /var and would have shown up when you did that find. Connect-time accounting is the only thing you have running. The only way to disable that is to remove wtmpx. That is legal and won't cause any system problems. But then you can't see who has been logging in to your system. So no one does that.

With most ftp servers, though, you can cause it to stop recording in wtmpx. If you are using Sun's ftpd, this is a -W option. You are probably running it from inetd, so you would edit /etc/inetd.conf and add the -W to the ftpd line. Then you need to "kill -1" inetd to make it reread the file. Read "man inetd" and "man ftpd".

You need to:
cp wtmpx wtmpx.old
cp /dev/null wtmpx
or something like that on a regular basis. Most people use a cron job.

Thanks once again for the info inetd and ftpd. While waiting for the response from our Software Customer Support, I'm still wondering that if ever I do a "cp /dev/null /var/adm/wtmpx", would it also increase the available disk space in my filesystem particularly /dev/dsk/c0t0d0s0 directory since wtmpx is on a different directory?

What filesystem do think contains that file? Not sure? Do:
df -k /var/adm/wtmpx

I have already ran �cp /dev/null /var/adm/wtmpx� and it seems that everything is running smoothly along with the software that we are using. Below is the result after emptying wtmpx file:

# df -k
Filesystem kbytes used avail capacity Mounted on
/dev/dsk/c0t0d0s0 741263 518123 163839 76% /
/proc 0 0 0 0% /proc
fd 0 0 0 0% /dev/fd
mnttab 0 0 0 0% /etc/mnttab
swap 727392 16 727376 1% /var/run
swap 728240 864 727376 1% /tmp
/dev/dsk/c0t0d0s7 36592677 913706 35313045 3% /export/home
/dev/dsk/c0t0d0s4 61407 9 55258 1% /var/dump
/dev/dsk/c0t0d0s3 493527 42 444133 1% /var/tmp

As you've noticed, the available used capacity in /dev/dsk/c0t0d0s0 directory decreased from 88% down to 76%. We gained about 12% more disk space now. I guess this would be enough to keep us going without the need of reformatting our hard disk anymore. Is there any more file/s that we need to empty to give us more additional disk space?

Everytime I open /dev/dsk/c0t0d0s0 directory, the only message I get is, �No open method defined for c0t0d0s0�. Obviously, it does not show what files are stored in it.

After running, �df -k /var/adm/wtmpx�, it came up with the following results:
Filesystem kbytes used avail capacity Mounted on
/dev/dsk/c0t0d0s0 741263 518123 163839 76% /

In additing to this, all files in /var/adm directory resulted to the same output after running df -k.

That means you don't have a separate filesystem for /var or /var/adm (which is a very bad idea, IMHO) and hence the wtmp file is gobbling up your root fs.

To find out which files are using how much you can use the following command:

# du -ks /* | sort -rn

which would give3 you a sorted list of directories along which the number of Kilobytes used by the files in them. Go through the biggest ones and find out, if they contain files you could remove or shorten. For instance, if the output is like that:

123480 /bin
112749 /fubar
...

you can use the command

# du -ks /fubar/* | sort -rn

to investigate further what needs so much space in /fubar and so on.

I'd recommend you ask someone knowledgeable in *NIX to assist you, though, because it needs some system expertise to interpret the output of these commands.

bakunin