/u01 is full as shown is system messages but not on ground

Hi there

i am an oracle DBA having solaris 10. in database alert log I am receiving following messages

Non Critical error ORA-48181 caught while writing to trace file "/u01/app/oracle/diag/rdbms/mygrid/mygrid12/trace/mygrid12_m000_210009.trc"
Error Messages: SVr4 Error: 28 : No space left on device 
writing to the above trace file is disabled for now on 
Error: Direct connection failure with ASM
WARNING:ASM communication error : op 0 state 0x0 (15055)

and in /var/adm/messages.0 showing following

May 20 14:08 xxxx ufs:  [ID 845546 kern.Notice]  NOTICE:alloc:  /u01:  file system full

out put of df -ah is as follwos

/dev/dsk/c0t500xxxxxxd0s4   439G   381G   54G   88%     /u01

all other filesystem have free space shown in df -ah. I can't write all of them

Nowhere is that file system which is full as all file systems are not 100 % full and have sufficient free space

please guide

Regards

Could you either show df -ah for the entire system or for all the /u0N partions.

On UFS filesystems there are two ways a filesystem can be full.

1. No free sectors
2. No free inodes

Please post the output of:

# df -i

to show inode usage.

applying df -i on bash , saying that unknown option

There is only one /u01 partition which I have shown or may be being beginner I am not understanding you as I don't see any other /u01 . There is only one mount point /u01 which points to the machine hardisk

ok a apply as df -oi /u01 and it shows following

FIlesystem                                                        iused             ifree             %iused           mounted on
/dev/dsk/c0t500xxxxxxd0s4                             34102175     21319009       62%                /u01

Regards

i already did and u can in above post that inode are only 62% used for /u01

So it seems that the filesystem space and inodes are not full so it's likely your Oracle limit on MAX_DUMP_FILE_SIZE is set too low.

You can see what it's currently set to:

SQL> show parameter max_dump_file_size

and increase it with, for example, to set to 500m:

SQL> alter system set max_dump_file_size='500m';

IMHO that's the first thing to try.

But it is also an OS warning, so the Operating System interfered, so maybe that parameter was set to unlimited and there were unusually big trace files and maybe also log files that were removed by Oracle after the fact?

Yes it is set to unlimited but when I see the size of adump, cdump or .trc files all of them are in mbs and they are all in /u01 filesystem so at this moment my instance is down and I can't even start it and receiving following error
ORA-09817 : write to audit file failed
SVR4 Error: 28: No space left on device
Additional_information: 9

Perhaps a filesystem corruption?
If Oracle is down then check it

umount /u01
fsck /u01
mount /u01

I thought also at something of the sort, as it happened to me one long ago, did not reply yet as I am still trying to figure what it was... but remembered once managed to umount (was not easy as a file was still open but could not be found as later I believed it was deleted...) after a few fsck it came back to normal...

i am not able to connect to ASM as well. why is that? like ASM are on FC SAN .

I do not understand how this is set up. I thought ASM works with raw partitions? But /u01 is cooked apparently ?

yes it worked with RAW partition

Ah OK, I understand, you still need a "cooked" filesystem to write trace and log files..

What happens when you execute:

fuser -c /u01

?

One of those processes may still have a file handle to a file on the filesystem that has been deleted and thus the diskspace is not freed. You can use lsof to find out which processes have deleted files.

When I execute the commmad fuser -c /u01
File is shown
1679c 1670c 1661ctm. 1636ctom 1629co and many of them

What does it mean

fsck /u01 shown following

/Dev/dsk/c0t50000cca....s4 is currently mounted

When processes hold it busy then it cannot be umounted.
Examine the PIDs shown by the fuser. The trailing letters show the kind of access, e. g. a c means "current work directory", and o means a file is opened.

ps -fp 1679
ps -fp 1670
...

Then try to terminate them in a friendly way.
E. g. use a shutdown procedure given by Oracle, or in an accessing shell cd out of it.

but i am worried what happen when my /u01 dismounted as there are 2 x instances ruuning as part of 2 Node RAC,. NOte that there is 7.2 GB of datat which i can delete from /u01. it is a folder at /u01/source. this sorce folder conatain sources of oracle etc so no use to me but ist i wan to copy it to sother SAN NFS share but it doesn't, like it did copied but not all data . i used cp -r /u01/source /export/home/oracle/source
the size is 7.2 Gb but when i copy it to destinaition the size is few Mbs. this is 2nd issue now

pl replt