Space not getting released on the mountpoint

root@atldc-oragrid-ux01:/oragrid_01> du -sg *
0.58    11gR2gridBase
47.31   Grid_11203
4.17    app
0.00    lost+found
0.01    oraInventory
root@atldc-oragrid-ux01:/oragrid_01> cd Grid_11203
root@atldc-oragrid-ux01:/oragrid_01/Grid_11203> du -sg *|sort
0.00    JRE
0.00    OPatch_old
0.00    assistants
0.00    auth
0.00    cfgtoollogs
0.00    clone
0.00    csmig
0.00    css
0.00    ctss
0.00    dbs
0.00    dc_ocm
0.00    deinstall
0.00    demo
0.00    diagnostics
0.00    eons
0.00    gipc
0.00    gns
0.00    gpnp
0.00    has
0.00    hs
0.00    instantclient
0.00    ldap
0.00    mdns
0.00    mesg
0.00    ohasd
0.00    ologgerd
0.00    opmn
0.00    oraInst.loc
0.00    osysmond
0.00    owm
0.00    patch
0.00    plsql
0.00    precomp
0.00    psu
0.00    racg
0.00    relnotes
0.00    root.sh
0.00    rootupgrade.sh
0.00    scheduler
0.00    slax
0.00    sqlplus
0.00    sysman
0.00    utl
0.00    wwg
0.00    xdk
0.01    crs
0.01    evm
0.01    ord
0.01    srvm
0.01    usm
0.02    jdbc
0.03    OPatch
0.03    ccr
0.03    jlib
0.03    md
0.03    nls
0.03    oui
0.03    p6880880_112000_AIX64-5L.zip
0.04    oracore
0.05    install
0.06    cdata
0.06    perl
0.08    jdk
0.09    cv
0.10    javavm
0.10    network
0.11    rdbms
0.18    oc4j
0.32    crf
0.69    inventory
1.07    lib
2.14    log
7.84    bin

The mountpoint Grid_11203 is showing 47.3 GB but when we browse inside this it just shows sum of al files size close to 12 GB

How can I troubleshoot further?

Best regards,
Vishal

There are two common possibilities.

1. Several very large files are held open by processes, that called unlink() on the file after it was opened. They keep the file open. This is an old UNIX security trick. The data and inode persist so the process can use the file; but the directory entry does not persist; so df sees the used space, du does not see it.

2. The mountpoint was a directory with large files in it. You can mount a filesystem on top of a directory. The directory is not lost it is just buried alive so to speak. The directory and files continue to use space and will magically reappear when the filesystem is unmounted.

Paste the output of df -g , you need to know is it a directory or filesystem.

It could be any of the reason Jim mentioned in 1st line
If it is a file system and no one is using it you can try umount/mount 'ing it.

Else for both directory or filesystem you can use the below.
fuser -cu <path> , it will report any open files and also (-u) flag will list the user.
You can kill it using -k flag.

"released" implies you deleted things and expect their space back. If they were open files, they will remain on disk until whatever was using them lets them close.

/dev/lv_oragrid     74.00     20.38   73%    24223     1% /oragrid_01

 fuser -cu /oragrid_01
/oragrid_01:  17825950c(oracle) 19660928c(oracle) 19988698c(oracle) 24641596c(oracle) 26279982c(oracle) 28442660c(oracle)
root@atldc-oragrid-ux01:/> ps -ef|grep 17825950c
root@atldc-oragrid-ux01:/> ps -ef|grep 17825950
  oracle 17825950 14024750   0 13:06:34  pts/2  0:00 -ksh
  oracle 26279982 17825950   0 13:06:34  pts/3  0:00 -ksh
root@atldc-oragrid-ux01:/> ps -ef|grep 19660928
  oracle 19660928 21364754   0 13:54:37  pts/4  0:00 -ksh
  oracle 19988698 19660928   0 13:54:37  pts/5  0:00 -ksh
root@atldc-oragrid-ux01:/> ps -ef|grep 19988698
  oracle 19988698 19660928   0 13:54:37  pts/5  0:00 -ksh
f0078563 26083538 19988698   0 14:10:18  pts/5  0:00 -ksh
root@atldc-oragrid-ux01:/> ps -ef|Grep  26279982
ksh: Grep:  not found.
root@atldc-oragrid-ux01:/> ps -ef|grep 26279982
  oracle 26279982 17825950   0 13:06:34  pts/3  0:00 -ksh
root@atldc-oragrid-ux01:/> ps -ef|grep  28442660
    root 26673332  5898392   0 14:19:23  pts/5  0:00 grep 28442660
  oracle 28442660 24641596   0 13:05:27  pts/1  0:00 -ksh

 umount /oragrid_01
umount: 0506-349 Cannot unmount /dev/lv_oragrid: The requested resource is busy.

Best regards,
Vishal

Ok,
We were talking about "Grid_11203", now we know its a directory, so run
fuser -cu /oragrid_01/Grid_11203 , now do you see any user have open files in here.

You cannot unmount that file system as oracle is using it, to do that you have to shutdown the grid.

But, you have to be sure that you can bring it down without causing havoc in the environment.

The grid is already down for maintenance activity.Now,all the fuser sessions that I can see are just the session logins pls see the output in

ps -ef|grep process_id

These are just the sessions which are logged in once and while unmount problem is that the session through which I am connecting is also coming in fuser command so how can I kill that and if I kill that how would I issue unmount command.

Best regards,
Vishal

Ok,
Close the other sessions if they are not required or least change the directory (say, go to default home directory of oracle). Nevertheless if you want to kill all the other sessions look below for fuser command
fuser -cuk /oragrid_01/Grid_11203 , this will kill all the user sessions and open files used by user in the directory.

Make sure you run the above command from a different directory (as root and not as oracle).
Run df -g , if you still see the space occupied unmount/mount the filesystem it should resolve the problem.

The * in the du arguments won't show hidden files and directories, add .* to the arguments.
At least there will be the OPatch hidden storage directory.
Edit: I've just read that the application processes are down for maintenance.

 du -sg .*
52.89   .
53.58   ..
39.64   .patch_storage

Does this mean that .patch_storage occupies 39 GB and does this increase every time when we attempt applying the patch even in an unsuccessful attempt?

Yes, it does.
This is due to a bug on AIX. Check the following MOS note
FAQ: OPatch/Patch Questions/Issues for Oracle Clusterware (Grid Infrastructure or CRS) and RAC Environments (Doc ID 1339140.1)
search for: OPatch reports: "Prerequisite check CheckSystemSpace failed".

Thanks!

This doc helped me in resolving the issue

Best regards,
Vishal