Huge difference in reported Disk usage between ls,df and du

UNIX AIX
1

IBM RS6000 F50
AIX 4.3.2

i am having trouble in calculating the actual size of a set of directories and reconciling the results with the actual Hard Disk space used

I have 33GB disk which is showing 7.8GB used, a byte count of the files in the directory/sub-dirs i`m interested in is 48GB, can anyone help as why this is showing as using more than is available?

Directory names have been anonymised, all file_nnnn_b[1-7] directories are copies of the original file_nnnn directory

Disk Detail:

df -k
/dev/xxxxlv    35192832  26916564   24%    12624     1% /xxxx

df -k | grep xxxx | awk '{print $2/1024/1024" "$3/1024/1024" "($2-$3)/1024/1024}'
33.5625    25.6696    7.89286
Total        Free         Used

Directory Detail:

from within directory /xxxx/yyyyy/zzzz/dir_726

du -sk .                 
965996

=943.355MB

ls -lrt | awk '{sum+=$5} END {print sum}'
981122905

=935.672MB

a confirmation check using fileplace gives approximately the same results

find /xxxx/yyyyy/zzzz/dir_726 -exec fileplace {} \; > dir_726.txt

total bytes from File Size for all files in the directory

grep vol dir_726.txt  | awk '{print $4}' | awk '{sum+=$1} END {print sum}'
981122905

=935.672 MB

Block count

total Allocated Bytes from fragments for all files in the directory

grep " frags" dir_726.txt | awk '{print $4}' | awk '{sum+=$1} END {print sum}'
989065216

=943.246 MB

Sparse Files

total Unallocated Bytes from fragments for all files in the directory

grep "unall" dir_726.txt | awk '{print $4}' | awk '{sum+=$1} END {print sum/1024/1024}'
0

for the remaining directories in /xxxx/yyyyy/zzzz/ the block count and Byte count are 48.655 GB and 48.167 GB respectively

Block count from du -sk       byte count from ls -lrt
991888         968.641MB     1007788171       961.102MB      dir_w_a
994276         970.973MB     1010214976       963.416MB      dir_w_b
35832           34.992MB       32909460        31.385MB      dir_29
36060           35.215MB       33146538        31.611MB      dir_30_3
937896         915.914MB      952405185       908.284MB      dir_w_c
937924         915.941MB      952412457       908.291MB      dir_w_d
938732         916.730MB      953242845       909.083MB      dir_w_e
36060           35.215MB       33146538        31.611MB      dir_30_2
937908         915.926MB      952406254       908.285MB      dir_w_f
36064           35.219MB       33148544        31.613MB      dir_32
953264         930.922MB      968126743       923.278MB      dir_698
36072           35.227MB       33151697        31.616MB      dir_32_1
36076           35.230MB       33154443        31.619MB      dir_4_a
953312         930.969MB      968137370       923.288MB      dir_4_b
963348         940.770MB      978444807       933.118MB      dir_707
963388         940.809MB      978448237       933.121MB      dir_4_c
940020         917.988MB      954525668       910.307MB      dir_697_1
36196           35.348MB       33270107        31.729MB      dir_w_f
953588         931.238MB      968417226       923.555MB      dir_4_d
964692         942.082MB      979772050       934.383MB      dir_710_1
36196           35.348MB       33270194        31.729MB      dir_33_1
35980           35.137MB       33037306        31.507MB      dir_3_a
964052         941.457MB      979123434       933.765MB      dir_5_a
965308         942.684MB      980411309       934.993MB      dir_5_b
954600         932.227MB      969452487       924.542MB      dir_698_1
937856         915.875MB      952365462       908.246MB      dir_692
939356         917.340MB      953848243       909.661MB      dir_697
939204         917.191MB      953727423       909.545MB      dir_695
965332         942.707MB      980447541       935.028MB      dir_714
965360         942.734MB      980442130       935.022MB      dir_720
36200           35.352MB       33273315        31.732MB      dir_33_2
965364         942.738MB      980444282       935.025MB      dir_720_1
965972         943.332MB      981107322       935.657MB      dir_720_2
36200           35.352MB       33273315        31.732MB      dir_33_3
36204           35.355MB       33275473        31.734MB      dir_4
965996         943.355MB      981113508       935.663MB      dir_722
965988         943.348MB      981118308       935.667MB      dir_724
966016         943.375MB      981118564       935.668MB      dir_w_g
965988         943.348MB      981118315       935.667MB      dir_w_h
36204           35.355MB       33275473        31.734MB      dir_33_4
966024         943.383MB      981122842       935.672MB      dir_z
36204           35.355MB       33275473        31.734MB      dir_3
965996         943.355MB      981122905       935.672MB      dir_726
36204           35.355MB       33275473        31.734MB      dir_4_b7
36204           35.355MB       33275473        31.734MB      dir_3_b7
36204           35.355MB       33275473        31.734MB      dir_4_b6
36204           35.355MB       33275473        31.734MB      dir_3_b6
36204           35.355MB       33275473        31.734MB      dir_4_b5
36204           35.355MB       33275473        31.734MB      dir_3_b5
36204           35.355MB       33275473        31.734MB      dir_4_b4
36204           35.355MB       33275473        31.734MB      dir_3_b4
36204           35.355MB       33275473        31.734MB      dir_4_b3
36204           35.355MB       33275473        31.734MB      dir_3_b3
36204           35.355MB       33275473        31.734MB      dir_4_b2
36204           35.355MB       33275473        31.734MB      dir_3_b2
966012         943.371MB      981116713       935.666MB      dir_1
965984         943.344MB      981116734       935.666MB      dir_2_b7
965984         943.344MB      981116734       935.666MB      dir_2_b6
965984         943.344MB      981116734       935.666MB      dir_2_b5
965984         943.344MB      981116734       935.666MB      dir_2_b4
965984         943.344MB      981116734       935.666MB      dir_2_b3
965984         943.344MB      981116734       935.666MB      dir_2_b2
965984         943.344MB      981116734       935.666MB      dir_2_b1
965984         943.344MB      981116734       935.666MB      dir_1_b7
965984         943.344MB      981116734       935.666MB      dir_1_b6
965984         943.344MB      981116734       935.666MB      dir_1_b5
965984         943.344MB      981116734       935.666MB      dir_1_b4
965984         943.344MB      981116734       935.666MB      dir_1_b3
965984         943.344MB      981116734       935.666MB      dir_1_b2
965984         943.344MB      981116734       935.666MB      dir_1_b1
36204           35.355MB       33275473        31.734MB      dir_4_b1
965992         943.352MB      981122863       935.672MB      dir_b7
965992         943.352MB      981122863       935.672MB      dir_b6
965992         943.352MB      981122863       935.672MB      dir_b5
965992         943.352MB      981122863       935.672MB      dir_b4
965992         943.352MB      981122863       935.672MB      dir_b3
965992         943.352MB      981122863       935.672MB      dir_b2
965992         943.352MB      981122863       935.672MB      dir_b1
36204           35.355MB       33275473        31.734MB      dir_3_b1
965988         943.348MB      981116713       935.666MB      dir_2

               48.655 GB                      48.167 GB

Any ideas how i can get an realistic byte usage?

Many Thanks

cooperuf

2

Definitely no other filesystems mounted under that mount point?

Are there any symlinks which could be causing you to count some files more than once?

3

There are no files under the mount point

There are no symbolic links in any of the directories.

4

What kind of filesystem are they on? Try mount -v | grep /xxxx.

Are you certain some kind of point-in-time-snapshot mechanism is not being used here rather than a complete copy of the dir_n to dir_n_bN directories?

What do you get from du -ks /xxxx?

Are there any hard-linked files on the filesystem? Try find /xxxx -type f -links +1 | xargs ls -lid.

5

The filesystems are all JFS.

du -ks /xxxx returns
7152560 /xxxx/

where as find /xxxx -type f -links +1 | xargs ls -lid lists between 20 and 52 files for 5000 uniq filenames, i guess that this is where my problem is, as a byte count of the 224,000 listed files is 45.97GB.

A few calculations to come up with a true byte count for the uniq names in that list, and i should be good.

Many thanks for you suggestion

Cooper