Filesystem / (root) run ouf of space

Linux and Unix-Like Linux
1

Good evening, I've got a question, in our production system there is an application called Intermediate which ftp service is the core to to send back and forth from/to diferent destinations

Gradually FS / was ruuning out space and we took a long time to figure out what precesses were eating up this FS /

our SA found a folder called /dcs/data13 taking up 5.4 GB of space located 0n FS / (root)

to checku up /dcs/data13 is not a FS but a folder mounted in FS /,

I've got a couple of questions:

1 why i cant see the path when execute the command df -h
2 why the application let me create /dcs/data13 asuming it was a path? it shoud have yield an error if filesytem /dcs/data13 didnt exist
3 is true if FS / get 100 % full is probably the server shut itself down ?

I appreciate your help in advanved, here is the scrren of FS mounted

df -h
Filesystem                              Size  Used Avail Use% Mounted on
/dev/sdfm2                              9.9G  8.9G  555M  95% /
tmpfs                                    48G   12K   48G   1% /dev/shm
/dev/sdfm1                              241M   36M  193M  16% /boot
/dev/mapper/vg_promeddb01-lv_home       8.9G  7.1G  1.4G  84% /home
/dev/mapper/vg_promeddb01-lv_opt        5.0G  1.2G  3.6G  25% /opt/dcs
/dev/mapper/vg_promeddb01-lv_opt_dcs    9.9G  6.5G  2.9G  70% /opt/dcs_7.0
/dev/mapper/vg_promeddb01-lv_tmp         20G   14G  5.7G  70% /tmp
/dev/mapper/vg_promeddb01-lv_usr        5.0G  1.9G  2.9G  40% /usr
/dev/mapper/vg_promeddb01-lv_var        6.0G  2.0G  3.7G  36% /var
/dev/mapper/vg_promeddb01-lv_var_dcs    9.9G  291M  9.1G   4% /var/dcs_7.0
/dev/mapper/vg_promeddb01-lv_usr_openv  9.9G  6.7G  2.8G  72% /usr/openv
tmpfs                                   4.0K     0  4.0K   0% /dev/vx
/dev/vx/dsk/vgreport/report              20G  7.4G   13G  38% /reportes
/dev/vx/dsk/data05_dg/data05_vol       1000G  734G  250G  75% /dcs/data05
/dev/vx/dsk/data04_dg/data04_vol        999G  767G  219G  78% /dcs/data04
/dev/vx/dsk/appl01_dg/appl01_vol        499G  265G  220G  55% /dcs/appl01
/dev/vx/dsk/data03_dg/data03_vol       1000G  832G  159G  84% /dcs/data03
/dev/vx/dsk/data01_dg/data01_vol       1000G  764G  222G  78% 
/dev/vx/dsk/data10_dg/data10_vol       1000G  772G  215G  79% /dcs/data10
2

du shows paths

 find / -type d | xargs du | sort -n | tail -20

This will run a while, but will show you a path for each directory, it chops off the list with the 20 largest directories. By largest I mean how much space the directory uses, this includes subdirectories. So with one subdirectory using lots of space each of the parent directories of the biggie will also show up in the list. Hence look for more than just one, I guessed 20, so one of them will likely be the bad guy directory itself.

If you are having problems with processes silently using space, consider implementing quotas on file systems you need to protect. Also, consider NOT having "ordinary applications" access to writable directories on the / FS. Put the directories somewhere else.

3

You must restrict find to the file system

find / -xdev ...

Or restrict du

cd /
du -kx

---------- Post updated at 08:23 AM ---------- Previous update was at 08:14 AM ----------

A folder mounted on /dcs/data13 is another file system.
The df shows all currently mounted file systems. I suspect that /dcs/data13 is not permanently mounted via fstab, but on-demand via automounter.

4

Thanks a lot for your help but i face this error with this FS.

find /dcs/data08 -type d | xargs du | sort -n | tail -10

This is the error:
xargs: xargs.c:450: main: Assertion `bc_ctl.arg_max <= (131072-2048)' failed.

this is the OS

uname -a
Linux promedcon02 2.6.32-431.el6.x86_64 #1 SMP Sun Nov 10 22:19:54 EST 2013 x86_64 GNU/Linux]

Thanks for your help in advanced

5

1 - df will show up only mounted FS...

2 -

Why??? If it were a file system /dcs/data13 would be a mount point no? So it needs to exist for the FS can be mounted, yet if the FS is not mounted then its just a directory and what is stopping you to write in it?...

6

Good morning, this command works in hp-ux but in linux do not work:

[find /dcs/data08 -type d | xargs du | sort -n | tail -10][/CODE]

This is the error:
xargs: xargs.c:450: main: Assertion `bc_ctl.arg_max <= (131072-2048)' failed.

dont kno w if is the version or there is anothe optioin replacing xargs ?

Thanks in advanced

7

I think but do not know for sure: you exceeded some limit, likely the number of arguments

  • in other words du is finding zillions of files. du -s gives a summary
    This will work in that case:
find /dcs/data08 | -type d |
while read dname
do
  du -s $dname
done | sort -n
8

Thanks for your help, i run those commands with a slight difference but i hat to cancel it since it was taking more than 4 hours and it never finished.

find /dcs/data08 -type d |
  while read dname
  do
    du -s $dname
  done | sort -n

Furthermore it was in the top list using the top command

is there a faster way to do the same ?

Thanks for your help in advanced

9

I would guess that you want something more like:

du -s /dcs/data/08/*/ | sort -rn

It will still be high on the list of output from top while it is running, but at least it will just be processing the directories located in /dcs/data/08 instead of every directory in the file hierarchy rooted in /dcs/data/08 . And, it will be invoking du once instead of once for each directory in that file hierarchy.

1 Like
10

Yes, du is recursive, and in combination with the recursive find there are lots of repeated du runs.
Give other processes priority over du:

nice du -s /dcs/data/08/*/ | sort -n
1 Like