CentOS 6 ran out of space, need to reclaim it

Hello everyone,

I am having an issue here with CentOS release 6.6 (Final) that shows all of the space used up, but I can't tell where the space went.

Seemingly I am using up 100%, according to

df -h

Filesystem                     Size  Used Avail Use% Mounted on
/dev/mapper/vg_sugar-lv_root   50G   49G     0 100% /

So that's 49G out of 50G, fair enough, but when I try to see what exactly is being used (x to remove mounts):

du -xsh /*

1.8G    /usr
2.0G    /var

That's all, these are the 2 biggest directories, all others combined are under 1G.

Here is info on Disk /dev/mapper/vg_sugar-lv_root , it does show close to 50G (this is a virtual machine):

fdisk -l

Disk /dev/mapper/vg_sugar-lv_root: 53.7 GB, 53687091200 bytes
255 heads, 63 sectors/track, 6527 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000

Node count is at 7%:

df -i

Filesystem                   Inodes  IUsed     IFree   IUse% Mounted on
/dev/mapper/vg_sugar-lv_root 3276800 200552    3076248    7% /

The system was rebooted several times, so any log files should've been deleted. I'd appreciate some tips in resolving this.

find / -xdev -type f -size +100M

returned only a single file. The problem is not in amount of large files, but that the space is hidden or used for something else, or not released.

 du -ahx / | sort -rh | head -20 
5.0G / 
2.0G /var 
1.8G /usr
1.1G /var/lib 
945M /var/lib/mysql 
909M /var/lib/mysql/sugarcrm 
759M /var/www 
745M /var/www/html 
707M /root 
672M /usr/lib64 
642M /var/www/html/sugar 
611M /usr/share 
480M /var/lib/mysql/sugarcrm/emails_text.MYD 
441M /lib 
392M /lib/modules 

228M /usr/lib

Rebooting does not automatically delete log-files on normally configured Linux systems., FYI.

Refine your search for smaller files, recently updated:

find / -xdev -type f -size +1000 -mtime -60

Search for deleted files:

lsof / | grep "(deleted)"

> Rebooting does not automatically delete log-files on normally configured Linux systems., FYI.

What does, then?

--- Post updated at 06:47 PM ---

What steps should I take to delete log files or reclaim the space, then?

--- Post updated at 07:02 PM ---

lsof / | grep "(deleted)"

returned no entries

lsof | grep -i del

shows file on /dev/zero only

find / -xdev -type f -size +1000 -mtime -60

returned some entries

-rw-r--r--  1 root   root    11M Aug 23 02:03 sugarcrm_1.log
-rw-r--r--  1 root   root    11M Aug 19 12:38 sugarcrm_2.log
-rw-r--r--  1 root   root    11M Aug 16 02:03 sugarcrm_3.log
-rw-r--r--  1 root   root    11M Aug 12 18:04 sugarcrm_4.log
-rw-r--r--  1 root   root    11M Aug 11 05:05 sugarcrm_5.log
-rw-r--r--  1 root   root    11M Aug  9 22:13 sugarcrm_6.log
-rw-r--r--  1 root   root    11M Aug  8 12:22 sugarcrm_7.log
-rw-r--r--  1 root   root    11M Aug  7 05:35 sugarcrm_8.log
-rw-r--r--  1 root   root    11M Aug  5 10:19 sugarcrm_9.log

I deleted those files, freed up 100 Mbs, but the MySQL is not starting, reporting
190826 11:00:55 InnoDB: Started; log sequence number 0 8747645
190826 11:00:55 [ERROR] /usr/libexec/mysqld: Error writing file '/var/run/mysqld/mysqld.pid' (Errcode: 28)
190826 11:00:55 [ERROR] Can't start server: can't create PID file: No space left on device

Ensure with

ls /var/run/mysqld
df /var/run/mysqld

that it is empty and on the / filesystem.
A command to find the biggest directories in the / file system:

find / -xdev -mindepth 2 -maxdepth 2 -exec du -skx {} + | sort -k1,1n

The following has no precise space count but ensures to not enter other filesystems:

find / -depth -xdev -ls | awk '{ cnt+=int($7/1024); if (split ($11,A,"/")==3) { print cnt, $0; cnt=0 } }' | sort -k1,1n

I made sure that ls /var/run/mysqld is empty, and df /var/run/mysqld is on /

Filesystem           1K-blocks     Used Available Use% Mounted on
/dev/mapper/vg_sugar-lv_root
                      51475068 50220120         0 100% /

find / -xdev -mindepth 2 -maxdepth 2 -exec du -skx {} + | sort -k1,1n
51164   /root/sugarcrm_full-2014-01-29T01-30-01.sql
77020   /root/sugarcrm_sqldump_2014_07_08.sql
80240   /root/SugarCE-Full-6.5.4
86672   /root/sugar_backup
90268   /var/log
152124  /root/sugar_6_4_backup
163828  /usr/bin
206900  /root/sugar_07_10_2013
232744  /usr/lib
400408  /lib/modules
625660  /usr/share
684664  /var/www
687556  /usr/lib64
1062204 /var/lib
13812205161     /mnt/backup

Is it normal for /mnt/backup to show? I thought -x in xdev would exclude external file systems?

df /mnt
Filesystem           1K-blocks     Used Available Use% Mounted on
/dev/mapper/vg_sugar-lv_root
                      51475068 50220300         0 100% /

find / -depth -xdev -ls | awk '{ cnt+=int($7/1024); if (split ($11,A,"/")==3) { print cnt, $0; cnt=0 } }' | sort -k1,1n

I've attached find.txt to this message with the output.

DannyBoyCentOS,

You will benefit from thinking about what you are saying about log files.

Neo: Rebooting does not automatically delete log-files on normally configured Linux systems., FYI.

DannyBoyCentOS:  What does, then?

Any system which would automatically delete log files on reboot would be a seriously misconfigured system.

Log files are the single most important tool any system administrator has to deal with configuration errors, security breaches, system errors, access control issues, database errors, and more.

Without logging "all is lost".... as one might say, so no normally configured Linux system automatically deletes a single log file during the boot process. When you post like this, an experienced Linux user like me (over two decades of Linux system admin) can only conclude you have very little experience on a server. So let me explain to you again.

Log files should never be automatically deleted on reboot because that means anytime the system reboots, all that logging information would be lost; and a server can reboot for a variety of reasons. Log files are generally rotated and compressed and saved by the system; and then they are generally archived for a certain period of time, and then eventually deleted manually. The could be deleted automatically after some set period of time, but that should normally be a few weeks out (actually it depends on the size of storage, system criticality, system usages and other site specific factors).

In your output, the big files I see are (mysql) dump files and backups. These are not log files, they are backups and dumps. Normally these files can get very big very quickly and need to be actively managed. I manage these on my servers by moving them to an archive site (generally another servers for that specific server); but how you choose to do that is entirely up to you.

Based on what I have seen in your output so far (which admittedly I did not spend too much time on it), the big files you have are .sql and .zip files. If you need disk space, move these to another disk, another server, the cloud or where ever you archive your large files, dumps and backups; and then decide what to keep and what to delete.

Cheers.

Please note, from your output attachment:

51162 931363 51164 -rw-r--r--   1 root     root     52389946 Jan 29  2014 /root/sugarcrm_full-2014-01-29T01-30-01.sql
77012 931389 77020 -rw-r--r--   1 root     root     78860965 Jul  8  2014 /root/sugarcrm_sqldump_2014_07_08.sql

Why do you have huge 5 year old database dump files on your server?

These are from 2014, but this is 2019 brother.

Based on this observation, it is logical to conclude that your system is not properly administered if your server is having disk space problems and you have huge database dumps and other backup file on the same server / disks.

If you want to "reclaim disk space" find all these huge database dump files and backup files and move them off this server; or to a "data archive disk" with space for these types of files.

Cheers and Regards,

Neo

Thank you for your suggestions, based on which I've removed 320 Mb worth of .sql files to /mnt/backup from /root :
this is size of my root now:
408688 /root

I really thought that would allow MySQL to start, but no, I am still getting

190826 23:08:42  InnoDB: Started; log sequence number 0 8747645
190826 23:08:42 [ERROR] /usr/libexec/mysqld: Error writing file '/var/run/mysqld/mysqld.pid' (Errcode: 28)
190826 23:08:42 [ERROR] Can't start server: can't create PID file: No space left on device

, even after truncating existing 50Mb MySQL log file:
truncate -s 0 /var/log/mysqld.log

I think I misunderstood your question. I thought it was referring to automatically releasing space after deleting log files, but apparently you meant just active log files. Absolutely, the log files are crucial.
And yes - nobody is managing this server - I am a C#/SQL software developer, trying to get the company's CRM Linux server back online.

Is /mnt/backup an actual mountpoint or just a directory under root filesystem.
This could be a source of your troubles.

Can you post output of df -h

Regards
Peasant.

df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/vg_sugar-lv_root
                       50G   48G     0 100% /
tmpfs                 1.9G     0  1.9G   0% /dev/shm
/dev/vda1             477M  102M  350M  23% /boot
/dev/mapper/vg_sugar-lv_home
                      9.3G   26M  8.8G   1% /home
ddhq01:/backup        1.2T  878G  280G  76% /mnt/backup

I think it is an actual mount point. No matter what files I delete, there is still not enough space. The latest suggestion I received was to truncate 480M /var/lib/mysql/sugarcrm/emails_text.MYD
However, I have deleted about 500Mb today, and that is still not enough to get the MySQL running:

InnoDB: buffer...
190826 23:40:41  InnoDB: Started; log sequence number 0 8747645
190826 23:40:41 [ERROR] /usr/libexec/mysqld: Error writing file '/var/run/mysqld/mysqld.pid' (Errcode: 28)
190826 23:40:41 [ERROR] Can't start server: can't create PID file: No space left on device

Please post your mysql.conf file(s)...

My first guess is that your mysql db is running and saving a binary log used to recover from a crash.

The binary log file will grow to an incredibly large size and cannot just be truncated.

The best approach is to look at the mysql config file first and confirm what mysql is doing.

Then, we can move the entire mysql directory to your home partition where there is plenty of space. You should not have the mysql data directory in the root partition anyway but my guess is that it is there.

Then after moving the entire mysql data dir to the home partition we edit the mysql config file to tell it where all the mysql data is.

Then we should be able to start mysql

Can you try umount of /mnt/backup and check if files are present under that directory after umount ?

I suspect the mount happend after stuff is writen under /mnt/backup directory.

Hope that helps.
Regards
Peasant.

Example:

myserver:/etc/mysql# cat mysql.cnf
#
# The MySQL database server configuration file.
#
# You can copy this to one of:
# - "/etc/mysql/my.cnf" to set global options,
# - "~/.my.cnf" to set user-specific options.
# 
# One can use all long options that the program supports.
# Run program with --help to get a list of available options and with
# --print-defaults to see which it would actually understand and use.
#
# For explanations see
# http://dev.mysql.com/doc/mysql/en/server-system-variables.html

#
# * IMPORTANT: Additional settings that can override those from this file!
#   The files must end with '.cnf', otherwise they'll be ignored.
#

!includedir /etc/mysql/conf.d/
!includedir /etc/mysql/mysql.conf.d/

myserver:/etc/mysql# cd mysql.conf.d/
myserver:/etc/mysql/mysql.conf.d# l
total 36
-rw-r--r-- 1 root root 4991 Jul 19 20:24 mysqld.cnf
-rw-r--r-- 1 root root 3453 Jun 14 23:32 mysqld.cnf.back
-rw-r--r-- 1 root root 4555 Jun 21 23:42 mysqld.cnf.back2

myserver:/etc/mysql/mysql.conf.d# cat mysqld.conf

myserver:/etc/mysql/mysql.conf.d# cat mysqld.cnf
#
# The MySQL database server configuration file.
#
# You can copy this to one of:
# - "/etc/mysql/my.cnf" to set global options,
# - "~/.my.cnf" to set user-specific options.
# 
# One can use all long options that the program supports.
# Run program with --help to get a list of available options and with
# --print-defaults to see which it would actually understand and use.
#
# For explanations see
# http://dev.mysql.com/doc/mysql/en/server-system-variables.html

# This will be passed to all mysql clients
# It has been reported that passwords should be enclosed with ticks/quotes
# escpecially if they contain "#" chars...
# Remember to edit /etc/mysql/debian.cnf when changing the socket location.

# Here is entries for some specific programs
# The following values assume you have at least 32M ram

[mysqld_safe]
socket		= /var/run/mysqld/mysqld.sock
nice		= 0

[mysqld]
#
# * Basic Settings
#
user		= mysql
pid-file	= /var/run/mysqld/mysqld.pid
socket		= /var/run/mysqld/mysqld.sock
port		= 3306
basedir		= /usr
datadir		= /var/lib/mysql
tmpdir		= /tmp
lc-messages-dir	= /usr/share/mysql
skip-external-locking
log_error_verbosity = 3
ft_min_word_len=3
open_files_limit = 65535
#
# Instead of skip-networking the default is now to listen only on
# localhost which is more compatible and is not less secure.
bind-address		= 127.0.0.1
#
# * Fine Tuning
#

#
# OMITTED REMAINDER OF THIS FILE FROM THIS CUT AND PASTE EXAMPLE 
#

You really care about the entry in red above.

I suggest you post your entire mysql.cnf file (in CODE tags, not ICODE tags) ... so we can best help you

Example:

myserver:/etc/mysql# cat mysql.cnf
#
# The MySQL database server configuration file.
#
# You can copy this to one of:
# - "/etc/mysql/my.cnf" to set global options,
# - "~/.my.cnf" to set user-specific options.
# 
# One can use all long options that the program supports.
# Run program with --help to get a list of available options and with
# --print-defaults to see which it would actually understand and use.
#
# For explanations see
# http://dev.mysql.com/doc/mysql/en/server-system-variables.html

#
# * IMPORTANT: Additional settings that can override those from this file!
#   The files must end with '.cnf', otherwise they'll be ignored.
#

!includedir /etc/mysql/conf.d/
!includedir /etc/mysql/mysql.conf.d/

myserver:/etc/mysql# cd mysql.conf.d/
myserver:/etc/mysql/mysql.conf.d# l
total 36
-rw-r--r-- 1 root root 4991 Jul 19 20:24 mysqld.cnf
-rw-r--r-- 1 root root 3453 Jun 14 23:32 mysqld.cnf.back
-rw-r--r-- 1 root root 4555 Jun 21 23:42 mysqld.cnf.back2

myserver:/etc/mysql/mysql.conf.d# cat mysqld.conf

myserver:/etc/mysql/mysql.conf.d# cat mysqld.cnf
#
# The MySQL database server configuration file.
#
# You can copy this to one of:
# - "/etc/mysql/my.cnf" to set global options,
# - "~/.my.cnf" to set user-specific options.
# 
# One can use all long options that the program supports.
# Run program with --help to get a list of available options and with
# --print-defaults to see which it would actually understand and use.
#
# For explanations see
# http://dev.mysql.com/doc/mysql/en/server-system-variables.html

# This will be passed to all mysql clients
# It has been reported that passwords should be enclosed with ticks/quotes
# escpecially if they contain "#" chars...
# Remember to edit /etc/mysql/debian.cnf when changing the socket location.

# Here is entries for some specific programs
# The following values assume you have at least 32M ram

[mysqld_safe]
socket		= /var/run/mysqld/mysqld.sock
nice		= 0

[mysqld]
#
# * Basic Settings
#
user		= mysql
pid-file	= /var/run/mysqld/mysqld.pid
socket		= /var/run/mysqld/mysqld.sock
port		= 3306
basedir		= /usr
datadir		= /var/lib/mysql
tmpdir		= /tmp
lc-messages-dir	= /usr/share/mysql
skip-external-locking
log_error_verbosity = 3
ft_min_word_len=3
open_files_limit = 65535
#
# Instead of skip-networking the default is now to listen only on
# localhost which is more compatible and is not less secure.
bind-address		= 127.0.0.1
#
# * Fine Tuning
#


# OMITTED FROM CUT AND PASTE EXAMPLE

You really care about the entry in red above.

Best you post your entire mysql.cnf file for us to review. Then, I can better tell you exactly what is going on and how to fix your problem

In parallel let me help you to look for the files that fill your / file system.
Your post#11 explains some things. (In your post#1 you said that "df -h" gave output for "/" only. That was misleading.)
Can you shortly umount the /mnt/backup and look for files? Such files are invisible when mounted.

umount /mnt/backup
ls -lah /mnt/backup
du -sh /mnt/backup
mount /mnt/backup

(Just seeing that Peasant has the same idea in post#13.)

Following up on my request to post your mysql.cnf file.... take a look at this example:

just_a_mysql_server:/var/lib/mysql# ls -l
total 254144
-rw-r----- 1 mysql mysql       56 Sep  5  2018 auto.cnf
-rw-r----- 1 mysql mysql     4134 Jul 19 20:24 ib_buffer_pool
-rw-r----- 1 mysql mysql 79691776 Aug 27 04:26 ibdata1
-rw-r----- 1 mysql mysql 50331648 Aug 27 04:26 ib_logfile0
-rw-r----- 1 mysql mysql 50331648 Jun 18 03:07 ib_logfile1
-rw-r----- 1 mysql mysql 79691776 Aug 27 04:29 ibtmp1

Note the HUGE binary log files above:

But before we can fully analyze and create a solution for you (which I think is very easy to do, BTW), we need to see your mysql.cnf file

Thank you for taking your time helping me.

here is my.cnf, it is shorter than your example, but I guess the initial set up was to leave a lot as a default:

less /etc/my.cnf
[mysqld]
datadir=/var/lib/mysql
socket=/var/lib/mysql/mysql.sock
user=mysql
port=3306
bind-address=0.0.0.0
# Disabling symbolic-links is recommended to prevent assorted security risks
symbolic-links=0

[mysqld_safe]
log-error=/var/log/mysqld.log
pid-file=/var/run/mysqld/mysqld.pid

/var/lib/mysql didn't have any huge files:

ls -lh
total 29M
-rw-rw---- 1 mysql mysql  18M Aug 21 03:15 ibdata1
-rw-rw---- 1 mysql mysql 5.0M Aug 27 07:21 ib_logfile0
-rw-rw---- 1 mysql mysql 5.0M Aug 27 07:21 ib_logfile1
drwx------ 2 mysql mysql 4.0K Oct 31  2014 mysql
srwxrwxrwx 1 mysql mysql    0 Aug 27 07:21 mysql.sock
drwx------ 2 mysql mysql  20K Oct 31  2014 SUGARBAK
drwx------ 2 mysql mysql  20K Jul 24  2018 sugarcrm

tail /var/log/mysqld.log
190827  7:35:26  InnoDB: Database was not shut down normally!
InnoDB: Starting crash recovery.
InnoDB: Reading tablespace information from the .ibd files...
InnoDB: Restoring possible half-written data pages from the doublewrite
InnoDB: buffer...
190827  7:35:26  InnoDB: Started; log sequence number 0 8747645
190827  7:35:26 [ERROR] /usr/libexec/mysqld: Error writing file '/var/run/mysqld/mysqld.pid' (Errcode: 28)
190827  7:35:26 [ERROR] Can't start server: can't create PID file: No space left on device
190827 07:35:26 mysqld_safe Number of processes running now: 0
190827 07:35:26 mysqld_safe mysqld restarted

Do you have a directory as follows:

/etc/mysql

?

I am guessing.... no ... correct?

i suggest you create a directory:

/home/var/lib/mysql

and copy your entire /var/lib/mysql dir to the new directory and then make sure your mysql process can write to the new dir.

Then edit your my.cnf file and change the data dir to your new dir in /home (for now just for testing) and start mysqld .

If you do this correctly mysqld should start OK.