CentOs server generating several alarms on partition /proc/

Greetings,

I have been facing one issue here and I am not understanding what is causing.
I am getting a lot of alarms complaining that the partition for example /proc/12345 , is full, but the partition does not existe on the server. Which is not supposed to happen, because /proc is not a real FileSystem.

I checked the server and when I checked the disk space :

[root@server ~]# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/vg_umpom01brb-LogVol01
                      260G   23G  224G  10% /
tmpfs                 7.9G   26M  7.9G   1% /dev/shm

Which is not supposed to happen because there are other mounted partition which should have been show there.

So I ran :

[root@server ~]# mount -l
/dev/mapper/vg_umpom01brb-LogVol01 on / type ext4 (rw)
proc on /proc type proc (rw)
sysfs on /sys type sysfs (rw)
devpts on /dev/pts type devpts (rw,gid=5,mode=620)
tmpfs on /dev/shm type tmpfs (rw)
sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw)
/usr/share/myspell/shell/.myempty on /proc/25669 type none (rw,bind)
/dev/sda1 on /boot type ext4 (rw)
10.000.000.00:/umplogs on /var/UMP_LOGS type nfs (rw,nosuid,addr=10.000.000.00)
10.000.000.00:/DBBACKUP on /var/DBBACKUP type nfs (rw,nosuid,addr=10.000.000.00)
10.000.000.00:/opt/sysreports on /opt/sysreports type nfs (rw,nosuid,vers=4,addr=10.000.000.00,clientaddr=10.000.000.00)

And them was okay :

[root@umpom01brb ~]# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/vg_umpom01brb-LogVol01
                      260G   23G  224G  10% /
tmpfs                 7.9G   26M  7.9G   1% /dev/shm
/dev/sda1             194M   58M  127M  32% /boot
10.238.202.15:/umplogs
                       89G   32G   57G  36% /var/UMP_LOGS
10.238.202.27:/DBBACKUP
                       99G   60G   40G  60% /var/DBBACKUP
10.238.202.65:/opt/sysreports
                       15G  2.1G   13G  15% /opt/sysreports

The above is the expected, and should be like that, but after a few hours I got the alarm again, and the partition which was supposed to be mounted is not there again :

[root@server ~]# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/vg_umpom01brb-LogVol01
                      260G   23G  224G  10% /
tmpfs                 7.9G   26M  7.9G   1% /dev/shm

@@@EDIT@@@

I think I am getting close to the issue

I was looking on /proc/self/mounts and I saw a lot of weird entries :

/dev/mapper/vg_umpom01brb-LogVol01 /proc/14779 ext4 rw,relatime,barrier=1,stripe=64,data=ordered 0 0
/dev/mapper/vg_umpom01brb-LogVol01 /proc/45537 ext4 rw,relatime,barrier=1,stripe=64,data=ordered 0 0
/dev/mapper/vg_umpom01brb-LogVol01 /proc/13170 ext4 rw,relatime,barrier=1,stripe=64,data=ordered 0 0

device /dev/mapper/vg_umpom01brb-LogVol01 mounted on /proc/13142 with fstype ext4
device /dev/mapper/vg_umpom01brb-LogVol01 mounted on /proc/30085 with fstype ext4
device /dev/mapper/vg_umpom01brb-LogVol01 mounted on /proc/38674 with fstype ext4

68 16 253:0 /usr/share/myspell/shell/.myempty /proc/20718 rw,relatime - ext4 /dev/mapper/vg_umpom01brb-LogVol01 rw,barrier=1,stripe=64,data=ordered
69 16 253:0 /usr/share/myspell/shell/.myempty /proc/29767 rw,relatime - ext4 /dev/mapper/vg_umpom01brb-LogVol01 rw,barrier=1,stripe=64,data=ordered
70 16 253:0 /usr/share/myspell/shell/.myempty /proc/38871 rw,relatime - ext4 /dev/mapper/vg_umpom01brb-LogVol01 rw,barrier=1,stripe=64,data=ordered

and all entries are to this myspell thing.

There are hundreds of entries like that, and all alarms I got was listed there. So looks like this is the "source" of all alarms .

Even removing the entry below from mtab, if keeps returning :

/usr/share/myspell/shell/.myempty /proc/9127 none rw,bind 0 0

So I think that my question now is :

How I am supposed to edit this file /proc/self/mounts, if this file is created by the Kernel and this one is "read only". ?

Why do you call /proc/12345 a "partition" ?

/proc/12345 is a directory in the proc file system which maintains system information for process "12345" .....

Are you saying this directory in proc is full?

man proc :

So I'm surprised the error message calls a proc "subdir" a "partition". Please post the entire error messages, and some info about e.g. memory status, like the output of free .

I think I am getting close to the issue

I was looking on /proc/self/mounts and I saw a lot of weird entries :

/dev/mapper/vg_umpom01brb-LogVol01 /proc/14779 ext4 rw,relatime,barrier=1,stripe=64,data=ordered 0 0
/dev/mapper/vg_umpom01brb-LogVol01 /proc/45537 ext4 rw,relatime,barrier=1,stripe=64,data=ordered 0 0
/dev/mapper/vg_umpom01brb-LogVol01 /proc/13170 ext4 rw,relatime,barrier=1,stripe=64,data=ordered 0 0

device /dev/mapper/vg_umpom01brb-LogVol01 mounted on /proc/13142 with fstype ext4
device /dev/mapper/vg_umpom01brb-LogVol01 mounted on /proc/30085 with fstype ext4
device /dev/mapper/vg_umpom01brb-LogVol01 mounted on /proc/38674 with fstype ext4

68 16 253:0 /usr/share/myspell/shell/.myempty /proc/20718 rw,relatime - ext4 /dev/mapper/vg_umpom01brb-LogVol01 rw,barrier=1,stripe=64,data=ordered
69 16 253:0 /usr/share/myspell/shell/.myempty /proc/29767 rw,relatime - ext4 /dev/mapper/vg_umpom01brb-LogVol01 rw,barrier=1,stripe=64,data=ordered
70 16 253:0 /usr/share/myspell/shell/.myempty /proc/38871 rw,relatime - ext4 /dev/mapper/vg_umpom01brb-LogVol01 rw,barrier=1,stripe=64,data=ordered

and all entries are to this myspell thing.

There are hundreds of entries like that, and all alarms I got was listed there. So looks like this is the "source" of all alarms .

So I think that my question now is :

How I am supposed to edit this file /proc/self/mounts, if this file is created by the Kernel and this one is "read only". ?

The application is mapping information in the proc filesystem to the application so the application can use interact with the information in the proc filesystem.

In your original post, you said:

You should post the exact error message or alarm.

/proc/12345 maps to a process.

If that process is not available because the process is dead or is throwing an error, then the system will be in an error state, of course.

However, if you do not actually post the exact error message, cut-and-paste in full, you are not really providing the necessary information to get expert help.

Thanks for replying back.

By alarm I mean the zabbix trigger which monitore the server. Like the last I got was this :

disk space is less than 20% on volume /proc/56371 

This /proc/56371, does not exist on the server, but is mapped on /proc/self/mounts.
I did not find any information on any other log. So I think that this will not help.

This could be besides the point, but I checked the other server of the cluster and the configuration are similar, except the :

/usr/share/myspell/shell/.myempty /proc/56371 rw,relatime - ext4 /dev/mapper/vg_umpom01brb-LogVol01 rw,barrier=1,stripe=64,data=ordered

And the other hundred of entries are different, just exist on this server.

Any suggestion in where in I might found a detailed error log or something that could help is much appreciate.

Thanks again, for the help.

So why is your Zabbix configuration monitoring disk space on the proc filesystem?

If I were you, I would immediately go into the Zabbix configuration and disable monitoring for the proc file system.

You should not be monitoring /proc with the same Zabbix tools used to monitor a normal filesystem. When you stop monitoring the /proc filesystem like it is a regular filesystem, those alarms with go away.

Thanks for replying !

Zabbix, monitores all mounted partition, for that reason it is generating alarms like that. Because somehow those /proc partition are been mounted and looks like that myspell thing is trying to write on /proc;Or is alarming because is not possible to write there.

Since I find the issue I suspend this from the monitoring.

The other guys are planning to reinstall the OS, and in order to avoid this I was trying to find a way to remove those entries from /proc/self/mounts or a way to prevent this.

Thanks again !

I don't think the information above is correct and so you should look deeper into it.

In Zabbix you can configure which file systems to monitor in the zabbix_agentd.conf file.

If you search the net, you will find many examples of zabbix_agentd.conf to research.

Why do you want to change mtab or mounts? Don't! It would confuse the system.
Maybe your change attempts cause the strange bind mounts?
Anyway, update your zabbix! A bind mount (regardless if strange or normal) must not be monitored at all!

Note that this might be caused by an automounter. Most of them mount only when the FS is accessed and therefore you don't see it in df s output when it was not accessed for some time. I had that (rather strange) effect once on a Solaris system.

I hope this helps.

bakunin