Production unexpectedly server rebooted

Linux and Unix-Like Red Hat
1

I am trying to figure out what might causing Production server unexpectedly reboot during last few months ..

Is auto reboot is set , I can check it is not set during the kernel panic but are they any other parameters which I am missing .

-bash-2.05b$ uname -a
Linux PD1011 2.4.21-53.ELhugemem #1 SMP Wed Nov 14 03:46:17 EST 2007 i686 i686 i386 GNU/Linux
-bash-2.05b$ cat /proc/version
Linux version 2.4.21-53.ELhugemem (brewbuilder@hs20-bc1-7.build.redhat.com) (gcc version 3.2.3 20030502 (Red Hat Linux 3.2.3-58)) #1 SMP Wed Nov 14 03:46:17 EST 2007
-bash-2.05b$

-bash-2.05b$ cat /etc/redhat-release
Red Hat Enterprise Linux AS release 3 (Taroon Update 9)

/var/log/messages

messages.4:Dec 19 04:02:02 rc1011 syslogd 1.4.1: restart.
messages.3:Dec 26 04:02:03 rc1011 syslogd 1.4.1: restart.
messages.2:Jan 2 04:02:02 rc1011 syslogd 1.4.1: restart.
messages.1:Jan 9 04:02:03 rc1011 syslogd 1.4.1: restart.
messages.1:Jan 10 03:03:14 rc1011 syslogd 1.4.1: restart.
messages.1:Jan 10 09:40:23 rc1011 syslogd 1.4.1: restart.
messages.1:Jan 12 09:49:45 rc1011 syslogd 1.4.1: restart.
messages.1:Jan 13 05:15:44 rc1011 syslogd 1.4.1: restart.
messages.1:Jan 13 21:48:33 rc1011 syslogd 1.4.1: restart.
messages:Jan 16 04:02:03 rc1011 syslogd 1.4.1: restart.

Detailed Previos Logs :

1st unexpected reboot of 2011
Jan 10 02:35:06 PD1011 cfengine:PD1011[14910]: Packages installed: /usr/bin/yum -y install collectd nrpe blip-nagios-plugins-nrpe
Jan 10 02:36:29 PD1011 gpagentd[3386]: SELinux group policy is not applicable for this system...
Jan 10 02:50:02 PD1011 cfengine:PD1011[22545]: Installing package(s) using '/usr/bin/yum -y install collectd nrpe blip-nagios-plugins-nrpe'
Jan 10 02:50:06 PD1011 cfengine:PD1011[22545]: collectd is installed and is the latest version.
Jan 10 02:50:06 PD1011 cfengine:PD1011[22545]: nrpe is installed and is the latest version.
Jan 10 02:50:06 PD1011 cfengine:PD1011[22545]: blip-nagios-plugins-nrpe is installed and is the latest version.
Jan 10 02:50:06 PD1011 cfengine:PD1011[22545]: Gathering header information file(s) from server(s)
Jan 10 02:50:06 PD1011 cfengine:PD1011[22545]: Server: RHEL 3U8 AS
Jan 10 02:50:06 PD1011 cfengine:PD1011[22545]: Server: RHEL 3U8 AS - latest
Jan 10 02:50:06 PD1011 cfengine:PD1011[22545]: Server: GID
Jan 10 02:50:06 PD1011 cfengine:PD1011[22545]: Server: GID - latest
Jan 10 02:50:06 PD1011 cfengine:PD1011[22545]: Finding updated packages
Jan 10 02:50:06 PD1011 cfengine:PD1011[22545]: Downloading needed headers
Jan 10 02:50:06 PD1011 cfengine:PD1011[22545]: No actions to take
Jan 10 02:50:06 PD1011 cfengine:PD1011[22545]: Packages installed: /usr/bin/yum -y install collectd nrpe blip-nagios-plugins-nrpe
Jan 10 03:03:14 PD1011 syslogd 1.4.1: restart.
Jan 10 03:03:14 PD1011 syslog: syslogd startup succeeded
Jan 10 03:03:14 PD1011 kernel: klogd 1.4.1, log source = /proc/kmsg started.
Jan 10 03:03:14 PD1011 syslog: klogd startup succeeded
Jan 10 03:03:14 PD1011 kernel: Linux version 2.4.21-53.ELhugemem (brewbuilder@hs20-bc1-7.build.redhat.com) (gcc version 3.2.3 20030502 (Red Hat Linux 3.2.3-58)) #1 SMP Wed Nov 14
03:46:17 EST 2007
Jan 10 03:03:14 PD1011 kernel: BIOS-provided physical RAM map:
Jan 10 03:03:14 PD1011 kernel: BIOS-e820: 0000000000000000 - 000000000009fc00 (usable)
Jan 10 03:03:14 PD1011 kernel: BIOS-e820: 000000000009fc00 - 00000000000a0000 (reserved)
Jan 10 03:03:14 PD1011 kernel: BIOS-e820: 00000000000e0000 - 0000000000100000 (reserved)
Jan 10 03:03:14 PD1011 kernel: BIOS-e820: 0000000000100000 - 00000000dfff0000 (usable)
Jan 10 03:03:14 PD1011 kernel: BIOS-e820: 00000000dfff0000 - 00000000dffff000 (ACPI data)
Jan 10 03:03:14 PD1011 kernel: BIOS-e820: 00000000dffff000 - 00000000e0000000 (ACPI NVS)
Jan 10 03:03:14 PD1011 kernel: BIOS-e820: 00000000fec00000 - 00000000fec03000 (reserved)
Jan 10 03:03:14 PD1011 kernel: BIOS-e820: 00000000fee00000 - 00000000fee01000 (reserved)
Jan 10 03:03:14 PD1011 kernel: BIOS-e820: 00000000ffe00000 - 0000000100000000 (reserved)
Jan 10 03:03:14 PD1011 kernel: BIOS-e820: 0000000100000000 - 0000000220000000 (usable)
Jan 10 03:03:14 PD1011 kernel: 4768MB HIGHMEM available.
Jan 10 03:03:14 PD1011 kernel: 3936MB LOWMEM available.
Jan 10 03:03:14 PD1011 kernel: found SMP MP-table at 000ff780

2nd unexpected reboot of 2011
Jan 10 09:20:06 PD1011 cfengine:PD1011[5754]: Packages installed: /usr/bin/yum -y install collectd nrpe blip-nagios-plugins-nrpe
Jan 10 09:33:57 PD1011 gpagentd[3386]: SELinux group policy is not applicable for this system...
Jan 10 09:35:02 PD1011 cfengine:PD1011[13545]: Installing package(s) using '/usr/bin/yum -y install collectd nrpe blip-nagios-plugins-nrpe'
Jan 10 09:35:06 PD1011 cfengine:PD1011[13545]: collectd is installed and is the latest version.
Jan 10 09:35:06 PD1011 cfengine:PD1011[13545]: nrpe is installed and is the latest version.
Jan 10 09:35:06 PD1011 cfengine:PD1011[13545]: blip-nagios-plugins-nrpe is installed and is the latest version.
Jan 10 09:35:06 PD1011 cfengine:PD1011[13545]: Gathering header information file(s) from server(s)
Jan 10 09:35:06 PD1011 cfengine:PD1011[13545]: Server: RHEL 3U8 AS
Jan 10 09:35:06 PD1011 cfengine:PD1011[13545]: Server: RHEL 3U8 AS - latest
Jan 10 09:35:06 PD1011 cfengine:PD1011[13545]: Server: GID
Jan 10 09:35:06 PD1011 cfengine:PD1011[13545]: Server: GID - latest
Jan 10 09:35:06 PD1011 cfengine:PD1011[13545]: Finding updated packages
Jan 10 09:35:06 PD1011 cfengine:PD1011[13545]: Downloading needed headers
Jan 10 09:35:06 PD1011 cfengine:PD1011[13545]: No actions to take
Jan 10 09:35:06 PD1011 cfengine:PD1011[13545]: Packages installed: /usr/bin/yum -y install collectd nrpe blip-nagios-plugins-nrpe
Jan 10 09:40:23 PD1011 syslogd 1.4.1: restart.
Jan 10 09:40:23 PD1011 syslog: syslogd startup succeeded
Jan 10 09:40:23 PD1011 kernel: klogd 1.4.1, log source = /proc/kmsg started.
Jan 10 09:40:23 PD1011 kernel: Linux version 2.4.21-53.ELhugemem (brewbuilder@hs20-bc1-7.build.redhat.com) (gcc version 3.2.3 20030502 (Red Hat Linux 3.2.3-58)) #1 SMP Wed Nov 14
03:46:17 EST 2007
Jan 10 09:40:23 PD1011 kernel: BIOS-provided physical RAM map:
Jan 10 09:40:23 PD1011 kernel: BIOS-e820: 0000000000000000 - 000000000009fc00 (usable)
Jan 10 09:40:23 PD1011 kernel: BIOS-e820: 000000000009fc00 - 00000000000a0000 (reserved)
Jan 10 09:40:23 PD1011 kernel: BIOS-e820: 00000000000e0000 - 0000000000100000 (reserved)
Jan 10 09:40:23 PD1011 kernel: BIOS-e820: 0000000000100000 - 00000000dfff0000 (usable)
Jan 10 09:40:23 PD1011 kernel: BIOS-e820: 00000000dfff0000 - 00000000dffff000 (ACPI data)
Jan 10 09:40:23 PD1011 kernel: BIOS-e820: 00000000dffff000 - 00000000e0000000 (ACPI NVS)
Jan 10 09:40:23 PD1011 kernel: BIOS-e820: 00000000fec00000 - 00000000fec03000 (reserved)
Jan 10 09:40:23 PD1011 kernel: BIOS-e820: 00000000fee00000 - 00000000fee01000 (reserved)
Jan 10 09:40:23 PD1011 kernel: BIOS-e820: 00000000ffe00000 - 0000000100000000 (reserved)
Jan 10 09:40:23 PD1011 kernel: BIOS-e820: 0000000100000000 - 0000000220000000 (usable)
Jan 10 09:40:23 PD1011 kernel: 4768MB HIGHMEM available.
Jan 10 09:40:23 PD1011 kernel: 3936MB LOWMEM available.
Jan 10 09:40:23 PD1011 kernel: found SMP MP-table at 000ff780

2

Might be problem of RAM. Reinsert the RAM and check.

3

I have checked the memory usage which is the same from last 1 year when i have taken the statistics through Wily monitor ..

---------- Post updated 01-19-11 at 07:03 AM ---------- Previous update was 01-18-11 at 07:29 PM ----------

Can some one please reply on this thread as unexpected server reboots are hampering the business & this is into the production

4

I see the line :

Packages installed: /usr/bin/yum -y install collectd nrpe blip-nagios-plugins-nrpe

before each server reboot, did you check your Nagios installation ?

5

Why is the nagios package being reinstalled?

Jan 10 02:35:06 PD1011 cfengine:PD1011[14910]: Packages installed: /usr/bin/yum -y install collectd nrpe blip-nagios-plugins-nrpe
....
....
Jan 10 09:20:06 PD1011 cfengine:PD1011[5754]: Packages installed: /usr/bin/yum -y install collectd nrpe blip-nagios-plugins-nrpe