I have this AIX machine (see boot process below). The problem was it hangs at:
The ctrmc Subsystem has been started. Subsystem PID is 11906..
I checked with other forums but their advise were just "strong punches in the air" with no specific explanation.
I really appreciate if you can help me boot this up directly to the user prompt.
Looking forward for your reply.
AIXXIA
-------------------------------------------------------------------------------
Welcome to AIX.
boot image timestamp: 14:44 10/18
The current time and date: 16:33:53 10/27/2007
number of processors: 2 size of memory: 2048MB
boot device: /pci@400000000110/pci@2,4/scsi@1/sd@8:2
kernel size: 12101874; 32 bit kernel
-------------------------------------------------------------------------------
Saving Base Customize Data to boot disk
Starting the sync daemon
Starting the error daemon
System initialization completed.
Setting tunable parameters...
Sat Oct 27 11:35:05 CDT 2007
Automatic Error Log Analysis for sysplanar0 has detected a problem.
The Service Request Number is
651-880: The CEC or SPCN reported an error. Report the SRN and the following
reference and physical location codes to your service provider..
complete
Starting Multi-user Initialization
Performing auto-varyon of Volume Groups
Activating all paging spaces
0517-075 swapon: Paging device /dev/hd6 is already active.
/dev/rhd1 (/home): ** Unmounted cleanly - Check suppressed
/dev/rhd10opt (/opt): ** Unmounted cleanly - Check suppressed
Performing all automatic mounts
Multi-user initialization completed
Checking for srcmstr active...complete
Starting tcpip daemons:
0513-059 The syslogd Subsystem has been started. Subsystem PID is 5996.
0513-059 The sendmail Subsystem has been started. Subsystem PID is 6496.
0513-059 The portmap Subsystem has been started. Subsystem PID is 8596.
0513-059 The inetd Subsystem has been started. Subsystem PID is 8280.
0513-059 The hostmibd Subsystem has been started. Subsystem PID is 7752.
0513-059 The snmpmibd Subsystem has been started. Subsystem PID is 10332.
0513-059 The snmpd Subsystem has been started. Subsystem PID is 10858.
0513-059 The aixmibd Subsystem has been started. Subsystem PID is 6274.
Finished starting tcpip daemons.
0513-059 The muxatmd Subsystem has been started. Subsystem PID is 13420.
0513-059
The ctrmc Subsystem has been started. Subsystem PID is 11906.
What do you mean by "hang"? Does the machine crash? freeze completely? freeze only for a longer time? become unresponsive? slows down?
What does the errpt say?
The ctrmc subsystem is one of the leftovers PSSP, the middleware of the (*sigh*, those were the times) SP/2. It will be necessary for CSM, HACMP or if the machine is connected to a HMC. If this is not the case you could remove it by "rmcctrl" but consult the manpages of rmcctrl first!
If you want to configure the RMC it is done by the command csmconfig, but read this link (Understanding RMC and resource managers) first. There is a configuration file (/var/ct/cfg/ctrmc.acls) you could have look at and a command (chrmcacl) to modify that file.
Hang :
> the logic of the boot process stops at "The ctrmc Subsystem has been started. Subsystem PID is xxxxxx.." The behaviour is UNRESPONSIVE even i wait for 30mins.
I dont know how to gather this.
It is possible to access the ">" prompt once the system becomes non-responsive such like in solaris. if it is: what it the kestroke to it (solaris - stop + A )
Please be informed that I am new to AIX, kindly consider my unfamiliarity with the system.
I read some AIX related online docs and advise to proceed to single user mode and finally i got the errpt.
# errpt
IDENTIFIER TIMESTAMP T C RESOURCE_NAME DESCRIPTION
BE0A03E5 0811092871 P H sysplanar0 ENVIRONMENTAL PROBLEM
BE0A03E5 0811092671 P H sysplanar0 ENVIRONMENTAL PROBLEM
BFE4C025 0811092371 P H sysplanar0 UNDETERMINED ERROR
9DBCFDEE 0811092471 T O errdemon ERROR LOGGING TURNED ON
BE0A03E5 0809084471 P H sysplanar0 ENVIRONMENTAL PROBLEM
67145A39 0809084271 U S SYSDUMP SYSTEM DUMP
BE0A03E5 0809084171 P H sysplanar0 ENVIRONMENTAL PROBLEM
F48137AC 0809084171 U O minidump COMPRESSED MINIMAL DUMP
1104AA28 0809084171 T S SYSPROC SYSTEM RESET INTERRUPT RECEIVED
9DBCFDEE 0809084371 T O errdemon ERROR LOGGING TURNED ON
BA431EB7 0809083271 P S SRC SOFTWARE PROGRAM ERROR
BA431EB7 0809083171 P S SRC SOFTWARE PROGRAM ERROR
A6DF45AA 0809083071 I O RMCdaemon The daemon is started.
BE0A03E5 0809082171 P H sysplanar0 ENVIRONMENTAL PROBLEM
2BFA76F6 0809082171 T S SYSPROC SYSTEM SHUTDOWN BY USER
9DBCFDEE 0809082271 T O errdemon ERROR LOGGING TURNED ON
192AC071 0808161371 T O errdemon ERROR LOGGING TURNED OFF
BE0A03E5 0808140271 P H sysplanar0 ENVIRONMENTAL PROBLEM
BFE4C025 0808135971 P H sysplanar0 UNDETERMINED ERROR
BA431EB7 0808135971 P S SRC SOFTWARE PROGRAM ERROR
9DBCFDEE 0808140171 T O errdemon ERROR LOGGING TURNED ON
BA431EB7 0807162371 P S SRC SOFTWARE PROGRAM ERROR
BA431EB7 0807162271 P S SRC SOFTWARE PROGRAM ERROR
BA431EB7 0807162171 P S SRC SOFTWARE PROGRAM ERROR
BA431EB7 0807162071 P S SRC SOFTWARE PROGRAM ERROR
BA431EB7 0807162071 P S SRC SOFTWARE PROGRAM ERROR
BA431EB7 0807161971 P S SRC SOFTWARE PROGRAM ERROR
BA431EB7 0807161871 P S SRC SOFTWARE PROGRAM ERROR
BA431EB7 0807161771 P S SRC SOFTWARE PROGRAM ERROR
BA431EB7 0807161671 P S SRC SOFTWARE PROGRAM ERROR
BA431EB7 0807161571 P S SRC SOFTWARE PROGRAM ERROR
BA431EB7 0807161471 P S SRC SOFTWARE PROGRAM ERROR
BA431EB7 0807161371 P S SRC SOFTWARE PROGRAM ERROR
BA431EB7 0807161271 P S SRC SOFTWARE PROGRAM ERROR
BA431EB7 0807161171 P S SRC SOFTWARE PROGRAM ERROR
BA431EB7 0807161071 P S SRC SOFTWARE PROGRAM ERROR
BA431EB7 0807160971 P S SRC SOFTWARE PROGRAM ERROR
BA431EB7 0807160971 P S SRC SOFTWARE PROGRAM ERROR
BA431EB7 0807160871 P S SRC SOFTWARE PROGRAM ERROR
BA431EB7 0807160771 P S SRC SOFTWARE PROGRAM ERROR
BA431EB7 0807160671 P S SRC SOFTWARE PROGRAM ERROR
BA431EB7 0807160571 P S SRC SOFTWARE PROGRAM ERROR
BA431EB7 0807160471 P S SRC SOFTWARE PROGRAM ERROR
BA431EB7 0807160371 P S SRC SOFTWARE PROGRAM ERROR
BA431EB7 0807160271 P S SRC SOFTWARE PROGRAM ERROR
BA431EB7 0807160171 P S SRC SOFTWARE PROGRAM ERROR
BA431EB7 0807160071 P S SRC SOFTWARE PROGRAM ERROR
BA431EB7 0807155971 P S SRC SOFTWARE PROGRAM ERROR
BA431EB7 0807155871 P S SRC SOFTWARE PROGRAM ERROR
BA431EB7 0807155871 P S SRC SOFTWARE PROGRAM ERROR
BA431EB7 0807155771 P S SRC SOFTWARE PROGRAM ERROR
BA431EB7 0807155671 P S SRC SOFTWARE PROGRAM ERROR
BA431EB7 0807155571 P S SRC SOFTWARE PROGRAM ERROR
BA431EB7 0807155471 P S SRC SOFTWARE PROGRAM ERROR
BA431EB7 0807155371 P S SRC SOFTWARE PROGRAM ERROR
BA431EB7 0807155271 P S SRC SOFTWARE PROGRAM ERROR
BA431EB7 0807155171 P S SRC SOFTWARE PROGRAM ERROR
BA431EB7 0807155071 P S SRC SOFTWARE PROGRAM ERROR
BA431EB7 0807154971 P S SRC SOFTWARE PROGRAM ERROR
BA431EB7 0807154871 P S SRC SOFTWARE PROGRAM ERROR
BA431EB7 0807154771 P S SRC SOFTWARE PROGRAM ERROR
BA431EB7 0807154771 P S SRC SOFTWARE PROGRAM ERROR
BA431EB7 0807154671 P S SRC SOFTWARE PROGRAM ERROR
BA431EB7 0807154471 P S SRC SOFTWARE PROGRAM ERROR
A6DF45AA 0807154471 I O RMCdaemon The daemon is started.
BE0A03E5 0807153571 P H sysplanar0 ENVIRONMENTAL PROBLEM
2BFA76F6 0807153571 T S SYSPROC SYSTEM SHUTDOWN BY USER
9DBCFDEE 0807153671 T O errdemon ERROR LOGGING TURNED ON
192AC071 0807153071 T O errdemon ERROR LOGGING TURNED OFF
BFE4C025 0807152671 P H sysplanar0 UNDETERMINED ERROR
BE0A03E5 0807152471 P H sysplanar0 ENVIRONMENTAL PROBLEM
BE0A03E5 0807152171 P H sysplanar0 ENVIRONMENTAL PROBLEM
E18E984F 0807152171 P S SRC SOFTWARE PROGRAM ERROR
9DBCFDEE 0807152371 T O errdemon ERROR LOGGING TURNED ON
E18E984F 0807151371 P S SRC SOFTWARE PROGRAM ERROR
BA431EB7 0807151371 P S SRC SOFTWARE PROGRAM ERROR
A6DF45AA 0807151371 I O RMCdaemon The daemon is started.
BFE4C025 0807151171 P H sysplanar0 UNDETERMINED ERROR
1BA7DF4E 0807151171 P S SRC SOFTWARE PROGRAM ERROR
9DBCFDEE 0807151271 T O errdemon ERROR LOGGING TURNED ON
1BA7DF4E 0807135771 P S SRC SOFTWARE PROGRAM ERROR
BA431EB7 0807135771 P S SRC SOFTWARE PROGRAM ERROR
BA431EB7 0807135771 P S SRC SOFTWARE PROGRAM ERROR
A6DF45AA 0807135771 I O RMCdaemon The daemon is started.
BE0A03E5 0807135571 P H sysplanar0 ENVIRONMENTAL PROBLEM
2BFA76F6 0807135571 T S SYSPROC SYSTEM SHUTDOWN BY USER
9DBCFDEE 0807135671 T O errdemon ERROR LOGGING TURNED ON
192AC071 0807135071 T O errdemon ERROR LOGGING TURNED OFF
67145A39 0807125271 U S SYSDUMP SYSTEM DUMP
BE0A03E5 0807125071 P H sysplanar0 ENVIRONMENTAL PROBLEM
BFE4C025 0807124871 P H sysplanar0 UNDETERMINED ERROR
BFE4C025 0807124671 P H sysplanar0 UNDETERMINED ERROR
BE0A03E5 0807124471 P H sysplanar0 ENVIRONMENTAL PROBLEM
A6D1BD62 0807124171 I H Firmware Event
F48137AC 0807124171 U O minidump COMPRESSED MINIMAL DUMP
1104AA28 0807124171 T S SYSPROC SYSTEM RESET INTERRUPT RECEIVED
9DBCFDEE 0807125271 T O errdemon ERROR LOGGING TURNED ON
BA431EB7 0807121771 P S SRC SOFTWARE PROGRAM ERROR
BA431EB7 0807121671 P S SRC SOFTWARE PROGRAM ERROR
BA431EB7 0807121571 P S SRC SOFTWARE PROGRAM ERROR
A6DF45AA 0807121471 I O RMCdaemon The daemon is started.
BE0A03E5 0807120671 P H sysplanar0 ENVIRONMENTAL PROBLEM
2BFA76F6 0807120671 T S SYSPROC SYSTEM SHUTDOWN BY USER
9DBCFDEE 0807120771 T O errdemon ERROR LOGGING TURNED ON
192AC071 0807120071 T O errdemon ERROR LOGGING TURNED OFF
67145A39 0807115171 U S SYSDUMP SYSTEM DUMP
BE0A03E5 0807114871 P H sysplanar0 ENVIRONMENTAL PROBLEM
F48137AC 0807114871 U O minidump COMPRESSED MINIMAL DUMP
1104AA28 0807114871 T S SYSPROC SYSTEM RESET INTERRUPT RECEIVED
9DBCFDEE 0807115171 T O errdemon ERROR LOGGING TURNED ON
BE0A03E5 0807113271 P H sysplanar0 ENVIRONMENTAL PROBLEM
67145A39 0807112971 U S SYSDUMP SYSTEM DUMP
BE0A03E5 0807112971 P H sysplanar0 ENVIRONMENTAL PROBLEM
F48137AC 0807112971 U O minidump COMPRESSED MINIMAL DUMP
1104AA28 0807112971 T S SYSPROC SYSTEM RESET INTERRUPT RECEIVED
9DBCFDEE 0807113071 T O errdemon ERROR LOGGING TURNED ON
BFE4C025 0807112171 P H sysplanar0 UNDETERMINED ERROR
9DBCFDEE 0807112271 T O errdemon ERROR LOGGING TURNED ON
BE0A03E5 0807111271 P H sysplanar0 ENVIRONMENTAL PROBLEM
2BFA76F6 0807111271 T S SYSPROC SYSTEM SHUTDOWN BY USER
9DBCFDEE 0807111371 T O errdemon ERROR LOGGING TURNED ON
192AC071 0807110771 T O errdemon ERROR LOGGING TURNED OFF
67145A39 0807100071 U S SYSDUMP SYSTEM DUMP
BE0A03E5 0807095971 P H sysplanar0 ENVIRONMENTAL PROBLEM
F48137AC 0807095971 U O minidump COMPRESSED MINIMAL DUMP
1104AA28 0807095971 T S SYSPROC SYSTEM RESET INTERRUPT RECEIVED
9DBCFDEE 0807100071 T O errdemon ERROR LOGGING TURNED ON
BE0A03E5 0807094271 P H sysplanar0 ENVIRONMENTAL PROBLEM
BFE4C025 0807093971 P H sysplanar0 UNDETERMINED ERROR
9DBCFDEE 0807094071 T O errdemon ERROR LOGGING TURNED ON
BE0A03E5 0806162871 P H sysplanar0 ENVIRONMENTAL PROBLEM
2BFA76F6 0806162871 T S SYSPROC SYSTEM SHUTDOWN BY USER
9DBCFDEE 0806162971 T O errdemon ERROR LOGGING TURNED ON
192AC071 0806162371 T O errdemon ERROR LOGGING TURNED OFF
67145A39 0806155171 U S SYSDUMP SYSTEM DUMP
BE0A03E5 0806155071 P H sysplanar0 ENVIRONMENTAL PROBLEM
F48137AC 0806155071 U O minidump COMPRESSED MINIMAL DUMP
1104AA28 0806155071 T S SYSPROC SYSTEM RESET INTERRUPT RECEIVED
9DBCFDEE 0806155271 T O errdemon ERROR LOGGING TURNED ON
BE0A03E5 0806153071 P H sysplanar0 ENVIRONMENTAL PROBLEM
A6DF45AA 0806152871 I O RMCdaemon The daemon is started.
67145A39 0806152771 U S SYSDUMP SYSTEM DUMP
BFE4C025 0806152771 P H sysplanar0 UNDETERMINED ERROR
F48137AC 0806152771 U O minidump COMPRESSED MINIMAL DUMP
1104AA28 0806152771 T S SYSPROC SYSTEM RESET INTERRUPT RECEIVED
9DBCFDEE 0806152871 T O errdemon ERROR LOGGING TURNED ON
BE0A03E5 0806151971 P H sysplanar0 ENVIRONMENTAL PROBLEM
A6DF45AA 0806151771 I O RMCdaemon The daemon is started.
BFE4C025 0806151671 P H sysplanar0 UNDETERMINED ERROR
A6DF45AA 0806151671 I O RMCdaemon The daemon is started.
9DBCFDEE 0806151771 T O errdemon ERROR LOGGING TURNED ON
A6DF45AA 0806141571 I O RMCdaemon The daemon is started.
67145A39 0806141471 U S SYSDUMP SYSTEM DUMP
BE0A03E5 0806141471 P H sysplanar0 ENVIRONMENTAL PROBLEM
F48137AC 0806141471 U O minidump COMPRESSED MINIMAL DUMP
1104AA28 0806141471 T S SYSPROC SYSTEM RESET INTERRUPT RECEIVED
9DBCFDEE 0806141571 T O errdemon ERROR LOGGING TURNED ON
BE0A03E5 0806135471 P H sysplanar0 ENVIRONMENTAL PROBLEM
A6DF45AA 0806135271 I O RMCdaemon The daemon is started.
BFE4C025 0806135171 P H sysplanar0 UNDETERMINED ERROR
A6DF45AA 0806135171 I O RMCdaemon The daemon is started.
9DBCFDEE 0806135271 T O errdemon ERROR LOGGING TURNED ON
A6DF45AA 0806122571 I O RMCdaemon The daemon is started.
BE0A03E5 0806122371 P H sysplanar0 ENVIRONMENTAL PROBLEM
2BFA76F6 0806122371 T S SYSPROC SYSTEM SHUTDOWN BY USER
9DBCFDEE 0806122471 T O errdemon ERROR LOGGING TURNED ON
A6DF45AA 0731175571 I O RMCdaemon The daemon is started.
CF8CADB6 0731175371 P H sysplanar0 SYSTEM FAILURE WITH SCAN DATA
BE0A03E5 0731175271 P H sysplanar0 ENVIRONMENTAL PROBLEM
2BFA76F6 0731175271 T S SYSPROC SYSTEM SHUTDOWN BY USER
9DBCFDEE 0731175371 T O errdemon ERROR LOGGING TURNED ON
#
I don't know for sure but this seems like it has nothing to do with the ctrmc subsystem. This seems to be the culprit:
IDENTIFIER TIMESTAMP T C RESOURCE_NAME DESCRIPTION
BE0A03E5 0811092871 P H sysplanar0 ENVIRONMENTAL PROBLEM
BE0A03E5 0811092671 P H sysplanar0 ENVIRONMENTAL PROBLEM
BFE4C025 0811092371 P H sysplanar0 UNDETERMINED ERROR
9DBCFDEE 0811092471 T O errdemon ERROR LOGGING TURNED ON
BE0A03E5 0809084471 P H sysplanar0 ENVIRONMENTAL PROBLEM
67145A39 0809084271 U S SYSDUMP SYSTEM DUMP
BE0A03E5 0809084171 P H sysplanar0 ENVIRONMENTAL PROBLEM
Boot again to single user mode, and issue "errpt -a | more" to see the details. Each of these entries is only the headline to a problem, "errpt -a" will give you the full text.
Without wanting to rush to conclusions this looks more lika problem suited for the technician than one suited for system administrators. "sysplanar0" is the system planar, something like the "motherboard" in PCs. This thing failing is usually pretty bad news from the hardware.
We are having the same problem-we have replaced the sysplaner board(s) and the issue is still happening..
os level is 4.3
are you using SAN attached storage?
