Dear Cluster Guru,
I had installed sc 3.2 on 2 box sunfire 240 \(solaris 10 update 7 ,each node 2 gb ram\) and use scsi juke box as share storage .
i had test cluster application :
1. apache HA
2 oracleHA \(oracle 9i\)
all running well (switch over between node working and oracle running good).
Here my step setup the cluster app :
ORACLE 11G installation :
-mount share storage as global device on node1 (as oracle_base and oracle_home)
-Install oracle
-test startup database use sql command from node1 and node2 (status working , database mount normal)
-change hotnname on listener file with logical hostname (oracleha) (/global/oracle10/11/product/11.1.0/db_1/network/admin/listener.ora)
Cluster APP setup :
# clresourcegroup create -n node1,node2 oracle11-rg
# clreslogicalhostname create -g oracle11-rg -h oracleha oracleha-rs
# clresource create -g oracle11-rg -t SUNW.HAStoragePlus -p FilesystemMountPoints=/global/oracle10 -p GlobalDevicePaths=oracle10 global_oracle11-rs
# clresourcegroup online -M oracle11-rg
# clresource create -g oracle11-rg -t SUNW.oracle_server -p Connect_string=SYSTEM/manager -p ORACLE_SID=ora11a -p ORACLE_HOME=/global/oracle10/11/product/11.1.0/db_1/ -p Alert_log_file=/global/oracle10/11/product/11.1.0/db_1/log/alert_ora11a.log -p Restart_type=RESOURCE_GROUP_RESTART -p Resource_dependencies=global_oracle11-rs ora11g-ora
# clresource create -g oracle11-rg -t SUNW.oracle_listener -p LISTENER_NAME=listener -p ORACLE_HOME=/global/oracle10/11/product/11.1.0/db_1 -p Resource_dependencies=global_oracle11-rs ora11g-lsr
When I try online the oracle11-rg and get problem :
clresourcegroup: (C748634) Resource group oracle11-rg failed to start on chosen node and might fail over to other node(s)
clresourcegroup: (C135343) No primary node could be found for resource group oracle11-rg; it remains offline
Resource Group Status
# cluster show
--- Resources for Group oracle11-rg ---
Resource: oracleha-rs
Type: SUNW.LogicalHostname:2
Type_version: 2
Group: oracle11-rg
R_description:
Resource_project_name: default
Enabled{node1}: True
Enabled{node2}: True
Monitored{node1}: True
Monitored{node2}: True
Resource: global_oracle11-rs
Type: SUNW.HAStoragePlus:4
Type_version: 4
Group: oracle11-rg
R_description:
Resource_project_name: default
Enabled{node1}: True
Enabled{node2}: True
Monitored{node1}: True
Monitored{node2}: True
Resource: ora11g-lsr
Type: SUNW.oracle_listener:5
Type_version: 5
Group: oracle11-rg
R_description:
Resource_project_name: default
Enabled{node1}: True
Enabled{node2}: True
Monitored{node1}: True
Monitored{node2}: True
Resource: ora11g-ora
Type: SUNW.oracle_server:6
Type_version: 6
Group: oracle11-rg
R_description:
Resource_project_name: default
Enabled{node1}: True
Enabled{node2}: True
Monitored{node1}: True
Monitored{node2}: True
=== DID Device Instances ===
DID Device Name: /dev/did/rdsk/d1
Full Device Path: node2:/dev/rdsk/c1t0d0
Replication: none
default_fencing: global
DID Device Name: /dev/did/rdsk/d2
Full Device Path: node2:/dev/rdsk/c3t2d0
Full Device Path: node1:/dev/rdsk/c3t2d0
Replication: none
default_fencing: global
DID Device Name: /dev/did/rdsk/d3
Full Device Path: node2:/dev/rdsk/c3t5d0
Full Device Path: node1:/dev/rdsk/c3t5d0
Replication: none
default_fencing: global
DID Device Name: /dev/did/rdsk/d4
Full Device Path: node1:/dev/rdsk/c0t0d0
Replication: none
default_fencing: global
DID Device Name: /dev/did/rdsk/d5
Full Device Path: node1:/dev/rdsk/c1t0d0
Replication: none
default_fencing: global
DID Device Name: /dev/did/rdsk/d6
Full Device Path: node2:/dev/rdsk/c0t0d0
Replication: none
default_fencing: global
=== NAS Devices ===
I try disable Resource: ora11g-ora and on oracle11-rg (the status online ,and can switch between node 1 to node2 )
from the oracle trace :
cat 20 /global/oracle10/11/diag/rdbms/ora11a/ora11a/trace/alert_ora11a.log
Thu Sep 10 14:42:56 2009
DIAG started with pid=4, OS id=11556
Thu Sep 10 14:42:56 2009
DBRM started with pid=5, OS id=11558
Thu Sep 10 14:42:56 2009
PSP0 started with pid=6, OS id=11560
Thu Sep 10 14:42:56 2009
DSKM started with pid=7, OS id=11562
Thu Sep 10 14:42:56 2009
DIA0 started with pid=8, OS id=11564
Thu Sep 10 14:42:56 2009
MMAN started with pid=9, OS id=11566
Thu Sep 10 14:42:56 2009
DBW0 started with pid=7, OS id=11581
Thu Sep 10 14:42:57 2009
LGWR started with pid=10, OS id=11590
Thu Sep 10 14:42:57 2009
CKPT started with pid=11, OS id=11594
Thu Sep 10 14:42:57 2009
SMON started with pid=12, OS id=11598
Thu Sep 10 14:42:57 2009
RECO started with pid=13, OS id=11604
Thu Sep 10 14:42:57 2009
MMON started with pid=14, OS id=11608
Thu Sep 10 14:42:57 2009
MMNL started with pid=15, OS id=11612
starting up 1 dispatcher(s) for network address '(ADDRESS=(PARTIAL=YES)(PROTOCOL=TCP))'...
starting up 1 shared server(s) ...
ORACLE_BASE not set in environment. It is recommended
that ORACLE_BASE be set in the environment
Reusing ORACLE_BASE from an earlier startup = /global/oracle10/11/
Thu Sep 10 14:42:58 2009
ALTER DATABASE MOUNT
Setting recovery target incarnation to 2
Successful mount of redo thread 1, with mount id 3817189298
Database mounted in Exclusive Mode
Lost write protection disabled
Completed: ALTER DATABASE MOUNT
Thu Sep 10 14:43:04 2009
ALTER DATABASE OPEN
Thread 1 opened at log sequence 4
Current log# 1 seq# 4 mem# 0: /global/oracle10/11/oradata/ora11a/redo01.log
Successful open of redo thread 1
MTTR advisory is disabled because FAST_START_MTTR_TARGET is not set
SMON: enabling cache recovery
Successfully onlined Undo Tablespace 2.
Verifying file header compatibility for 11g tablespace encryption..
Verifying 11g file header compatibility for tablespace encryption completed
SMON: enabling tx recovery
Database Characterset is WE8MSWIN1252
Starting background process SMCO
Opening with internal Resource Manager plan
Thu Sep 10 14:43:07 2009
SMCO started with pid=19, OS id=11667
Starting background process FBDA
Thu Sep 10 14:43:07 2009
FBDA started with pid=20, OS id=11669
replication_dependency_tracking turned off (no async multimaster replication found)
Starting background process QMNC
Thu Sep 10 14:43:08 2009
QMNC started with pid=21, OS id=11671
Thu Sep 10 14:43:12 2009
db_recovery_file_dest_size of 2048 MB is 0.00% used. This is a
user-specified limit on the amount of space that will be used by this
database for recovery-related files, and does not reflect the amount of
space available in the underlying filesystem or ASM diskgroup.
Thu Sep 10 14:43:39 2009
Completed: ALTER DATABASE OPEN
Thu Sep 10 14:43:49 2009
Stopping background process SMCO
Stopping background process FBDA
Shutting down instance: further logons disabled
Stopping background process QMNC
Stopping background process MMNL
Stopping background process MMON
Shutting down instance (immediate)
License high water mark = 1
All dispatchers and shared servers shutdown
ALTER DATABASE CLOSE NORMAL
Thu Sep 10 14:43:52 2009
SMON: disabling tx recovery
SMON: disabling cache recovery
Thu Sep 10 14:43:53 2009
Shutting down archive processes
Archiving is disabled
Archive process shutdown avoided: 0 active
Thread 1 closed at log sequence 4
Successful close of redo thread 1
Completed: ALTER DATABASE CLOSE NORMAL
ALTER DATABASE DISMOUNT
Completed: ALTER DATABASE DISMOUNT
ARCH: Archival disabled due to shutdown: 1089
Shutting down archive processes
Archiving is disabled
Archive process shutdown avoided: 0 active
Thu Sep 10 14:43:54 2009
Stopping background process VKTM:
ARCH: Archival disabled due to shutdown: 1089
Shutting down archive processes
Archiving is disabled
Archive process shutdown avoided: 0 active
Thu Sep 10 14:43:57 2009
Instance shutdown complete
Thu Sep 10 14:46:56 2009
Starting ORACLE instance (normal)
LICENSE_MAX_SESSION = 0
LICENSE_SESSIONS_WARNING = 0
Picked latch-free SCN scheme 3
Using LOG_ARCHIVE_DEST_1 parameter default value as /global/oracle10/11/product/11.1.0/db_1//dbs/arch
Using LOG_ARCHIVE_DEST_10 parameter default value as USE_DB_RECOVERY_FILE_DEST
Autotune of undo retention is turned on.
IMODE=BR
ILAT =18
LICENSE_MAX_USERS = 0
SYS auditing is disabled
Starting up ORACLE RDBMS Version: 11.1.0.6.0.
Using parameter settings in server-side spfile /global/oracle10/11/product/11.1.0/db_1/dbs/spfileora11a.ora
System parameters with non-default values:
processes = 150
sga_target = 752M
control_files = "/global/oracle10/11/oradata/ora11a/control01.ctl"
control_files = "/global/oracle10/11/oradata/ora11a/control02.ctl"
control_files = "/global/oracle10/11/oradata/ora11a/control03.ctl"
db_block_size = 8192
compatible = "11.1.0.0.0"
db_recovery_file_dest = "/global/oracle10/11/flash_recovery_area"
db_recovery_file_dest_size= 2G
undo_tablespace = "UNDOTBS1"
remote_login_passwordfile= "EXCLUSIVE"
db_domain = ""
dispatchers = "(PROTOCOL=TCP) (SERVICE=ora11aXDB)"
audit_file_dest = "/global/oracle10/11/admin/ora11a/adump"
audit_trail = "DB"
db_name = "ora11a"
open_cursors = 300
pga_aggregate_target = 250M
diagnostic_dest = "/global/oracle10/11"
Thu Sep 10 14:47:00 2009
PMON started with pid=2, OS id=12220
Thu Sep 10 14:47:00 2009
VKTM started with pid=3, OS id=12224 at elevated priority
VKTM running at (20)ms precision
Thu Sep 10 14:47:00 2009
DIAG started with pid=4, OS id=12228
Thu Sep 10 14:47:00 2009
DBRM started with pid=5, OS id=12230
Thu Sep 10 14:47:00 2009
PSP0 started with pid=6, OS id=12232
Thu Sep 10 14:47:00 2009
DSKM started with pid=7, OS id=12234
Thu Sep 10 14:47:01 2009
DIA0 started with pid=8, OS id=12237
Thu Sep 10 14:47:01 2009
MMAN started with pid=7, OS id=12242
Thu Sep 10 14:47:01 2009
DBW0 started with pid=9, OS id=12249
Thu Sep 10 14:47:01 2009
LGWR started with pid=10, OS id=12251
Thu Sep 10 14:47:01 2009
CKPT started with pid=11, OS id=12253
Thu Sep 10 14:47:01 2009
SMON started with pid=12, OS id=12255
Thu Sep 10 14:47:01 2009
RECO started with pid=13, OS id=12257
Thu Sep 10 14:47:01 2009
MMON started with pid=14, OS id=12259
Thu Sep 10 14:47:01 2009
MMNL started with pid=15, OS id=12261
starting up 1 dispatcher(s) for network address '(ADDRESS=(PARTIAL=YES)(PROTOCOL=TCP))'...
starting up 1 shared server(s) ...
ORACLE_BASE not set in environment. It is recommended
that ORACLE_BASE be set in the environment
Reusing ORACLE_BASE from an earlier startup = /global/oracle10/11/
Thu Sep 10 14:47:02 2009
ALTER DATABASE MOUNT
Setting recovery target incarnation to 2
Successful mount of redo thread 1, with mount id 3817161638
Database mounted in Exclusive Mode
Lost write protection disabled
Completed: ALTER DATABASE MOUNT
Thu Sep 10 14:47:08 2009
ALTER DATABASE OPEN
Thread 1 opened at log sequence 4
Current log# 1 seq# 4 mem# 0: /global/oracle10/11/oradata/ora11a/redo01.log
Successful open of redo thread 1
MTTR advisory is disabled because FAST_START_MTTR_TARGET is not set
SMON: enabling cache recovery
Successfully onlined Undo Tablespace 2.
Verifying file header compatibility for 11g tablespace encryption..
Verifying 11g file header compatibility for tablespace encryption completed
SMON: enabling tx recovery
Database Characterset is WE8MSWIN1252
Starting background process SMCO
Opening with internal Resource Manager plan
Thu Sep 10 14:47:10 2009
SMCO started with pid=19, OS id=12273
Starting background process FBDA
Thu Sep 10 14:47:11 2009
FBDA started with pid=20, OS id=12275
replication_dependency_tracking turned off (no async multimaster replication found)
Starting background process QMNC
Thu Sep 10 14:47:12 2009
QMNC started with pid=21, OS id=12277
Thu Sep 10 14:47:15 2009
db_recovery_file_dest_size of 2048 MB is 0.00% used. This is a
user-specified limit on the amount of space that will be used by this
database for recovery-related files, and does not reflect the amount of
space available in the underlying filesystem or ASM diskgroup.
Thu Sep 10 14:47:40 2009
Completed: ALTER DATABASE OPEN
Thu Sep 10 14:47:45 2009
Stopping background process SMCO
Stopping background process FBDA
Shutting down instance: further logons disabled
Stopping background process QMNC
Stopping background process MMNL
Stopping background process MMON
Shutting down instance (immediate)
License high water mark = 1
All dispatchers and shared servers shutdown
ALTER DATABASE CLOSE NORMAL
Thu Sep 10 14:47:49 2009
SMON: disabling tx recovery
SMON: disabling cache recovery
Thu Sep 10 14:47:50 2009
Shutting down archive processes
Archiving is disabled
Archive process shutdown avoided: 0 active
Thread 1 closed at log sequence 4
Successful close of redo thread 1
Completed: ALTER DATABASE CLOSE NORMAL
ALTER DATABASE DISMOUNT
Completed: ALTER DATABASE DISMOUNT
ARCH: Archival disabled due to shutdown: 1089
Shutting down archive processes
Archiving is disabled
Archive process shutdown avoided: 0 active
Thu Sep 10 14:47:52 2009
Stopping background process VKTM:
ARCH: Archival disabled due to shutdown: 1089
Shutting down archive processes
Archiving is disabled
Archive process shutdown avoided: 0 active
Thu Sep 10 14:47:55 2009
Instance shutdown complete
from sun /var/adm/messages
Sep 10 14:32:21 node1 Cluster.RGM.rgmd: [ID 927765 daemon.notice] 44 fe_rpc_command: cmd_type(enum):<1>:cmd=</usr/cluster/lib/rgm/rt/hastorageplus/hastorageplus_postnet_stop>:tag=<oracle11-rg.global_oracle11-rs.11>: Calling security_clnt_connect(..., host=<node1>, sec_type {0:WEAK, 1:STRONG, 2:DES} =<1>, ...)
Sep 10 14:32:21 node1 Cluster.RGM.rgmd: [ID 515159 daemon.notice] method <hastorageplus_postnet_stop> completed successfully for resource <global_oracle11-rs>, resource group <oracle11-rg>, node <node1>, time used: 0% of timeout <1800 seconds>
Sep 10 14:40:39 node1 Cluster.RGM.rgmd: [ID 224900 daemon.notice] launching method <bin/oracle_server_fini> for resource <ora11g-ora>, resource group <oracle11-rg>, node <node1>, timeout <30> seconds
Sep 10 14:40:39 node1 Cluster.RGM.rgmd: [ID 927765 daemon.notice] 44 fe_rpc_command: cmd_type(enum):<1>:cmd=</opt/SUNWscor/oracle_server/bin/oracle_server_fini>:tag=<oracle11-rg.ora11g-ora.5>: Calling security_clnt_connect(..., host=<node1>, sec_type {0:WEAK, 1:STRONG, 2:DES} =<1>, ...)
Sep 10 14:40:39 node1 Cluster.RGM.rgmd: [ID 515159 daemon.notice] method <bin/oracle_server_fini> completed successfully for resource <ora11g-ora>, resource group <oracle11-rg>, node <node1>, time used: 0% of timeout <30 seconds>
Sep 10 14:40:39 node1 Cluster.RGM.rgmd: [ID 927765 daemon.notice] 44 fe_rpc_command: cmd_type(enum):<5>:cmd=<null>:tag=<>: Calling security_clnt_connect(..., host=<node1>, sec_type {0:WEAK, 1:STRONG, 2:DES} =<1>, ...)
Sep 10 14:40:39 node1 Cluster.CCR: [ID 491081 daemon.notice] resource ora11g-ora removed.
Sep 10 14:41:05 node1 rmclomv: [ID 585172 kern.crit] VOLTAGE_SENSOR @ MB.BAT.V_BAT has exceeded low warning threshold.
Sep 10 14:41:35 node1 Cluster.RGM.rgmd: [ID 224900 daemon.notice] launching method <bin/oracle_server_validate> for resource <ora11g-ora>, resource group <oracle11-rg>, node <node1>, timeout <120> seconds
Sep 10 14:41:35 node1 Cluster.RGM.rgmd: [ID 285716 daemon.notice] 20 fe_rpc_command: cmd_type(enum):<1>:cmd=</opt/SUNWscor/oracle_server/bin/oracle_server_validate>:tag=<oracle11-rg.ora11g-ora.2>: Calling security_clnt_connect(..., host=<node1>, sec_type {0:WEAK, 1:STRONG, 2:DES} =<1>, ...)
Sep 10 14:41:35 node1 Cluster.RGM.rgmd: [ID 515159 daemon.notice] method <bin/oracle_server_validate> completed successfully for resource <ora11g-ora>, resource group <oracle11-rg>, node <node1>, time used: 0% of timeout <120 seconds>
Sep 10 14:41:36 node1 Cluster.RGM.rgmd: [ID 224900 daemon.notice] launching method <bin/oracle_server_init> for resource <ora11g-ora>, resource group <oracle11-rg>, node <node1>, timeout <30> seconds
Sep 10 14:41:36 node1 Cluster.RGM.rgmd: [ID 927765 daemon.notice] 44 fe_rpc_command: cmd_type(enum):<1>:cmd=</opt/SUNWscor/oracle_server/bin/oracle_server_init>:tag=<oracle11-rg.ora11g-ora.4>: Calling security_clnt_connect(..., host=<node1>, sec_type {0:WEAK, 1:STRONG, 2:DES} =<1>, ...)
Sep 10 14:41:36 node1 Cluster.RGM.rgmd: [ID 515159 daemon.notice] method <bin/oracle_server_init> completed successfully for resource <ora11g-ora>, resource group <oracle11-rg>, node <node1>, time used: 0% of timeout <30 seconds>
Sep 10 14:41:36 node1 Cluster.CCR: [ID 973933 daemon.notice] resource ora11g-ora added.
Sep 10 14:42:44 node1 Cluster.RGM.rgmd: [ID 224900 daemon.notice] launching method <hafoip_prenet_start> for resource <oracleha-rs>, resource group <oracle11-rg>, node <node1>, timeout <300> seconds
Sep 10 14:42:44 node1 Cluster.RGM.rgmd: [ID 927765 daemon.notice] 44 fe_rpc_command: cmd_type(enum):<1>:cmd=</usr/cluster/lib/rgm/rt/hafoip/hafoip_prenet_start>:tag=<oracle11-rg.oracleha-rs.10>: Calling security_clnt_connect(..., host=<node1>, sec_type {0:WEAK, 1:STRONG, 2:DES} =<1>, ...)
Sep 10 14:42:45 node1 Cluster.RGM.rgmd: [ID 515159 daemon.notice] method <hafoip_prenet_start> completed successfully for resource <oracleha-rs>, resource group <oracle11-rg>, node <node1>, time used: 0% of timeout <300 seconds>
Sep 10 14:42:45 node1 Cluster.RGM.rgmd: [ID 224900 daemon.notice] launching method <hastorageplus_prenet_start> for resource <global_oracle11-rs>, resource group <oracle11-rg>, node <node1>, timeout <1800> seconds
Sep 10 14:42:45 node1 Cluster.RGM.rgmd: [ID 927765 daemon.notice] 44 fe_rpc_command: cmd_type(enum):<1>:cmd=</usr/cluster/lib/rgm/rt/hastorageplus/hastorageplus_prenet_start>:tag=<oracle11-rg.global_oracle11-rs.10>: Calling security_clnt_connect(..., host=<node1>, sec_type {0:WEAK, 1:STRONG, 2:DES} =<1>, ...)
Sep 10 14:42:45 node1 Cluster.RGM.rgmd: [ID 515159 daemon.notice] method <hastorageplus_prenet_start> completed successfully for resource <global_oracle11-rs>, resource group <oracle11-rg>, node <node1>, time used: 0% of timeout <1800 seconds>
Sep 10 14:42:45 node1 Cluster.RGM.rgmd: [ID 224900 daemon.notice] launching method <hafoip_start> for resource <oracleha-rs>, resource group <oracle11-rg>, node <node1>, timeout <500> seconds
Sep 10 14:42:45 node1 Cluster.RGM.rgmd: [ID 927765 daemon.notice] 44 fe_rpc_command: cmd_type(enum):<1>:cmd=</usr/cluster/lib/rgm/rt/hafoip/hafoip_start>:tag=<oracle11-rg.oracleha-rs.0>: Calling security_clnt_connect(..., host=<node1>, sec_type {0:WEAK, 1:STRONG, 2:DES} =<1>, ...)
Sep 10 14:42:45 node1 Cluster.RGM.rgmd: [ID 515159 daemon.notice] method <hafoip_start> completed successfully for resource <oracleha-rs>, resource group <oracle11-rg>, node <node1>, time used: 0% of timeout <500 seconds>
Sep 10 14:42:45 node1 Cluster.RGM.rgmd: [ID 224900 daemon.notice] launching method <hastorageplus_start> for resource <global_oracle11-rs>, resource group <oracle11-rg>, node <node1>, timeout <90> seconds
Sep 10 14:42:45 node1 Cluster.RGM.rgmd: [ID 927765 daemon.notice] 44 fe_rpc_command: cmd_type(enum):<1>:cmd=</usr/cluster/lib/rgm/rt/hastorageplus/hastorageplus_start>:tag=<oracle11-rg.global_oracle11-rs.0>: Calling security_clnt_connect(..., host=<node1>, sec_type {0:WEAK, 1:STRONG, 2:DES} =<1>, ...)
Sep 10 14:42:45 node1 Cluster.RGM.rgmd: [ID 224900 daemon.notice] launching method <hafoip_monitor_start> for resource <oracleha-rs>, resource group <oracle11-rg>, node <node1>, timeout <300> seconds
Sep 10 14:42:45 node1 Cluster.RGM.rgmd: [ID 268902 daemon.notice] 45 fe_rpc_command: cmd_type(enum):<1>:cmd=</usr/cluster/lib/rgm/rt/hafoip/hafoip_monitor_start>:tag=<oracle11-rg.oracleha-rs.7>: Calling security_clnt_connect(..., host=<node1>, sec_type {0:WEAK, 1:STRONG, 2:DES} =<1>, ...)
Sep 10 14:42:46 node1 Cluster.RGM.rgmd: [ID 515159 daemon.notice] method <hafoip_monitor_start> completed successfully for resource <oracleha-rs>, resource group <oracle11-rg>, node <node1>, time used: 0% of timeout <300 seconds>
Sep 10 14:42:46 node1 Cluster.RGM.rgmd: [ID 515159 daemon.notice] method <hastorageplus_start> completed successfully for resource <global_oracle11-rs>, resource group <oracle11-rg>, node <node1>, time used: 0% of timeout <90 seconds>
Sep 10 14:42:46 node1 Cluster.RGM.rgmd: [ID 224900 daemon.notice] launching method <bin/oracle_listener_start> for resource <ora11g-lsr>, resource group <oracle11-rg>, node <node1>, timeout <300> seconds
Sep 10 14:42:46 node1 Cluster.RGM.rgmd: [ID 268902 daemon.notice] 45 fe_rpc_command: cmd_type(enum):<1>:cmd=</opt/SUNWscor/oracle_listener/bin/oracle_listener_start>:tag=<oracle11-rg.ora11g-lsr.0>: Calling security_clnt_connect(..., host=<node1>, sec_type {0:WEAK, 1:STRONG, 2:DES} =<1>, ...)
Sep 10 14:42:46 node1 Cluster.RGM.rgmd: [ID 224900 daemon.notice] launching method <hastorageplus_monitor_start> for resource <global_oracle11-rs>, resource group <oracle11-rg>, node <node1>, timeout <90> seconds
Sep 10 14:42:46 node1 Cluster.RGM.rgmd: [ID 686647 daemon.notice] 43 fe_rpc_command: cmd_type(enum):<1>:cmd=</usr/cluster/lib/rgm/rt/hastorageplus/hastorageplus_monitor_start>:tag=<oracle11-rg.global_oracle11-rs.7>: Calling security_clnt_connect(..., host=<node1>, sec_type {0:WEAK, 1:STRONG, 2:DES} =<1>, ...)
Sep 10 14:42:46 node1 Cluster.RGM.rgmd: [ID 224900 daemon.notice] launching method <bin/oracle_server_start> for resource <ora11g-ora>, resource group <oracle11-rg>, node <node1>, timeout <600> seconds
Sep 10 14:42:46 node1 Cluster.RGM.rgmd: [ID 927765 daemon.notice] 44 fe_rpc_command: cmd_type(enum):<1>:cmd=</opt/SUNWscor/oracle_server/bin/oracle_server_start>:tag=<oracle11-rg.ora11g-ora.0>: Calling security_clnt_connect(..., host=<node1>, sec_type {0:WEAK, 1:STRONG, 2:DES} =<1>, ...)
Sep 10 14:42:46 node1 Cluster.RGM.rgmd: [ID 515159 daemon.notice] method <hastorageplus_monitor_start> completed successfully for resource <global_oracle11-rs>, resource group <oracle11-rg>, node <node1>, time used: 0% of timeout <90 seconds>
Sep 10 14:42:56 node1 Cluster.RGM.rgmd: [ID 515159 daemon.notice] method <bin/oracle_listener_start> completed successfully for resource <ora11g-lsr>, resource group <oracle11-rg>, node <node1>, time used: 3% of timeout <300 seconds>
Sep 10 14:42:56 node1 Cluster.RGM.rgmd: [ID 224900 daemon.notice] launching method <bin/oracle_listener_monitor_start> for resource <ora11g-lsr>, resource group <oracle11-rg>, node <node1>, timeout <180> seconds
Sep 10 14:42:56 node1 Cluster.RGM.rgmd: [ID 268902 daemon.notice] 45 fe_rpc_command: cmd_type(enum):<1>:cmd=</opt/SUNWscor/oracle_listener/bin/oracle_listener_monitor_start>:tag=<oracle11-rg.ora11g-lsr.7>: Calling security_clnt_connect(..., host=<node1>, sec_type {0:WEAK, 1:STRONG, 2:DES} =<1>, ...)
Sep 10 14:43:00 node1 Cluster.RGM.rgmd: [ID 515159 daemon.notice] method <bin/oracle_listener_monitor_start> completed successfully for resource <ora11g-lsr>, resource group <oracle11-rg>, node <node1>, time used: 1% of timeout <180 seconds>
Sep 10 14:43:41 node1 SC[SUNWscor.oracle_server.start]:oracle11-rg:ora11g-ora: [ID 412106 daemon.error] Internal Error. Unable to get fault monitor name
Sep 10 14:43:41 node1 SC[SUNWscor.oracle_server.start]:oracle11-rg:ora11g-ora: [ID 690463 daemon.error] Cannot bring server online on this node.
Sep 10 14:43:41 node1 Cluster.RGM.rgmd: [ID 938318 daemon.error] Method <bin/oracle_server_start> failed on resource <ora11g-ora> in resource group <oracle11-rg> [exit code <255>, time used: 9% of timeout <600 seconds>]
Sep 10 14:43:41 node1 Cluster.RGM.rgmd: [ID 224900 daemon.notice] launching method <bin/oracle_server_stop> for resource <ora11g-ora>, resource group <oracle11-rg>, node <node1>, timeout <600> seconds
Sep 10 14:43:41 node1 Cluster.RGM.rgmd: [ID 927765 daemon.notice] 44 fe_rpc_command: cmd_type(enum):<1>:cmd=</opt/SUNWscor/oracle_server/bin/oracle_server_stop>:tag=<oracle11-rg.ora11g-ora.1>: Calling security_clnt_connect(..., host=<node1>, sec_type {0:WEAK, 1:STRONG, 2:DES} =<1>, ...)
Sep 10 14:43:41 node1 Cluster.RGM.rgmd: [ID 224900 daemon.notice] launching method <bin/oracle_listener_monitor_stop> for resource <ora11g-lsr>, resource group <oracle11-rg>, node <node1>, timeout <120> seconds
Sep 10 14:43:41 node1 Cluster.RGM.rgmd: [ID 268902 daemon.notice] 45 fe_rpc_command: cmd_type(enum):<1>:cmd=</opt/SUNWscor/oracle_listener/bin/oracle_listener_monitor_stop>:tag=<oracle11-rg.ora11g-lsr.8>: Calling security_clnt_connect(..., host=<node1>, sec_type {0:WEAK, 1:STRONG, 2:DES} =<1>, ...)
Sep 10 14:43:41 node1 Cluster.RGM.rgmd: [ID 224900 daemon.notice] launching method <hastorageplus_monitor_stop> for resource <global_oracle11-rs>, resource group <oracle11-rg>, node <node1>, timeout <90> seconds
Sep 10 14:43:41 node1 Cluster.RGM.rgmd: [ID 686647 daemon.notice] 43 fe_rpc_command: cmd_type(enum):<1>:cmd=</usr/cluster/lib/rgm/rt/hastorageplus/hastorageplus_monitor_stop>:tag=<oracle11-rg.global_oracle11-rs.8>: Calling security_clnt_connect(..., host=<node1>, sec_type {0:WEAK, 1:STRONG, 2:DES} =<1>, ...)
Sep 10 14:43:41 node1 Cluster.RGM.rgmd: [ID 652764 daemon.notice] libsecurity, door_call: Resource temporarily unavailable; will retry
Sep 10 14:43:41 node1 Cluster.RGM.rgmd: [ID 224900 daemon.notice] launching method <hafoip_monitor_stop> for resource <oracleha-rs>, resource group <oracle11-rg>, node <node1>, timeout <300> seconds
Sep 10 14:43:41 node1 Cluster.RGM.rgmd: [ID 510020 daemon.notice] 46 fe_rpc_command: cmd_type(enum):<1>:cmd=</usr/cluster/lib/rgm/rt/hafoip/hafoip_monitor_stop>:tag=<oracle11-rg.oracleha-rs.8>: Calling security_clnt_connect(..., host=<node1>, sec_type {0:WEAK, 1:STRONG, 2:DES} =<1>, ...)
Sep 10 14:43:41 node1 Cluster.RGM.rgmd: [ID 652764 daemon.notice] libsecurity, door_call: Resource temporarily unavailable; will retry
Sep 10 14:43:43 node1 Cluster.RGM.rgmd: [ID 515159 daemon.notice] method <hastorageplus_monitor_stop> completed successfully for resource <global_oracle11-rs>, resource group <oracle11-rg>, node <node1>, time used: 1% of timeout <90 seconds>
Sep 10 14:43:43 node1 Cluster.RGM.rgmd: [ID 515159 daemon.notice] method <hafoip_monitor_stop> completed successfully for resource <oracleha-rs>, resource group <oracle11-rg>, node <node1>, time used: 0% of timeout <300 seconds>
Sep 10 14:43:46 node1 Cluster.RGM.rgmd: [ID 515159 daemon.notice] method <bin/oracle_listener_monitor_stop> completed successfully for resource <ora11g-lsr>, resource group <oracle11-rg>, node <node1>, time used: 4% of timeout <120 seconds>
Sep 10 14:43:46 node1 Cluster.RGM.rgmd: [ID 224900 daemon.notice] launching method <bin/oracle_listener_stop> for resource <ora11g-lsr>, resource group <oracle11-rg>, node <node1>, timeout <300> seconds
Sep 10 14:43:46 node1 Cluster.RGM.rgmd: [ID 268902 daemon.notice] 45 fe_rpc_command: cmd_type(enum):<1>:cmd=</opt/SUNWscor/oracle_listener/bin/oracle_listener_stop>:tag=<oracle11-rg.ora11g-lsr.1>: Calling security_clnt_connect(..., host=<node1>, sec_type {0:WEAK, 1:STRONG, 2:DES} =<1>, ...)
Sep 10 14:43:57 node1 SC[SUNWscor.oracle_server.stop]:oracle11-rg:ora11g-ora: [ID 167253 daemon.notice] Server stopped successfully.
Sep 10 14:43:57 node1 Cluster.RGM.rgmd: [ID 515159 daemon.notice] method <bin/oracle_server_stop> completed successfully for resource <ora11g-ora>, resource group <oracle11-rg>, node <node1>, time used: 2% of timeout <600 seconds>
Sep 10 14:44:01 node1 Cluster.RGM.rgmd: [ID 515159 daemon.notice] method <bin/oracle_listener_stop> completed successfully for resource <ora11g-lsr>, resource group <oracle11-rg>, node <node1>, time used: 5% of timeout <300 seconds>
Sep 10 14:44:01 node1 Cluster.RGM.rgmd: [ID 224900 daemon.notice] launching method <hastorageplus_stop> for resource <global_oracle11-rs>, resource group <oracle11-rg>, node <node1>, timeout <1800> seconds
Sep 10 14:44:01 node1 Cluster.RGM.rgmd: [ID 268902 daemon.notice] 45 fe_rpc_command: cmd_type(enum):<1>:cmd=</usr/cluster/lib/rgm/rt/hastorageplus/hastorageplus_stop>:tag=<oracle11-rg.global_oracle11-rs.1>: Calling security_clnt_connect(..., host=<node1>, sec_type {0:WEAK, 1:STRONG, 2:DES} =<1>, ...)
Sep 10 14:44:01 node1 Cluster.RGM.rgmd: [ID 515159 daemon.notice] method <hastorageplus_stop> completed successfully for resource <global_oracle11-rs>, resource group <oracle11-rg>, node <node1>, time used: 0% of timeout <1800 seconds>
Sep 10 14:44:01 node1 Cluster.RGM.rgmd: [ID 224900 daemon.notice] launching method <hafoip_stop> for resource <oracleha-rs>, resource group <oracle11-rg>, node <node1>, timeout <300> seconds
Sep 10 14:44:01 node1 Cluster.RGM.rgmd: [ID 268902 daemon.notice] 45 fe_rpc_command: cmd_type(enum):<1>:cmd=</usr/cluster/lib/rgm/rt/hafoip/hafoip_stop>:tag=<oracle11-rg.oracleha-rs.1>: Calling security_clnt_connect(..., host=<node1>, sec_type {0:WEAK, 1:STRONG, 2:DES} =<1>, ...)
Sep 10 14:44:02 node1 ip: [ID 678092 kern.notice] TCP_IOC_ABORT_CONN: local = 172.016.002.241:0, remote = 000.000.000.000:0, start = -2, end = 6
Sep 10 14:44:02 node1 ip: [ID 302654 kern.notice] TCP_IOC_ABORT_CONN: aborted 2 connections
Sep 10 14:44:02 node1 Cluster.RGM.rgmd: [ID 515159 daemon.notice] method <hafoip_stop> completed successfully for resource <oracleha-rs>, resource group <oracle11-rg>, node <node1>, time used: 0% of timeout <300 seconds>
Sep 10 14:44:02 node1 Cluster.RGM.rgmd: [ID 224900 daemon.notice] launching method <hastorageplus_postnet_stop> for resource <global_oracle11-rs>, resource group <oracle11-rg>, node <node1>, timeout <1800> seconds
Sep 10 14:44:02 node1 Cluster.RGM.rgmd: [ID 268902 daemon.notice] 45 fe_rpc_command: cmd_type(enum):<1>:cmd=</usr/cluster/lib/rgm/rt/hastorageplus/hastorageplus_postnet_stop>:tag=<oracle11-rg.global_oracle11-rs.11>: Calling security_clnt_connect(..., host=<node1>, sec_type {0:WEAK, 1:STRONG, 2:DES} =<1>, ...)
Sep 10 14:44:02 node1 Cluster.RGM.rgmd: [ID 515159 daemon.notice] method <hastorageplus_postnet_stop> completed successfully for resource <global_oracle11-rs>, resource group <oracle11-rg>, node <node1>, time used: 0% of timeout <1800 seconds>
Sep 10 14:46:48 node1 Cluster.RGM.rgmd: [ID 224900 daemon.notice] launching method <hafoip_prenet_start> for resource <oracleha-rs>, resource group <oracle11-rg>, node <node1>, timeout <300> seconds
Sep 10 14:46:48 node1 Cluster.RGM.rgmd: [ID 268902 daemon.notice] 45 fe_rpc_command: cmd_type(enum):<1>:cmd=</usr/cluster/lib/rgm/rt/hafoip/hafoip_prenet_start>:tag=<oracle11-rg.oracleha-rs.10>: Calling security_clnt_connect(..., host=<node1>, sec_type {0:WEAK, 1:STRONG, 2:DES} =<1>, ...)
Sep 10 14:46:50 node1 Cluster.RGM.rgmd: [ID 515159 daemon.notice] method <hafoip_prenet_start> completed successfully for resource <oracleha-rs>, resource group <oracle11-rg>, node <node1>, time used: 0% of timeout <300 seconds>
Sep 10 14:46:50 node1 Cluster.RGM.rgmd: [ID 224900 daemon.notice] launching method <hastorageplus_prenet_start> for resource <global_oracle11-rs>, resource group <oracle11-rg>, node <node1>, timeout <1800> seconds
Sep 10 14:46:50 node1 Cluster.RGM.rgmd: [ID 268902 daemon.notice] 45 fe_rpc_command: cmd_type(enum):<1>:cmd=</usr/cluster/lib/rgm/rt/hastorageplus/hastorageplus_prenet_start>:tag=<oracle11-rg.global_oracle11-rs.10>: Calling security_clnt_connect(..., host=<node1>, sec_type {0:WEAK, 1:STRONG, 2:DES} =<1>, ...)
Sep 10 14:46:50 node1 Cluster.RGM.rgmd: [ID 515159 daemon.notice] method <hastorageplus_prenet_start> completed successfully for resource <global_oracle11-rs>, resource group <oracle11-rg>, node <node1>, time used: 0% of timeout <1800 seconds>
Sep 10 14:46:50 node1 Cluster.RGM.rgmd: [ID 224900 daemon.notice] launching method <hafoip_start> for resource <oracleha-rs>, resource group <oracle11-rg>, node <node1>, timeout <500> seconds
Sep 10 14:46:50 node1 Cluster.RGM.rgmd: [ID 268902 daemon.notice] 45 fe_rpc_command: cmd_type(enum):<1>:cmd=</usr/cluster/lib/rgm/rt/hafoip/hafoip_start>:tag=<oracle11-rg.oracleha-rs.0>: Calling security_clnt_connect(..., host=<node1>, sec_type {0:WEAK, 1:STRONG, 2:DES} =<1>, ...)
Sep 10 14:46:50 node1 Cluster.RGM.rgmd: [ID 515159 daemon.notice] method <hafoip_start> completed successfully for resource <oracleha-rs>, resource group <oracle11-rg>, node <node1>, time used: 0% of timeout <500 seconds>
Sep 10 14:46:50 node1 Cluster.RGM.rgmd: [ID 224900 daemon.notice] launching method <hastorageplus_start> for resource <global_oracle11-rs>, resource group <oracle11-rg>, node <node1>, timeout <90> seconds
Sep 10 14:46:50 node1 Cluster.RGM.rgmd: [ID 268902 daemon.notice] 45 fe_rpc_command: cmd_type(enum):<1>:cmd=</usr/cluster/lib/rgm/rt/hastorageplus/hastorageplus_start>:tag=<oracle11-rg.global_oracle11-rs.0>: Calling security_clnt_connect(..., host=<node1>, sec_type {0:WEAK, 1:STRONG, 2:DES} =<1>, ...)
Sep 10 14:46:50 node1 Cluster.RGM.rgmd: [ID 224900 daemon.notice] launching method <hafoip_monitor_start> for resource <oracleha-rs>, resource group <oracle11-rg>, node <node1>, timeout <300> seconds
Sep 10 14:46:50 node1 Cluster.RGM.rgmd: [ID 927765 daemon.notice] 44 fe_rpc_command: cmd_type(enum):<1>:cmd=</usr/cluster/lib/rgm/rt/hafoip/hafoip_monitor_start>:tag=<oracle11-rg.oracleha-rs.7>: Calling security_clnt_connect(..., host=<node1>, sec_type {0:WEAK, 1:STRONG, 2:DES} =<1>, ...)
Sep 10 14:46:51 node1 Cluster.RGM.rgmd: [ID 515159 daemon.notice] method <hastorageplus_start> completed successfully for resource <global_oracle11-rs>, resource group <oracle11-rg>, node <node1>, time used: 0% of timeout <90 seconds>
Sep 10 14:46:51 node1 Cluster.RGM.rgmd: [ID 224900 daemon.notice] launching method <bin/oracle_listener_start> for resource <ora11g-lsr>, resource group <oracle11-rg>, node <node1>, timeout <300> seconds
Sep 10 14:46:51 node1 Cluster.RGM.rgmd: [ID 510020 daemon.notice] 46 fe_rpc_command: cmd_type(enum):<1>:cmd=</opt/SUNWscor/oracle_listener/bin/oracle_listener_start>:tag=<oracle11-rg.ora11g-lsr.0>: Calling security_clnt_connect(..., host=<node1>, sec_type {0:WEAK, 1:STRONG, 2:DES} =<1>, ...)
Sep 10 14:46:51 node1 Cluster.RGM.rgmd: [ID 224900 daemon.notice] launching method <hastorageplus_monitor_start> for resource <global_oracle11-rs>, resource group <oracle11-rg>, node <node1>, timeout <90> seconds
Sep 10 14:46:51 node1 Cluster.RGM.rgmd: [ID 686647 daemon.notice] 43 fe_rpc_command: cmd_type(enum):<1>:cmd=</usr/cluster/lib/rgm/rt/hastorageplus/hastorageplus_monitor_start>:tag=<oracle11-rg.global_oracle11-rs.7>: Calling security_clnt_connect(..., host=<node1>, sec_type {0:WEAK, 1:STRONG, 2:DES} =<1>, ...)
Sep 10 14:46:51 node1 Cluster.RGM.rgmd: [ID 224900 daemon.notice] launching method <bin/oracle_server_start> for resource <ora11g-ora>, resource group <oracle11-rg>, node <node1>, timeout <600> seconds
Sep 10 14:46:51 node1 Cluster.RGM.rgmd: [ID 268902 daemon.notice] 45 fe_rpc_command: cmd_type(enum):<1>:cmd=</opt/SUNWscor/oracle_server/bin/oracle_server_start>:tag=<oracle11-rg.ora11g-ora.0>: Calling security_clnt_connect(..., host=<node1>, sec_type {0:WEAK, 1:STRONG, 2:DES} =<1>, ...)
Sep 10 14:46:51 node1 Cluster.RGM.rgmd: [ID 515159 daemon.notice] method <hafoip_monitor_start> completed successfully for resource <oracleha-rs>, resource group <oracle11-rg>, node <node1>, time used: 0% of timeout <300 seconds>
Sep 10 14:46:51 node1 Cluster.RGM.rgmd: [ID 515159 daemon.notice] method <hastorageplus_monitor_start> completed successfully for resource <global_oracle11-rs>, resource group <oracle11-rg>, node <node1>, time used: 0% of timeout <90 seconds>
Sep 10 14:46:59 node1 Cluster.RGM.rgmd: [ID 515159 daemon.notice] method <bin/oracle_listener_start> completed successfully for resource <ora11g-lsr>, resource group <oracle11-rg>, node <node1>, time used: 2% of timeout <300 seconds>
Sep 10 14:46:59 node1 Cluster.RGM.rgmd: [ID 224900 daemon.notice] launching method <bin/oracle_listener_monitor_start> for resource <ora11g-lsr>, resource group <oracle11-rg>, node <node1>, timeout <180> seconds
Sep 10 14:46:59 node1 Cluster.RGM.rgmd: [ID 510020 daemon.notice] 46 fe_rpc_command: cmd_type(enum):<1>:cmd=</opt/SUNWscor/oracle_listener/bin/oracle_listener_monitor_start>:tag=<oracle11-rg.ora11g-lsr.7>: Calling security_clnt_connect(..., host=<node1>, sec_type {0:WEAK, 1:STRONG, 2:DES} =<1>, ...)
Sep 10 14:47:00 node1 Cluster.RGM.rgmd: [ID 515159 daemon.notice] method <bin/oracle_listener_monitor_start> completed successfully for resource <ora11g-lsr>, resource group <oracle11-rg>, node <node1>, time used: 0% of timeout <180 seconds>
Sep 10 14:47:41 node1 SC[SUNWscor.oracle_server.start]:oracle11-rg:ora11g-ora: [ID 412106 daemon.error] Internal Error. Unable to get fault monitor name
Sep 10 14:47:41 node1 SC[SUNWscor.oracle_server.start]:oracle11-rg:ora11g-ora: [ID 690463 daemon.error] Cannot bring server online on this node.
Sep 10 14:47:42 node1 Cluster.RGM.rgmd: [ID 938318 daemon.error] Method <bin/oracle_server_start> failed on resource <ora11g-ora> in resource group <oracle11-rg> [exit code <255>, time used: 8% of timeout <600 seconds>]
Sep 10 14:47:42 node1 Cluster.RGM.rgmd: [ID 224900 daemon.notice] launching method <bin/oracle_server_stop> for resource <ora11g-ora>, resource group <oracle11-rg>, node <node1>, timeout <600> seconds
Sep 10 14:47:42 node1 Cluster.RGM.rgmd: [ID 268902 daemon.notice] 45 fe_rpc_command: cmd_type(enum):<1>:cmd=</opt/SUNWscor/oracle_server/bin/oracle_server_stop>:tag=<oracle11-rg.ora11g-ora.1>: Calling security_clnt_connect(..., host=<node1>, sec_type {0:WEAK, 1:STRONG, 2:DES} =<1>, ...)
Sep 10 14:47:42 node1 Cluster.RGM.rgmd: [ID 224900 daemon.notice] launching method <bin/oracle_listener_monitor_stop> for resource <ora11g-lsr>, resource group <oracle11-rg>, node <node1>, timeout <120> seconds
Sep 10 14:47:42 node1 Cluster.RGM.rgmd: [ID 510020 daemon.notice] 46 fe_rpc_command: cmd_type(enum):<1>:cmd=</opt/SUNWscor/oracle_listener/bin/oracle_listener_monitor_stop>:tag=<oracle11-rg.ora11g-lsr.8>: Calling security_clnt_connect(..., host=<node1>, sec_type {0:WEAK, 1:STRONG, 2:DES} =<1>, ...)
Sep 10 14:47:42 node1 Cluster.RGM.rgmd: [ID 224900 daemon.notice] launching method <hastorageplus_monitor_stop> for resource <global_oracle11-rs>, resource group <oracle11-rg>, node <node1>, timeout <90> seconds
Sep 10 14:47:42 node1 Cluster.RGM.rgmd: [ID 686647 daemon.notice] 43 fe_rpc_command: cmd_type(enum):<1>:cmd=</usr/cluster/lib/rgm/rt/hastorageplus/hastorageplus_monitor_stop>:tag=<oracle11-rg.global_oracle11-rs.8>: Calling security_clnt_connect(..., host=<node1>, sec_type {0:WEAK, 1:STRONG, 2:DES} =<1>, ...)
Sep 10 14:47:42 node1 Cluster.RGM.rgmd: [ID 224900 daemon.notice] launching method <hafoip_monitor_stop> for resource <oracleha-rs>, resource group <oracle11-rg>, node <node1>, timeout <300> seconds
Sep 10 14:47:42 node1 Cluster.RGM.rgmd: [ID 927765 daemon.notice] 44 fe_rpc_command: cmd_type(enum):<1>:cmd=</usr/cluster/lib/rgm/rt/hafoip/hafoip_monitor_stop>:tag=<oracle11-rg.oracleha-rs.8>: Calling security_clnt_connect(..., host=<node1>, sec_type {0:WEAK, 1:STRONG, 2:DES} =<1>, ...)
Sep 10 14:47:42 node1 Cluster.RGM.rgmd: [ID 652764 daemon.notice] libsecurity, door_call: Resource temporarily unavailable; will retry
Sep 10 14:47:42 node1 Cluster.RGM.rgmd: [ID 515159 daemon.notice] method <hastorageplus_monitor_stop> completed successfully for resource <global_oracle11-rs>, resource group <oracle11-rg>, node <node1>, time used: 0% of timeout <90 seconds>
Sep 10 14:47:43 node1 Cluster.RGM.rgmd: [ID 515159 daemon.notice] method <hafoip_monitor_stop> completed successfully for resource <oracleha-rs>, resource group <oracle11-rg>, node <node1>, time used: 0% of timeout <300 seconds>
Sep 10 14:47:43 node1 Cluster.RGM.rgmd: [ID 515159 daemon.notice] method <bin/oracle_listener_monitor_stop> completed successfully for resource <ora11g-lsr>, resource group <oracle11-rg>, node <node1>, time used: 1% of timeout <120 seconds>
Sep 10 14:47:43 node1 Cluster.RGM.rgmd: [ID 224900 daemon.notice] launching method <bin/oracle_listener_stop> for resource <ora11g-lsr>, resource group <oracle11-rg>, node <node1>, timeout <300> seconds
Sep 10 14:47:43 node1 Cluster.RGM.rgmd: [ID 510020 daemon.notice] 46 fe_rpc_command: cmd_type(enum):<1>:cmd=</opt/SUNWscor/oracle_listener/bin/oracle_listener_stop>:tag=<oracle11-rg.ora11g-lsr.1>: Calling security_clnt_connect(..., host=<node1>, sec_type {0:WEAK, 1:STRONG, 2:DES} =<1>, ...)
Sep 10 14:47:55 node1 Cluster.RGM.rgmd: [ID 515159 daemon.notice] method <bin/oracle_listener_stop> completed successfully for resource <ora11g-lsr>, resource group <oracle11-rg>, node <node1>, time used: 3% of timeout <300 seconds>
Sep 10 14:47:55 node1 SC[SUNWscor.oracle_server.stop]:oracle11-rg:ora11g-ora: [ID 167253 daemon.notice] Server stopped successfully.
Sep 10 14:47:55 node1 Cluster.RGM.rgmd: [ID 515159 daemon.notice] method <bin/oracle_server_stop> completed successfully for resource <ora11g-ora>, resource group <oracle11-rg>, node <node1>, time used: 2% of timeout <600 seconds>
Sep 10 14:47:55 node1 Cluster.RGM.rgmd: [ID 224900 daemon.notice] launching method <hastorageplus_stop> for resource <global_oracle11-rs>, resource group <oracle11-rg>, node <node1>, timeout <1800> seconds
Sep 10 14:47:55 node1 Cluster.RGM.rgmd: [ID 268902 daemon.notice] 45 fe_rpc_command: cmd_type(enum):<1>:cmd=</usr/cluster/lib/rgm/rt/hastorageplus/hastorageplus_stop>:tag=<oracle11-rg.global_oracle11-rs.1>: Calling security_clnt_connect(..., host=<node1>, sec_type {0:WEAK, 1:STRONG, 2:DES} =<1>, ...)
Sep 10 14:47:55 node1 Cluster.RGM.rgmd: [ID 515159 daemon.notice] method <hastorageplus_stop> completed successfully for resource <global_oracle11-rs>, resource group <oracle11-rg>, node <node1>, time used: 0% of timeout <1800 seconds>
Sep 10 14:47:55 node1 Cluster.RGM.rgmd: [ID 224900 daemon.notice] launching method <hafoip_stop> for resource <oracleha-rs>, resource group <oracle11-rg>, node <node1>, timeout <300> seconds
Sep 10 14:47:55 node1 Cluster.RGM.rgmd: [ID 268902 daemon.notice] 45 fe_rpc_command: cmd_type(enum):<1>:cmd=</usr/cluster/lib/rgm/rt/hafoip/hafoip_stop>:tag=<oracle11-rg.oracleha-rs.1>: Calling security_clnt_connect(..., host=<node1>, sec_type {0:WEAK, 1:STRONG, 2:DES} =<1>, ...)
Sep 10 14:47:55 node1 ip: [ID 678092 kern.notice] TCP_IOC_ABORT_CONN: local = 172.016.002.241:0, remote = 000.000.000.000:0, start = -2, end = 6
Sep 10 14:47:56 node1 ip: [ID 302654 kern.notice] TCP_IOC_ABORT_CONN: aborted 2 connections
Sep 10 14:47:56 node1 Cluster.RGM.rgmd: [ID 515159 daemon.notice] method <hafoip_stop> completed successfully for resource <oracleha-rs>, resource group <oracle11-rg>, node <node1>, time used: 0% of timeout <300 seconds>
Sep 10 14:47:56 node1 Cluster.RGM.rgmd: [ID 224900 daemon.notice] launching method <hastorageplus_postnet_stop> for resource <global_oracle11-rs>, resource group <oracle11-rg>, node <node1>, timeout <1800> seconds
Sep 10 14:47:56 node1 Cluster.RGM.rgmd: [ID 268902 daemon.notice] 45 fe_rpc_command: cmd_type(enum):<1>:cmd=</usr/cluster/lib/rgm/rt/hastorageplus/hastorageplus_postnet_stop>:tag=<oracle11-rg.global_oracle11-rs.11>: Calling security_clnt_connect(..., host=<node1>, sec_type {0:WEAK, 1:STRONG, 2:DES} =<1>, ...)
Sep 10 14:47:56 node1 Cluster.RGM.rgmd: [ID 515159 daemon.notice] method <hastorageplus_postnet_stop> completed successfully for resource <global_oracle11-rs>, resource group <oracle11-rg>, node <node1>, time used: 0% of timeout <1800 seconds>
From the oracle log , i see database online but in few second the databse automatic stop and cluster switch to second node , please advise us how to solve this issue .
thank
Hadi