sun cluster 3.2 + oracle 11g HA problem C748634 ,C135343

Dear Cluster Guru,

   I had installed sc 3.2 on 2 box sunfire 240 \(solaris 10 update 7 ,each node 2 gb ram\) and use scsi juke box as share storage .

i had test cluster application :

       1. apache HA 
        2 oracleHA \(oracle 9i\) 

all running well (switch over between node working and oracle running good).

Here my step setup the cluster app :

ORACLE 11G installation :

-mount share storage as global device on node1 (as oracle_base and oracle_home)
-Install oracle
-test startup database use sql command from node1 and node2 (status working , database mount normal)
-change hotnname on listener file with logical hostname (oracleha) (/global/oracle10/11/product/11.1.0/db_1/network/admin/listener.ora)

Cluster APP setup :

# clresourcegroup create -n node1,node2 oracle11-rg
# clreslogicalhostname create -g oracle11-rg -h oracleha oracleha-rs
# clresource create -g oracle11-rg -t SUNW.HAStoragePlus -p FilesystemMountPoints=/global/oracle10 -p GlobalDevicePaths=oracle10 global_oracle11-rs
# clresourcegroup online -M oracle11-rg
# clresource create -g oracle11-rg -t SUNW.oracle_server -p Connect_string=SYSTEM/manager  -p ORACLE_SID=ora11a -p ORACLE_HOME=/global/oracle10/11/product/11.1.0/db_1/ -p Alert_log_file=/global/oracle10/11/product/11.1.0/db_1/log/alert_ora11a.log -p Restart_type=RESOURCE_GROUP_RESTART -p Resource_dependencies=global_oracle11-rs ora11g-ora
# clresource create -g oracle11-rg -t SUNW.oracle_listener -p LISTENER_NAME=listener -p ORACLE_HOME=/global/oracle10/11/product/11.1.0/db_1 -p Resource_dependencies=global_oracle11-rs ora11g-lsr

When I try online the oracle11-rg and get problem :

   clresourcegroup: (C748634) Resource group oracle11-rg failed to start on chosen node and might fail over to other node(s)
   clresourcegroup: (C135343) No primary node could be found for resource group oracle11-rg; it remains offline
    Resource Group Status
# cluster show

 --- Resources for Group oracle11-rg ---

    Resource:                                   oracleha-rs
      Type:                                        SUNW.LogicalHostname:2
      Type_version:                                2
      Group:                                       oracle11-rg
      R_description:
      Resource_project_name:                       default
      Enabled{node1}:                              True
      Enabled{node2}:                              True
      Monitored{node1}:                            True
      Monitored{node2}:                            True

    Resource:                                   global_oracle11-rs
      Type:                                        SUNW.HAStoragePlus:4
      Type_version:                                4
      Group:                                       oracle11-rg
      R_description:
      Resource_project_name:                       default
      Enabled{node1}:                              True
      Enabled{node2}:                              True
      Monitored{node1}:                            True
      Monitored{node2}:                            True

    Resource:                                   ora11g-lsr
      Type:                                        SUNW.oracle_listener:5
      Type_version:                                5
      Group:                                       oracle11-rg
      R_description:
      Resource_project_name:                       default
      Enabled{node1}:                              True
      Enabled{node2}:                              True
      Monitored{node1}:                            True
      Monitored{node2}:                            True

    Resource:                                   ora11g-ora
      Type:                                        SUNW.oracle_server:6
      Type_version:                                6
      Group:                                       oracle11-rg
      R_description:
      Resource_project_name:                       default
      Enabled{node1}:                              True
      Enabled{node2}:                              True
      Monitored{node1}:                            True
      Monitored{node2}:                            True

  === DID Device Instances ===

  DID Device Name:                              /dev/did/rdsk/d1
    Full Device Path:                              node2:/dev/rdsk/c1t0d0
    Replication:                                   none
    default_fencing:                               global

  DID Device Name:                              /dev/did/rdsk/d2
    Full Device Path:                              node2:/dev/rdsk/c3t2d0
    Full Device Path:                              node1:/dev/rdsk/c3t2d0
    Replication:                                   none
    default_fencing:                               global

  DID Device Name:                              /dev/did/rdsk/d3
    Full Device Path:                              node2:/dev/rdsk/c3t5d0
    Full Device Path:                              node1:/dev/rdsk/c3t5d0
    Replication:                                   none
    default_fencing:                               global

  DID Device Name:                              /dev/did/rdsk/d4
    Full Device Path:                              node1:/dev/rdsk/c0t0d0
    Replication:                                   none
    default_fencing:                               global

  DID Device Name:                              /dev/did/rdsk/d5
    Full Device Path:                              node1:/dev/rdsk/c1t0d0
    Replication:                                   none
    default_fencing:                               global

  DID Device Name:                              /dev/did/rdsk/d6
    Full Device Path:                              node2:/dev/rdsk/c0t0d0
    Replication:                                   none
    default_fencing:                               global

=== NAS Devices ===

I try disable Resource: ora11g-ora and on oracle11-rg (the status online ,and can switch between node 1 to node2 )

from the oracle trace :

cat  20 /global/oracle10/11/diag/rdbms/ora11a/ora11a/trace/alert_ora11a.log
Thu Sep 10 14:42:56 2009
DIAG started with pid=4, OS id=11556
Thu Sep 10 14:42:56 2009
DBRM started with pid=5, OS id=11558
Thu Sep 10 14:42:56 2009
PSP0 started with pid=6, OS id=11560
Thu Sep 10 14:42:56 2009
DSKM started with pid=7, OS id=11562
Thu Sep 10 14:42:56 2009
DIA0 started with pid=8, OS id=11564
Thu Sep 10 14:42:56 2009
MMAN started with pid=9, OS id=11566
Thu Sep 10 14:42:56 2009
DBW0 started with pid=7, OS id=11581
Thu Sep 10 14:42:57 2009
LGWR started with pid=10, OS id=11590
Thu Sep 10 14:42:57 2009
CKPT started with pid=11, OS id=11594
Thu Sep 10 14:42:57 2009
SMON started with pid=12, OS id=11598
Thu Sep 10 14:42:57 2009
RECO started with pid=13, OS id=11604
Thu Sep 10 14:42:57 2009
MMON started with pid=14, OS id=11608
Thu Sep 10 14:42:57 2009
MMNL started with pid=15, OS id=11612
starting up 1 dispatcher(s) for network address '(ADDRESS=(PARTIAL=YES)(PROTOCOL=TCP))'...
starting up 1 shared server(s) ...
ORACLE_BASE not set in environment. It is recommended
that ORACLE_BASE be set in the environment
Reusing ORACLE_BASE from an earlier startup = /global/oracle10/11/
Thu Sep 10 14:42:58 2009
ALTER DATABASE   MOUNT
Setting recovery target incarnation to 2
Successful mount of redo thread 1, with mount id 3817189298
Database mounted in Exclusive Mode
Lost write protection disabled
Completed: ALTER DATABASE   MOUNT
Thu Sep 10 14:43:04 2009
ALTER DATABASE OPEN
Thread 1 opened at log sequence 4
  Current log# 1 seq# 4 mem# 0: /global/oracle10/11/oradata/ora11a/redo01.log
Successful open of redo thread 1
MTTR advisory is disabled because FAST_START_MTTR_TARGET is not set
SMON: enabling cache recovery
Successfully onlined Undo Tablespace 2.
Verifying file header compatibility for 11g tablespace encryption..
Verifying 11g file header compatibility for tablespace encryption completed
SMON: enabling tx recovery
Database Characterset is WE8MSWIN1252
Starting background process SMCO
Opening with internal Resource Manager plan
Thu Sep 10 14:43:07 2009
SMCO started with pid=19, OS id=11667
Starting background process FBDA
Thu Sep 10 14:43:07 2009
FBDA started with pid=20, OS id=11669
replication_dependency_tracking turned off (no async multimaster replication found)
Starting background process QMNC
Thu Sep 10 14:43:08 2009
QMNC started with pid=21, OS id=11671
Thu Sep 10 14:43:12 2009
db_recovery_file_dest_size of 2048 MB is 0.00% used. This is a
user-specified limit on the amount of space that will be used by this
database for recovery-related files, and does not reflect the amount of
space available in the underlying filesystem or ASM diskgroup.
Thu Sep 10 14:43:39 2009
Completed: ALTER DATABASE OPEN
Thu Sep 10 14:43:49 2009
Stopping background process SMCO
Stopping background process FBDA
Shutting down instance: further logons disabled
Stopping background process QMNC
Stopping background process MMNL
Stopping background process MMON
Shutting down instance (immediate)
License high water mark = 1
All dispatchers and shared servers shutdown
ALTER DATABASE CLOSE NORMAL
Thu Sep 10 14:43:52 2009
SMON: disabling tx recovery
SMON: disabling cache recovery
Thu Sep 10 14:43:53 2009
Shutting down archive processes
Archiving is disabled
Archive process shutdown avoided: 0 active
Thread 1 closed at log sequence 4
Successful close of redo thread 1
Completed: ALTER DATABASE CLOSE NORMAL
ALTER DATABASE DISMOUNT
Completed: ALTER DATABASE DISMOUNT
ARCH: Archival disabled due to shutdown: 1089
Shutting down archive processes
Archiving is disabled
Archive process shutdown avoided: 0 active
Thu Sep 10 14:43:54 2009
Stopping background process VKTM:
ARCH: Archival disabled due to shutdown: 1089
Shutting down archive processes
Archiving is disabled
Archive process shutdown avoided: 0 active
Thu Sep 10 14:43:57 2009
Instance shutdown complete
Thu Sep 10 14:46:56 2009
Starting ORACLE instance (normal)
LICENSE_MAX_SESSION = 0
LICENSE_SESSIONS_WARNING = 0
Picked latch-free SCN scheme 3
Using LOG_ARCHIVE_DEST_1 parameter default value as /global/oracle10/11/product/11.1.0/db_1//dbs/arch
Using LOG_ARCHIVE_DEST_10 parameter default value as USE_DB_RECOVERY_FILE_DEST
Autotune of undo retention is turned on.
IMODE=BR
ILAT =18
LICENSE_MAX_USERS = 0
SYS auditing is disabled
Starting up ORACLE RDBMS Version: 11.1.0.6.0.
Using parameter settings in server-side spfile /global/oracle10/11/product/11.1.0/db_1/dbs/spfileora11a.ora
System parameters with non-default values:
  processes                = 150
  sga_target               = 752M
  control_files            = "/global/oracle10/11/oradata/ora11a/control01.ctl"
  control_files            = "/global/oracle10/11/oradata/ora11a/control02.ctl"
  control_files            = "/global/oracle10/11/oradata/ora11a/control03.ctl"
  db_block_size            = 8192
  compatible               = "11.1.0.0.0"
  db_recovery_file_dest    = "/global/oracle10/11/flash_recovery_area"
  db_recovery_file_dest_size= 2G
  undo_tablespace          = "UNDOTBS1"
  remote_login_passwordfile= "EXCLUSIVE"
  db_domain                = ""
  dispatchers              = "(PROTOCOL=TCP) (SERVICE=ora11aXDB)"
  audit_file_dest          = "/global/oracle10/11/admin/ora11a/adump"
  audit_trail              = "DB"
  db_name                  = "ora11a"
  open_cursors             = 300
  pga_aggregate_target     = 250M
  diagnostic_dest          = "/global/oracle10/11"
Thu Sep 10 14:47:00 2009
PMON started with pid=2, OS id=12220
Thu Sep 10 14:47:00 2009
VKTM started with pid=3, OS id=12224 at elevated priority
VKTM running at (20)ms precision
Thu Sep 10 14:47:00 2009
DIAG started with pid=4, OS id=12228
Thu Sep 10 14:47:00 2009
DBRM started with pid=5, OS id=12230
Thu Sep 10 14:47:00 2009
PSP0 started with pid=6, OS id=12232
Thu Sep 10 14:47:00 2009
DSKM started with pid=7, OS id=12234
Thu Sep 10 14:47:01 2009
DIA0 started with pid=8, OS id=12237
Thu Sep 10 14:47:01 2009
MMAN started with pid=7, OS id=12242
Thu Sep 10 14:47:01 2009
DBW0 started with pid=9, OS id=12249
Thu Sep 10 14:47:01 2009
LGWR started with pid=10, OS id=12251
Thu Sep 10 14:47:01 2009
CKPT started with pid=11, OS id=12253
Thu Sep 10 14:47:01 2009
SMON started with pid=12, OS id=12255
Thu Sep 10 14:47:01 2009
RECO started with pid=13, OS id=12257
Thu Sep 10 14:47:01 2009
MMON started with pid=14, OS id=12259
Thu Sep 10 14:47:01 2009
MMNL started with pid=15, OS id=12261
starting up 1 dispatcher(s) for network address '(ADDRESS=(PARTIAL=YES)(PROTOCOL=TCP))'...
starting up 1 shared server(s) ...
ORACLE_BASE not set in environment. It is recommended
that ORACLE_BASE be set in the environment
Reusing ORACLE_BASE from an earlier startup = /global/oracle10/11/
Thu Sep 10 14:47:02 2009
ALTER DATABASE   MOUNT
Setting recovery target incarnation to 2
Successful mount of redo thread 1, with mount id 3817161638
Database mounted in Exclusive Mode
Lost write protection disabled
Completed: ALTER DATABASE   MOUNT
Thu Sep 10 14:47:08 2009
ALTER DATABASE OPEN
Thread 1 opened at log sequence 4
  Current log# 1 seq# 4 mem# 0: /global/oracle10/11/oradata/ora11a/redo01.log
Successful open of redo thread 1
MTTR advisory is disabled because FAST_START_MTTR_TARGET is not set
SMON: enabling cache recovery
Successfully onlined Undo Tablespace 2.
Verifying file header compatibility for 11g tablespace encryption..
Verifying 11g file header compatibility for tablespace encryption completed
SMON: enabling tx recovery
Database Characterset is WE8MSWIN1252
Starting background process SMCO
Opening with internal Resource Manager plan
Thu Sep 10 14:47:10 2009
SMCO started with pid=19, OS id=12273
Starting background process FBDA
Thu Sep 10 14:47:11 2009
FBDA started with pid=20, OS id=12275
replication_dependency_tracking turned off (no async multimaster replication found)
Starting background process QMNC
Thu Sep 10 14:47:12 2009
QMNC started with pid=21, OS id=12277
Thu Sep 10 14:47:15 2009
db_recovery_file_dest_size of 2048 MB is 0.00% used. This is a
user-specified limit on the amount of space that will be used by this
database for recovery-related files, and does not reflect the amount of
space available in the underlying filesystem or ASM diskgroup.
Thu Sep 10 14:47:40 2009
Completed: ALTER DATABASE OPEN
Thu Sep 10 14:47:45 2009
Stopping background process SMCO
Stopping background process FBDA
Shutting down instance: further logons disabled
Stopping background process QMNC
Stopping background process MMNL
Stopping background process MMON
Shutting down instance (immediate)
License high water mark = 1
All dispatchers and shared servers shutdown
ALTER DATABASE CLOSE NORMAL
Thu Sep 10 14:47:49 2009
SMON: disabling tx recovery
SMON: disabling cache recovery
Thu Sep 10 14:47:50 2009
Shutting down archive processes
Archiving is disabled
Archive process shutdown avoided: 0 active
Thread 1 closed at log sequence 4
Successful close of redo thread 1
Completed: ALTER DATABASE CLOSE NORMAL
ALTER DATABASE DISMOUNT
Completed: ALTER DATABASE DISMOUNT
ARCH: Archival disabled due to shutdown: 1089
Shutting down archive processes
Archiving is disabled
Archive process shutdown avoided: 0 active
Thu Sep 10 14:47:52 2009
Stopping background process VKTM:
ARCH: Archival disabled due to shutdown: 1089
Shutting down archive processes
Archiving is disabled
Archive process shutdown avoided: 0 active
Thu Sep 10 14:47:55 2009
Instance shutdown complete

from sun /var/adm/messages

Sep 10 14:32:21 node1 Cluster.RGM.rgmd: [ID 927765 daemon.notice] 44 fe_rpc_command: cmd_type(enum):<1>:cmd=</usr/cluster/lib/rgm/rt/hastorageplus/hastorageplus_postnet_stop>:tag=<oracle11-rg.global_oracle11-rs.11>: Calling security_clnt_connect(..., host=<node1>, sec_type {0:WEAK, 1:STRONG, 2:DES} =<1>, ...)
Sep 10 14:32:21 node1 Cluster.RGM.rgmd: [ID 515159 daemon.notice] method <hastorageplus_postnet_stop> completed successfully for resource <global_oracle11-rs>, resource group <oracle11-rg>, node <node1>, time used: 0% of timeout <1800 seconds>
Sep 10 14:40:39 node1 Cluster.RGM.rgmd: [ID 224900 daemon.notice] launching method <bin/oracle_server_fini> for resource <ora11g-ora>, resource group <oracle11-rg>, node <node1>, timeout <30> seconds
Sep 10 14:40:39 node1 Cluster.RGM.rgmd: [ID 927765 daemon.notice] 44 fe_rpc_command: cmd_type(enum):<1>:cmd=</opt/SUNWscor/oracle_server/bin/oracle_server_fini>:tag=<oracle11-rg.ora11g-ora.5>: Calling security_clnt_connect(..., host=<node1>, sec_type {0:WEAK, 1:STRONG, 2:DES} =<1>, ...)
Sep 10 14:40:39 node1 Cluster.RGM.rgmd: [ID 515159 daemon.notice] method <bin/oracle_server_fini> completed successfully for resource <ora11g-ora>, resource group <oracle11-rg>, node <node1>, time used: 0% of timeout <30 seconds>
Sep 10 14:40:39 node1 Cluster.RGM.rgmd: [ID 927765 daemon.notice] 44 fe_rpc_command: cmd_type(enum):<5>:cmd=<null>:tag=<>: Calling security_clnt_connect(..., host=<node1>, sec_type {0:WEAK, 1:STRONG, 2:DES} =<1>, ...)
Sep 10 14:40:39 node1 Cluster.CCR: [ID 491081 daemon.notice] resource ora11g-ora removed.
Sep 10 14:41:05 node1 rmclomv: [ID 585172 kern.crit] VOLTAGE_SENSOR @ MB.BAT.V_BAT has exceeded low warning threshold.
Sep 10 14:41:35 node1 Cluster.RGM.rgmd: [ID 224900 daemon.notice] launching method <bin/oracle_server_validate> for resource <ora11g-ora>, resource group <oracle11-rg>, node <node1>, timeout <120> seconds
Sep 10 14:41:35 node1 Cluster.RGM.rgmd: [ID 285716 daemon.notice] 20 fe_rpc_command: cmd_type(enum):<1>:cmd=</opt/SUNWscor/oracle_server/bin/oracle_server_validate>:tag=<oracle11-rg.ora11g-ora.2>: Calling security_clnt_connect(..., host=<node1>, sec_type {0:WEAK, 1:STRONG, 2:DES} =<1>, ...)
Sep 10 14:41:35 node1 Cluster.RGM.rgmd: [ID 515159 daemon.notice] method <bin/oracle_server_validate> completed successfully for resource <ora11g-ora>, resource group <oracle11-rg>, node <node1>, time used: 0% of timeout <120 seconds>
Sep 10 14:41:36 node1 Cluster.RGM.rgmd: [ID 224900 daemon.notice] launching method <bin/oracle_server_init> for resource <ora11g-ora>, resource group <oracle11-rg>, node <node1>, timeout <30> seconds
Sep 10 14:41:36 node1 Cluster.RGM.rgmd: [ID 927765 daemon.notice] 44 fe_rpc_command: cmd_type(enum):<1>:cmd=</opt/SUNWscor/oracle_server/bin/oracle_server_init>:tag=<oracle11-rg.ora11g-ora.4>: Calling security_clnt_connect(..., host=<node1>, sec_type {0:WEAK, 1:STRONG, 2:DES} =<1>, ...)
Sep 10 14:41:36 node1 Cluster.RGM.rgmd: [ID 515159 daemon.notice] method <bin/oracle_server_init> completed successfully for resource <ora11g-ora>, resource group <oracle11-rg>, node <node1>, time used: 0% of timeout <30 seconds>
Sep 10 14:41:36 node1 Cluster.CCR: [ID 973933 daemon.notice] resource ora11g-ora added.
Sep 10 14:42:44 node1 Cluster.RGM.rgmd: [ID 224900 daemon.notice] launching method <hafoip_prenet_start> for resource <oracleha-rs>, resource group <oracle11-rg>, node <node1>, timeout <300> seconds
Sep 10 14:42:44 node1 Cluster.RGM.rgmd: [ID 927765 daemon.notice] 44 fe_rpc_command: cmd_type(enum):<1>:cmd=</usr/cluster/lib/rgm/rt/hafoip/hafoip_prenet_start>:tag=<oracle11-rg.oracleha-rs.10>: Calling security_clnt_connect(..., host=<node1>, sec_type {0:WEAK, 1:STRONG, 2:DES} =<1>, ...)
Sep 10 14:42:45 node1 Cluster.RGM.rgmd: [ID 515159 daemon.notice] method <hafoip_prenet_start> completed successfully for resource <oracleha-rs>, resource group <oracle11-rg>, node <node1>, time used: 0% of timeout <300 seconds>
Sep 10 14:42:45 node1 Cluster.RGM.rgmd: [ID 224900 daemon.notice] launching method <hastorageplus_prenet_start> for resource <global_oracle11-rs>, resource group <oracle11-rg>, node <node1>, timeout <1800> seconds
Sep 10 14:42:45 node1 Cluster.RGM.rgmd: [ID 927765 daemon.notice] 44 fe_rpc_command: cmd_type(enum):<1>:cmd=</usr/cluster/lib/rgm/rt/hastorageplus/hastorageplus_prenet_start>:tag=<oracle11-rg.global_oracle11-rs.10>: Calling security_clnt_connect(..., host=<node1>, sec_type {0:WEAK, 1:STRONG, 2:DES} =<1>, ...)
Sep 10 14:42:45 node1 Cluster.RGM.rgmd: [ID 515159 daemon.notice] method <hastorageplus_prenet_start> completed successfully for resource <global_oracle11-rs>, resource group <oracle11-rg>, node <node1>, time used: 0% of timeout <1800 seconds>
Sep 10 14:42:45 node1 Cluster.RGM.rgmd: [ID 224900 daemon.notice] launching method <hafoip_start> for resource <oracleha-rs>, resource group <oracle11-rg>, node <node1>, timeout <500> seconds
Sep 10 14:42:45 node1 Cluster.RGM.rgmd: [ID 927765 daemon.notice] 44 fe_rpc_command: cmd_type(enum):<1>:cmd=</usr/cluster/lib/rgm/rt/hafoip/hafoip_start>:tag=<oracle11-rg.oracleha-rs.0>: Calling security_clnt_connect(..., host=<node1>, sec_type {0:WEAK, 1:STRONG, 2:DES} =<1>, ...)
Sep 10 14:42:45 node1 Cluster.RGM.rgmd: [ID 515159 daemon.notice] method <hafoip_start> completed successfully for resource <oracleha-rs>, resource group <oracle11-rg>, node <node1>, time used: 0% of timeout <500 seconds>
Sep 10 14:42:45 node1 Cluster.RGM.rgmd: [ID 224900 daemon.notice] launching method <hastorageplus_start> for resource <global_oracle11-rs>, resource group <oracle11-rg>, node <node1>, timeout <90> seconds
Sep 10 14:42:45 node1 Cluster.RGM.rgmd: [ID 927765 daemon.notice] 44 fe_rpc_command: cmd_type(enum):<1>:cmd=</usr/cluster/lib/rgm/rt/hastorageplus/hastorageplus_start>:tag=<oracle11-rg.global_oracle11-rs.0>: Calling security_clnt_connect(..., host=<node1>, sec_type {0:WEAK, 1:STRONG, 2:DES} =<1>, ...)
Sep 10 14:42:45 node1 Cluster.RGM.rgmd: [ID 224900 daemon.notice] launching method <hafoip_monitor_start> for resource <oracleha-rs>, resource group <oracle11-rg>, node <node1>, timeout <300> seconds
Sep 10 14:42:45 node1 Cluster.RGM.rgmd: [ID 268902 daemon.notice] 45 fe_rpc_command: cmd_type(enum):<1>:cmd=</usr/cluster/lib/rgm/rt/hafoip/hafoip_monitor_start>:tag=<oracle11-rg.oracleha-rs.7>: Calling security_clnt_connect(..., host=<node1>, sec_type {0:WEAK, 1:STRONG, 2:DES} =<1>, ...)
Sep 10 14:42:46 node1 Cluster.RGM.rgmd: [ID 515159 daemon.notice] method <hafoip_monitor_start> completed successfully for resource <oracleha-rs>, resource group <oracle11-rg>, node <node1>, time used: 0% of timeout <300 seconds>
Sep 10 14:42:46 node1 Cluster.RGM.rgmd: [ID 515159 daemon.notice] method <hastorageplus_start> completed successfully for resource <global_oracle11-rs>, resource group <oracle11-rg>, node <node1>, time used: 0% of timeout <90 seconds>
Sep 10 14:42:46 node1 Cluster.RGM.rgmd: [ID 224900 daemon.notice] launching method <bin/oracle_listener_start> for resource <ora11g-lsr>, resource group <oracle11-rg>, node <node1>, timeout <300> seconds
Sep 10 14:42:46 node1 Cluster.RGM.rgmd: [ID 268902 daemon.notice] 45 fe_rpc_command: cmd_type(enum):<1>:cmd=</opt/SUNWscor/oracle_listener/bin/oracle_listener_start>:tag=<oracle11-rg.ora11g-lsr.0>: Calling security_clnt_connect(..., host=<node1>, sec_type {0:WEAK, 1:STRONG, 2:DES} =<1>, ...)
Sep 10 14:42:46 node1 Cluster.RGM.rgmd: [ID 224900 daemon.notice] launching method <hastorageplus_monitor_start> for resource <global_oracle11-rs>, resource group <oracle11-rg>, node <node1>, timeout <90> seconds
Sep 10 14:42:46 node1 Cluster.RGM.rgmd: [ID 686647 daemon.notice] 43 fe_rpc_command: cmd_type(enum):<1>:cmd=</usr/cluster/lib/rgm/rt/hastorageplus/hastorageplus_monitor_start>:tag=<oracle11-rg.global_oracle11-rs.7>: Calling security_clnt_connect(..., host=<node1>, sec_type {0:WEAK, 1:STRONG, 2:DES} =<1>, ...)
Sep 10 14:42:46 node1 Cluster.RGM.rgmd: [ID 224900 daemon.notice] launching method <bin/oracle_server_start> for resource <ora11g-ora>, resource group <oracle11-rg>, node <node1>, timeout <600> seconds
Sep 10 14:42:46 node1 Cluster.RGM.rgmd: [ID 927765 daemon.notice] 44 fe_rpc_command: cmd_type(enum):<1>:cmd=</opt/SUNWscor/oracle_server/bin/oracle_server_start>:tag=<oracle11-rg.ora11g-ora.0>: Calling security_clnt_connect(..., host=<node1>, sec_type {0:WEAK, 1:STRONG, 2:DES} =<1>, ...)
Sep 10 14:42:46 node1 Cluster.RGM.rgmd: [ID 515159 daemon.notice] method <hastorageplus_monitor_start> completed successfully for resource <global_oracle11-rs>, resource group <oracle11-rg>, node <node1>, time used: 0% of timeout <90 seconds>
Sep 10 14:42:56 node1 Cluster.RGM.rgmd: [ID 515159 daemon.notice] method <bin/oracle_listener_start> completed successfully for resource <ora11g-lsr>, resource group <oracle11-rg>, node <node1>, time used: 3% of timeout <300 seconds>
Sep 10 14:42:56 node1 Cluster.RGM.rgmd: [ID 224900 daemon.notice] launching method <bin/oracle_listener_monitor_start> for resource <ora11g-lsr>, resource group <oracle11-rg>, node <node1>, timeout <180> seconds
Sep 10 14:42:56 node1 Cluster.RGM.rgmd: [ID 268902 daemon.notice] 45 fe_rpc_command: cmd_type(enum):<1>:cmd=</opt/SUNWscor/oracle_listener/bin/oracle_listener_monitor_start>:tag=<oracle11-rg.ora11g-lsr.7>: Calling security_clnt_connect(..., host=<node1>, sec_type {0:WEAK, 1:STRONG, 2:DES} =<1>, ...)
Sep 10 14:43:00 node1 Cluster.RGM.rgmd: [ID 515159 daemon.notice] method <bin/oracle_listener_monitor_start> completed successfully for resource <ora11g-lsr>, resource group <oracle11-rg>, node <node1>, time used: 1% of timeout <180 seconds>
Sep 10 14:43:41 node1 SC[SUNWscor.oracle_server.start]:oracle11-rg:ora11g-ora: [ID 412106 daemon.error] Internal Error. Unable to get fault monitor name
Sep 10 14:43:41 node1 SC[SUNWscor.oracle_server.start]:oracle11-rg:ora11g-ora: [ID 690463 daemon.error] Cannot bring server online on this node.
Sep 10 14:43:41 node1 Cluster.RGM.rgmd: [ID 938318 daemon.error] Method <bin/oracle_server_start> failed on resource <ora11g-ora> in resource group <oracle11-rg> [exit code <255>, time used: 9% of timeout <600 seconds>]
Sep 10 14:43:41 node1 Cluster.RGM.rgmd: [ID 224900 daemon.notice] launching method <bin/oracle_server_stop> for resource <ora11g-ora>, resource group <oracle11-rg>, node <node1>, timeout <600> seconds
Sep 10 14:43:41 node1 Cluster.RGM.rgmd: [ID 927765 daemon.notice] 44 fe_rpc_command: cmd_type(enum):<1>:cmd=</opt/SUNWscor/oracle_server/bin/oracle_server_stop>:tag=<oracle11-rg.ora11g-ora.1>: Calling security_clnt_connect(..., host=<node1>, sec_type {0:WEAK, 1:STRONG, 2:DES} =<1>, ...)
Sep 10 14:43:41 node1 Cluster.RGM.rgmd: [ID 224900 daemon.notice] launching method <bin/oracle_listener_monitor_stop> for resource <ora11g-lsr>, resource group <oracle11-rg>, node <node1>, timeout <120> seconds
Sep 10 14:43:41 node1 Cluster.RGM.rgmd: [ID 268902 daemon.notice] 45 fe_rpc_command: cmd_type(enum):<1>:cmd=</opt/SUNWscor/oracle_listener/bin/oracle_listener_monitor_stop>:tag=<oracle11-rg.ora11g-lsr.8>: Calling security_clnt_connect(..., host=<node1>, sec_type {0:WEAK, 1:STRONG, 2:DES} =<1>, ...)
Sep 10 14:43:41 node1 Cluster.RGM.rgmd: [ID 224900 daemon.notice] launching method <hastorageplus_monitor_stop> for resource <global_oracle11-rs>, resource group <oracle11-rg>, node <node1>, timeout <90> seconds
Sep 10 14:43:41 node1 Cluster.RGM.rgmd: [ID 686647 daemon.notice] 43 fe_rpc_command: cmd_type(enum):<1>:cmd=</usr/cluster/lib/rgm/rt/hastorageplus/hastorageplus_monitor_stop>:tag=<oracle11-rg.global_oracle11-rs.8>: Calling security_clnt_connect(..., host=<node1>, sec_type {0:WEAK, 1:STRONG, 2:DES} =<1>, ...)
Sep 10 14:43:41 node1 Cluster.RGM.rgmd: [ID 652764 daemon.notice] libsecurity, door_call: Resource temporarily unavailable; will retry
Sep 10 14:43:41 node1 Cluster.RGM.rgmd: [ID 224900 daemon.notice] launching method <hafoip_monitor_stop> for resource <oracleha-rs>, resource group <oracle11-rg>, node <node1>, timeout <300> seconds
Sep 10 14:43:41 node1 Cluster.RGM.rgmd: [ID 510020 daemon.notice] 46 fe_rpc_command: cmd_type(enum):<1>:cmd=</usr/cluster/lib/rgm/rt/hafoip/hafoip_monitor_stop>:tag=<oracle11-rg.oracleha-rs.8>: Calling security_clnt_connect(..., host=<node1>, sec_type {0:WEAK, 1:STRONG, 2:DES} =<1>, ...)
Sep 10 14:43:41 node1 Cluster.RGM.rgmd: [ID 652764 daemon.notice] libsecurity, door_call: Resource temporarily unavailable; will retry
Sep 10 14:43:43 node1 Cluster.RGM.rgmd: [ID 515159 daemon.notice] method <hastorageplus_monitor_stop> completed successfully for resource <global_oracle11-rs>, resource group <oracle11-rg>, node <node1>, time used: 1% of timeout <90 seconds>
Sep 10 14:43:43 node1 Cluster.RGM.rgmd: [ID 515159 daemon.notice] method <hafoip_monitor_stop> completed successfully for resource <oracleha-rs>, resource group <oracle11-rg>, node <node1>, time used: 0% of timeout <300 seconds>
Sep 10 14:43:46 node1 Cluster.RGM.rgmd: [ID 515159 daemon.notice] method <bin/oracle_listener_monitor_stop> completed successfully for resource <ora11g-lsr>, resource group <oracle11-rg>, node <node1>, time used: 4% of timeout <120 seconds>
Sep 10 14:43:46 node1 Cluster.RGM.rgmd: [ID 224900 daemon.notice] launching method <bin/oracle_listener_stop> for resource <ora11g-lsr>, resource group <oracle11-rg>, node <node1>, timeout <300> seconds
Sep 10 14:43:46 node1 Cluster.RGM.rgmd: [ID 268902 daemon.notice] 45 fe_rpc_command: cmd_type(enum):<1>:cmd=</opt/SUNWscor/oracle_listener/bin/oracle_listener_stop>:tag=<oracle11-rg.ora11g-lsr.1>: Calling security_clnt_connect(..., host=<node1>, sec_type {0:WEAK, 1:STRONG, 2:DES} =<1>, ...)
Sep 10 14:43:57 node1 SC[SUNWscor.oracle_server.stop]:oracle11-rg:ora11g-ora: [ID 167253 daemon.notice] Server stopped successfully.
Sep 10 14:43:57 node1 Cluster.RGM.rgmd: [ID 515159 daemon.notice] method <bin/oracle_server_stop> completed successfully for resource <ora11g-ora>, resource group <oracle11-rg>, node <node1>, time used: 2% of timeout <600 seconds>
Sep 10 14:44:01 node1 Cluster.RGM.rgmd: [ID 515159 daemon.notice] method <bin/oracle_listener_stop> completed successfully for resource <ora11g-lsr>, resource group <oracle11-rg>, node <node1>, time used: 5% of timeout <300 seconds>
Sep 10 14:44:01 node1 Cluster.RGM.rgmd: [ID 224900 daemon.notice] launching method <hastorageplus_stop> for resource <global_oracle11-rs>, resource group <oracle11-rg>, node <node1>, timeout <1800> seconds
Sep 10 14:44:01 node1 Cluster.RGM.rgmd: [ID 268902 daemon.notice] 45 fe_rpc_command: cmd_type(enum):<1>:cmd=</usr/cluster/lib/rgm/rt/hastorageplus/hastorageplus_stop>:tag=<oracle11-rg.global_oracle11-rs.1>: Calling security_clnt_connect(..., host=<node1>, sec_type {0:WEAK, 1:STRONG, 2:DES} =<1>, ...)
Sep 10 14:44:01 node1 Cluster.RGM.rgmd: [ID 515159 daemon.notice] method <hastorageplus_stop> completed successfully for resource <global_oracle11-rs>, resource group <oracle11-rg>, node <node1>, time used: 0% of timeout <1800 seconds>
Sep 10 14:44:01 node1 Cluster.RGM.rgmd: [ID 224900 daemon.notice] launching method <hafoip_stop> for resource <oracleha-rs>, resource group <oracle11-rg>, node <node1>, timeout <300> seconds
Sep 10 14:44:01 node1 Cluster.RGM.rgmd: [ID 268902 daemon.notice] 45 fe_rpc_command: cmd_type(enum):<1>:cmd=</usr/cluster/lib/rgm/rt/hafoip/hafoip_stop>:tag=<oracle11-rg.oracleha-rs.1>: Calling security_clnt_connect(..., host=<node1>, sec_type {0:WEAK, 1:STRONG, 2:DES} =<1>, ...)
Sep 10 14:44:02 node1 ip: [ID 678092 kern.notice] TCP_IOC_ABORT_CONN: local = 172.016.002.241:0, remote = 000.000.000.000:0, start = -2, end = 6
Sep 10 14:44:02 node1 ip: [ID 302654 kern.notice] TCP_IOC_ABORT_CONN: aborted 2 connections
Sep 10 14:44:02 node1 Cluster.RGM.rgmd: [ID 515159 daemon.notice] method <hafoip_stop> completed successfully for resource <oracleha-rs>, resource group <oracle11-rg>, node <node1>, time used: 0% of timeout <300 seconds>
Sep 10 14:44:02 node1 Cluster.RGM.rgmd: [ID 224900 daemon.notice] launching method <hastorageplus_postnet_stop> for resource <global_oracle11-rs>, resource group <oracle11-rg>, node <node1>, timeout <1800> seconds
Sep 10 14:44:02 node1 Cluster.RGM.rgmd: [ID 268902 daemon.notice] 45 fe_rpc_command: cmd_type(enum):<1>:cmd=</usr/cluster/lib/rgm/rt/hastorageplus/hastorageplus_postnet_stop>:tag=<oracle11-rg.global_oracle11-rs.11>: Calling security_clnt_connect(..., host=<node1>, sec_type {0:WEAK, 1:STRONG, 2:DES} =<1>, ...)
Sep 10 14:44:02 node1 Cluster.RGM.rgmd: [ID 515159 daemon.notice] method <hastorageplus_postnet_stop> completed successfully for resource <global_oracle11-rs>, resource group <oracle11-rg>, node <node1>, time used: 0% of timeout <1800 seconds>
Sep 10 14:46:48 node1 Cluster.RGM.rgmd: [ID 224900 daemon.notice] launching method <hafoip_prenet_start> for resource <oracleha-rs>, resource group <oracle11-rg>, node <node1>, timeout <300> seconds
Sep 10 14:46:48 node1 Cluster.RGM.rgmd: [ID 268902 daemon.notice] 45 fe_rpc_command: cmd_type(enum):<1>:cmd=</usr/cluster/lib/rgm/rt/hafoip/hafoip_prenet_start>:tag=<oracle11-rg.oracleha-rs.10>: Calling security_clnt_connect(..., host=<node1>, sec_type {0:WEAK, 1:STRONG, 2:DES} =<1>, ...)
Sep 10 14:46:50 node1 Cluster.RGM.rgmd: [ID 515159 daemon.notice] method <hafoip_prenet_start> completed successfully for resource <oracleha-rs>, resource group <oracle11-rg>, node <node1>, time used: 0% of timeout <300 seconds>
Sep 10 14:46:50 node1 Cluster.RGM.rgmd: [ID 224900 daemon.notice] launching method <hastorageplus_prenet_start> for resource <global_oracle11-rs>, resource group <oracle11-rg>, node <node1>, timeout <1800> seconds
Sep 10 14:46:50 node1 Cluster.RGM.rgmd: [ID 268902 daemon.notice] 45 fe_rpc_command: cmd_type(enum):<1>:cmd=</usr/cluster/lib/rgm/rt/hastorageplus/hastorageplus_prenet_start>:tag=<oracle11-rg.global_oracle11-rs.10>: Calling security_clnt_connect(..., host=<node1>, sec_type {0:WEAK, 1:STRONG, 2:DES} =<1>, ...)
Sep 10 14:46:50 node1 Cluster.RGM.rgmd: [ID 515159 daemon.notice] method <hastorageplus_prenet_start> completed successfully for resource <global_oracle11-rs>, resource group <oracle11-rg>, node <node1>, time used: 0% of timeout <1800 seconds>
Sep 10 14:46:50 node1 Cluster.RGM.rgmd: [ID 224900 daemon.notice] launching method <hafoip_start> for resource <oracleha-rs>, resource group <oracle11-rg>, node <node1>, timeout <500> seconds
Sep 10 14:46:50 node1 Cluster.RGM.rgmd: [ID 268902 daemon.notice] 45 fe_rpc_command: cmd_type(enum):<1>:cmd=</usr/cluster/lib/rgm/rt/hafoip/hafoip_start>:tag=<oracle11-rg.oracleha-rs.0>: Calling security_clnt_connect(..., host=<node1>, sec_type {0:WEAK, 1:STRONG, 2:DES} =<1>, ...)
Sep 10 14:46:50 node1 Cluster.RGM.rgmd: [ID 515159 daemon.notice] method <hafoip_start> completed successfully for resource <oracleha-rs>, resource group <oracle11-rg>, node <node1>, time used: 0% of timeout <500 seconds>
Sep 10 14:46:50 node1 Cluster.RGM.rgmd: [ID 224900 daemon.notice] launching method <hastorageplus_start> for resource <global_oracle11-rs>, resource group <oracle11-rg>, node <node1>, timeout <90> seconds
Sep 10 14:46:50 node1 Cluster.RGM.rgmd: [ID 268902 daemon.notice] 45 fe_rpc_command: cmd_type(enum):<1>:cmd=</usr/cluster/lib/rgm/rt/hastorageplus/hastorageplus_start>:tag=<oracle11-rg.global_oracle11-rs.0>: Calling security_clnt_connect(..., host=<node1>, sec_type {0:WEAK, 1:STRONG, 2:DES} =<1>, ...)
Sep 10 14:46:50 node1 Cluster.RGM.rgmd: [ID 224900 daemon.notice] launching method <hafoip_monitor_start> for resource <oracleha-rs>, resource group <oracle11-rg>, node <node1>, timeout <300> seconds
Sep 10 14:46:50 node1 Cluster.RGM.rgmd: [ID 927765 daemon.notice] 44 fe_rpc_command: cmd_type(enum):<1>:cmd=</usr/cluster/lib/rgm/rt/hafoip/hafoip_monitor_start>:tag=<oracle11-rg.oracleha-rs.7>: Calling security_clnt_connect(..., host=<node1>, sec_type {0:WEAK, 1:STRONG, 2:DES} =<1>, ...)
Sep 10 14:46:51 node1 Cluster.RGM.rgmd: [ID 515159 daemon.notice] method <hastorageplus_start> completed successfully for resource <global_oracle11-rs>, resource group <oracle11-rg>, node <node1>, time used: 0% of timeout <90 seconds>
Sep 10 14:46:51 node1 Cluster.RGM.rgmd: [ID 224900 daemon.notice] launching method <bin/oracle_listener_start> for resource <ora11g-lsr>, resource group <oracle11-rg>, node <node1>, timeout <300> seconds
Sep 10 14:46:51 node1 Cluster.RGM.rgmd: [ID 510020 daemon.notice] 46 fe_rpc_command: cmd_type(enum):<1>:cmd=</opt/SUNWscor/oracle_listener/bin/oracle_listener_start>:tag=<oracle11-rg.ora11g-lsr.0>: Calling security_clnt_connect(..., host=<node1>, sec_type {0:WEAK, 1:STRONG, 2:DES} =<1>, ...)
Sep 10 14:46:51 node1 Cluster.RGM.rgmd: [ID 224900 daemon.notice] launching method <hastorageplus_monitor_start> for resource <global_oracle11-rs>, resource group <oracle11-rg>, node <node1>, timeout <90> seconds
Sep 10 14:46:51 node1 Cluster.RGM.rgmd: [ID 686647 daemon.notice] 43 fe_rpc_command: cmd_type(enum):<1>:cmd=</usr/cluster/lib/rgm/rt/hastorageplus/hastorageplus_monitor_start>:tag=<oracle11-rg.global_oracle11-rs.7>: Calling security_clnt_connect(..., host=<node1>, sec_type {0:WEAK, 1:STRONG, 2:DES} =<1>, ...)
Sep 10 14:46:51 node1 Cluster.RGM.rgmd: [ID 224900 daemon.notice] launching method <bin/oracle_server_start> for resource <ora11g-ora>, resource group <oracle11-rg>, node <node1>, timeout <600> seconds
Sep 10 14:46:51 node1 Cluster.RGM.rgmd: [ID 268902 daemon.notice] 45 fe_rpc_command: cmd_type(enum):<1>:cmd=</opt/SUNWscor/oracle_server/bin/oracle_server_start>:tag=<oracle11-rg.ora11g-ora.0>: Calling security_clnt_connect(..., host=<node1>, sec_type {0:WEAK, 1:STRONG, 2:DES} =<1>, ...)
Sep 10 14:46:51 node1 Cluster.RGM.rgmd: [ID 515159 daemon.notice] method <hafoip_monitor_start> completed successfully for resource <oracleha-rs>, resource group <oracle11-rg>, node <node1>, time used: 0% of timeout <300 seconds>
Sep 10 14:46:51 node1 Cluster.RGM.rgmd: [ID 515159 daemon.notice] method <hastorageplus_monitor_start> completed successfully for resource <global_oracle11-rs>, resource group <oracle11-rg>, node <node1>, time used: 0% of timeout <90 seconds>
Sep 10 14:46:59 node1 Cluster.RGM.rgmd: [ID 515159 daemon.notice] method <bin/oracle_listener_start> completed successfully for resource <ora11g-lsr>, resource group <oracle11-rg>, node <node1>, time used: 2% of timeout <300 seconds>
Sep 10 14:46:59 node1 Cluster.RGM.rgmd: [ID 224900 daemon.notice] launching method <bin/oracle_listener_monitor_start> for resource <ora11g-lsr>, resource group <oracle11-rg>, node <node1>, timeout <180> seconds
Sep 10 14:46:59 node1 Cluster.RGM.rgmd: [ID 510020 daemon.notice] 46 fe_rpc_command: cmd_type(enum):<1>:cmd=</opt/SUNWscor/oracle_listener/bin/oracle_listener_monitor_start>:tag=<oracle11-rg.ora11g-lsr.7>: Calling security_clnt_connect(..., host=<node1>, sec_type {0:WEAK, 1:STRONG, 2:DES} =<1>, ...)
Sep 10 14:47:00 node1 Cluster.RGM.rgmd: [ID 515159 daemon.notice] method <bin/oracle_listener_monitor_start> completed successfully for resource <ora11g-lsr>, resource group <oracle11-rg>, node <node1>, time used: 0% of timeout <180 seconds>
Sep 10 14:47:41 node1 SC[SUNWscor.oracle_server.start]:oracle11-rg:ora11g-ora: [ID 412106 daemon.error] Internal Error. Unable to get fault monitor name
Sep 10 14:47:41 node1 SC[SUNWscor.oracle_server.start]:oracle11-rg:ora11g-ora: [ID 690463 daemon.error] Cannot bring server online on this node.
Sep 10 14:47:42 node1 Cluster.RGM.rgmd: [ID 938318 daemon.error] Method <bin/oracle_server_start> failed on resource <ora11g-ora> in resource group <oracle11-rg> [exit code <255>, time used: 8% of timeout <600 seconds>]
Sep 10 14:47:42 node1 Cluster.RGM.rgmd: [ID 224900 daemon.notice] launching method <bin/oracle_server_stop> for resource <ora11g-ora>, resource group <oracle11-rg>, node <node1>, timeout <600> seconds
Sep 10 14:47:42 node1 Cluster.RGM.rgmd: [ID 268902 daemon.notice] 45 fe_rpc_command: cmd_type(enum):<1>:cmd=</opt/SUNWscor/oracle_server/bin/oracle_server_stop>:tag=<oracle11-rg.ora11g-ora.1>: Calling security_clnt_connect(..., host=<node1>, sec_type {0:WEAK, 1:STRONG, 2:DES} =<1>, ...)
Sep 10 14:47:42 node1 Cluster.RGM.rgmd: [ID 224900 daemon.notice] launching method <bin/oracle_listener_monitor_stop> for resource <ora11g-lsr>, resource group <oracle11-rg>, node <node1>, timeout <120> seconds
Sep 10 14:47:42 node1 Cluster.RGM.rgmd: [ID 510020 daemon.notice] 46 fe_rpc_command: cmd_type(enum):<1>:cmd=</opt/SUNWscor/oracle_listener/bin/oracle_listener_monitor_stop>:tag=<oracle11-rg.ora11g-lsr.8>: Calling security_clnt_connect(..., host=<node1>, sec_type {0:WEAK, 1:STRONG, 2:DES} =<1>, ...)
Sep 10 14:47:42 node1 Cluster.RGM.rgmd: [ID 224900 daemon.notice] launching method <hastorageplus_monitor_stop> for resource <global_oracle11-rs>, resource group <oracle11-rg>, node <node1>, timeout <90> seconds
Sep 10 14:47:42 node1 Cluster.RGM.rgmd: [ID 686647 daemon.notice] 43 fe_rpc_command: cmd_type(enum):<1>:cmd=</usr/cluster/lib/rgm/rt/hastorageplus/hastorageplus_monitor_stop>:tag=<oracle11-rg.global_oracle11-rs.8>: Calling security_clnt_connect(..., host=<node1>, sec_type {0:WEAK, 1:STRONG, 2:DES} =<1>, ...)
Sep 10 14:47:42 node1 Cluster.RGM.rgmd: [ID 224900 daemon.notice] launching method <hafoip_monitor_stop> for resource <oracleha-rs>, resource group <oracle11-rg>, node <node1>, timeout <300> seconds
Sep 10 14:47:42 node1 Cluster.RGM.rgmd: [ID 927765 daemon.notice] 44 fe_rpc_command: cmd_type(enum):<1>:cmd=</usr/cluster/lib/rgm/rt/hafoip/hafoip_monitor_stop>:tag=<oracle11-rg.oracleha-rs.8>: Calling security_clnt_connect(..., host=<node1>, sec_type {0:WEAK, 1:STRONG, 2:DES} =<1>, ...)
Sep 10 14:47:42 node1 Cluster.RGM.rgmd: [ID 652764 daemon.notice] libsecurity, door_call: Resource temporarily unavailable; will retry
Sep 10 14:47:42 node1 Cluster.RGM.rgmd: [ID 515159 daemon.notice] method <hastorageplus_monitor_stop> completed successfully for resource <global_oracle11-rs>, resource group <oracle11-rg>, node <node1>, time used: 0% of timeout <90 seconds>
Sep 10 14:47:43 node1 Cluster.RGM.rgmd: [ID 515159 daemon.notice] method <hafoip_monitor_stop> completed successfully for resource <oracleha-rs>, resource group <oracle11-rg>, node <node1>, time used: 0% of timeout <300 seconds>
Sep 10 14:47:43 node1 Cluster.RGM.rgmd: [ID 515159 daemon.notice] method <bin/oracle_listener_monitor_stop> completed successfully for resource <ora11g-lsr>, resource group <oracle11-rg>, node <node1>, time used: 1% of timeout <120 seconds>
Sep 10 14:47:43 node1 Cluster.RGM.rgmd: [ID 224900 daemon.notice] launching method <bin/oracle_listener_stop> for resource <ora11g-lsr>, resource group <oracle11-rg>, node <node1>, timeout <300> seconds
Sep 10 14:47:43 node1 Cluster.RGM.rgmd: [ID 510020 daemon.notice] 46 fe_rpc_command: cmd_type(enum):<1>:cmd=</opt/SUNWscor/oracle_listener/bin/oracle_listener_stop>:tag=<oracle11-rg.ora11g-lsr.1>: Calling security_clnt_connect(..., host=<node1>, sec_type {0:WEAK, 1:STRONG, 2:DES} =<1>, ...)
Sep 10 14:47:55 node1 Cluster.RGM.rgmd: [ID 515159 daemon.notice] method <bin/oracle_listener_stop> completed successfully for resource <ora11g-lsr>, resource group <oracle11-rg>, node <node1>, time used: 3% of timeout <300 seconds>
Sep 10 14:47:55 node1 SC[SUNWscor.oracle_server.stop]:oracle11-rg:ora11g-ora: [ID 167253 daemon.notice] Server stopped successfully.
Sep 10 14:47:55 node1 Cluster.RGM.rgmd: [ID 515159 daemon.notice] method <bin/oracle_server_stop> completed successfully for resource <ora11g-ora>, resource group <oracle11-rg>, node <node1>, time used: 2% of timeout <600 seconds>
Sep 10 14:47:55 node1 Cluster.RGM.rgmd: [ID 224900 daemon.notice] launching method <hastorageplus_stop> for resource <global_oracle11-rs>, resource group <oracle11-rg>, node <node1>, timeout <1800> seconds
Sep 10 14:47:55 node1 Cluster.RGM.rgmd: [ID 268902 daemon.notice] 45 fe_rpc_command: cmd_type(enum):<1>:cmd=</usr/cluster/lib/rgm/rt/hastorageplus/hastorageplus_stop>:tag=<oracle11-rg.global_oracle11-rs.1>: Calling security_clnt_connect(..., host=<node1>, sec_type {0:WEAK, 1:STRONG, 2:DES} =<1>, ...)
Sep 10 14:47:55 node1 Cluster.RGM.rgmd: [ID 515159 daemon.notice] method <hastorageplus_stop> completed successfully for resource <global_oracle11-rs>, resource group <oracle11-rg>, node <node1>, time used: 0% of timeout <1800 seconds>
Sep 10 14:47:55 node1 Cluster.RGM.rgmd: [ID 224900 daemon.notice] launching method <hafoip_stop> for resource <oracleha-rs>, resource group <oracle11-rg>, node <node1>, timeout <300> seconds
Sep 10 14:47:55 node1 Cluster.RGM.rgmd: [ID 268902 daemon.notice] 45 fe_rpc_command: cmd_type(enum):<1>:cmd=</usr/cluster/lib/rgm/rt/hafoip/hafoip_stop>:tag=<oracle11-rg.oracleha-rs.1>: Calling security_clnt_connect(..., host=<node1>, sec_type {0:WEAK, 1:STRONG, 2:DES} =<1>, ...)
Sep 10 14:47:55 node1 ip: [ID 678092 kern.notice] TCP_IOC_ABORT_CONN: local = 172.016.002.241:0, remote = 000.000.000.000:0, start = -2, end = 6
Sep 10 14:47:56 node1 ip: [ID 302654 kern.notice] TCP_IOC_ABORT_CONN: aborted 2 connections
Sep 10 14:47:56 node1 Cluster.RGM.rgmd: [ID 515159 daemon.notice] method <hafoip_stop> completed successfully for resource <oracleha-rs>, resource group <oracle11-rg>, node <node1>, time used: 0% of timeout <300 seconds>
Sep 10 14:47:56 node1 Cluster.RGM.rgmd: [ID 224900 daemon.notice] launching method <hastorageplus_postnet_stop> for resource <global_oracle11-rs>, resource group <oracle11-rg>, node <node1>, timeout <1800> seconds
Sep 10 14:47:56 node1 Cluster.RGM.rgmd: [ID 268902 daemon.notice] 45 fe_rpc_command: cmd_type(enum):<1>:cmd=</usr/cluster/lib/rgm/rt/hastorageplus/hastorageplus_postnet_stop>:tag=<oracle11-rg.global_oracle11-rs.11>: Calling security_clnt_connect(..., host=<node1>, sec_type {0:WEAK, 1:STRONG, 2:DES} =<1>, ...)
Sep 10 14:47:56 node1 Cluster.RGM.rgmd: [ID 515159 daemon.notice] method <hastorageplus_postnet_stop> completed successfully for resource <global_oracle11-rs>, resource group <oracle11-rg>, node <node1>, time used: 0% of timeout <1800 seconds>

From the oracle log , i see database online but in few second the databse automatic stop and cluster switch to second node , please advise us how to solve this issue .

thank

Hadi

To keep the forums high quality for all users, please take the time to format your posts correctly.

First of all, use Code Tags when you post any code or data samples so others can easily read your code. You can easily do this by highlighting your code and then clicking on the # in the editing menu. (You can also type code tags

```text
 and 
```

by hand.)

Second, avoid adding color or different fonts and font size to your posts. Selective use of color to highlight a single word or phrase can be useful at times, but using color, in general, makes the forums harder to read, especially bright colors like red.

Third, be careful when you cut-and-paste, edit any odd characters and make sure all links are working property.

Thank You.

The UNIX and Linux Forums

Have you installed the latest patches for the Oracle HA data service? I seem to recall that we needed a fix to support Oracle 11g due to library name changes. The error messages that concern me are:

Sep 10 14:47:41 node1 SC[SUNWscor.oracle_server.start]:oracle11-rg:ora11g-ora: [ID 412106 daemon.error] Internal Error. Unable to get fault monitor name
Sep 10 14:47:41 node1 SC[SUNWscor.oracle_server.start]:oracle11-rg:ora11g-ora: [ID 690463 daemon.error] Cannot bring server online on this node.

By the way, there is an entire forum devoted to Solaris Cluster here and is monitored by many of the developers.

Tim
---

Dear All ,

   The problem solve after install patches 126047-07. 

Thanks

hadi