VCS Log

ronny_nch · April 9, 2009, 4:19am

Hi All,

Can anyone help me to analyse this VCS log. One of my applications failover suddenly and I need to find out the reason behind this.

2009/04/09 10:58:57 VCS ERROR V-16-2-13067 (CS49PAPS2) Agent is calling clean for resource(Web-ebill2app) because the resource became OFFLINE unexpectedly, on its own.
2009/04/09 10:58:58 VCS NOTICE V-16-55008-10424 (CS49PAPS2) WebLogic9:Web-ebill2app:clean:VCSagentFW:SetupLogging:[clean] Entered by resource instance [Web-ebill2app] with clean reason [4][Unexpected Offline]
2009/04/09 10:58:58 VCS NOTICE V-16-55008-20080 (CS49PAPS2) WebLogic9:Web-ebill2app:clean:<WebLogic9::Kill>:WebLogic Server process not found.
2009/04/09 10:58:58 VCS INFO V-16-2-13068 (CS49PAPS2) Resource(Web-ebill2app) - clean completed successfully.
2009/04/09 10:59:00 VCS INFO V-16-1-10307 Resource Web-ebill2app (Owner: unknown, Group: Ebill2ap-app) is offline on CS49PAPS2 (Not initiated by VCS)
2009/04/09 10:59:00 VCS NOTICE V-16-1-10300 Initiating Offline of Resource Bip-ebill2app (Owner: unknown, Group: Ebill2ap-app) on System CS49PAPS2
2009/04/09 10:59:00 VCS NOTICE V-16-1-10300 Initiating Offline of Resource Mnt-ebill2app (Owner: unknown, Group: Ebill2ap-app) on System CS49PAPS2
2009/04/09 10:59:00 VCS INFO V-16-6-15004 (CS49PAPS2) hatrigger:Failed to send trigger for resfault; script doesn't exist
2009/04/09 10:59:00 VCS INFO V-16-1-10305 Resource Bip-ebill2app (Owner: unknown, Group: Ebill2ap-app) is offline on CS49PAPS2 (VCS initiated)
2009/04/09 10:59:01 VCS INFO V-16-1-10305 Resource Mnt-ebill2app (Owner: unknown, Group: Ebill2ap-app) is offline on CS49PAPS2 (VCS initiated)
2009/04/09 10:59:01 VCS NOTICE V-16-1-10300 Initiating Offline of Resource Vol-ebill2app (Owner: unknown, Group: Ebill2ap-app) on System CS49PAPS2
2009/04/09 10:59:02 VCS INFO V-16-1-10305 Resource Vol-ebill2app (Owner: unknown, Group: Ebill2ap-app) is offline on CS49PAPS2 (VCS initiated)
2009/04/09 10:59:02 VCS NOTICE V-16-1-10300 Initiating Offline of Resource Dg-ebill2app (Owner: unknown, Group: Ebill2ap-app) on System CS49PAPS2
2009/04/09 10:59:12 VCS INFO V-16-1-10305 Resource Dg-ebill2app (Owner: unknown, Group: Ebill2ap-app) is offline on CS49PAPS2 (VCS initiated)
2009/04/09 10:59:12 VCS ERROR V-16-1-10205 Group Ebill2ap-app is faulted on system CS49PAPS2
2009/04/09 10:59:12 VCS NOTICE V-16-1-10446 Group Ebill2ap-app is offline on system CS49PAPS2
2009/04/09 10:59:12 VCS INFO V-16-1-10493 Evaluating CS49PAPS2 as potential target node for group Ebill2ap-app
2009/04/09 10:59:12 VCS INFO V-16-1-50010 Group Ebill2ap-app is online or faulted on system CS49PAPS2
2009/04/09 10:59:12 VCS INFO V-16-1-10493 Evaluating TS49SDAS2 as potential target node for group Ebill2ap-app
2009/04/09 10:59:12 VCS NOTICE V-16-1-10301 Initiating Online of Resource Bip-ebill2app (Owner: unknown, Group: Ebill2ap-app) on System TS49SDAS2
2009/04/09 10:59:12 VCS NOTICE V-16-1-10301 Initiating Online of Resource Dg-ebill2app (Owner: unknown, Group: Ebill2ap-app) on System TS49SDAS2
2009/04/09 10:59:12 VCS INFO V-16-1-10298 Resource Bip-ebill2app (Owner: unknown, Group: Ebill2ap-app) is online on TS49SDAS2 (VCS initiated)
2009/04/09 10:59:13 VCS INFO V-16-6-15002 (CS49PAPS2) hatrigger:hatrigger executed /opt/VRTSvcs/bin/triggers/nfs_postoffline CS49PAPS2 Ebill2ap-app successfully
2009/04/09 10:59:13 VCS INFO V-16-6-15004 (CS49PAPS2) hatrigger:Failed to send trigger for lvmvg_postoffline; script doesn't exist
2009/04/09 10:59:13 VCS INFO V-16-6-15004 (CS49PAPS2) hatrigger:Failed to send trigger for postoffline; script doesn't exist
2009/04/09 10:59:13 VCS WARNING V-16-10001-1014 (TS49SDAS2) DiskGroup: Dg-ebill2app: online: Diskgroups will be imported without reservations
2009/04/09 10:59:15 VCS NOTICE V-16-10001-1009 (TS49SDAS2) DiskGroup: Dg-ebill2app: online:vxdg import succeeded on Disk Group ebill2apdg
2009/04/09 10:59:15 VCS NOTICE V-16-10001-1010 (TS49SDAS2) DiskGroup: Dg-ebill2app: online:Volumes in Disk Group ebill2apdg are started. Any mirrors are updated in background
2009/04/09 10:59:17 VCS INFO V-16-1-10298 Resource Dg-ebill2app (Owner: unknown, Group: Ebill2ap-app) is online on TS49SDAS2 (VCS initiated)
2009/04/09 10:59:17 VCS NOTICE V-16-1-10301 Initiating Online of Resource Vol-ebill2app (Owner: unknown, Group: Ebill2ap-app) on System TS49SDAS2
2009/04/09 10:59:17 VCS NOTICE V-16-10001-11501 (TS49SDAS2) Volume:Vol-ebill2app: online:Volume ebill2apvol is started. Any mirrors are updated in background
2009/04/09 10:59:18 VCS INFO V-16-1-10298 Resource Vol-ebill2app (Owner: unknown, Group: Ebill2ap-app) is online on TS49SDAS2 (VCS initiated)
2009/04/09 10:59:18 VCS NOTICE V-16-1-10301 Initiating Online of Resource Mnt-ebill2app (Owner: unknown, Group: Ebill2ap-app) on System TS49SDAS2
2009/04/09 10:59:19 VCS INFO V-16-1-10298 Resource Mnt-ebill2app (Owner: unknown, Group: Ebill2ap-app) is online on TS49SDAS2 (VCS initiated)
2009/04/09 10:59:19 VCS NOTICE V-16-1-10301 Initiating Online of Resource Web-ebill2app (Owner: unknown, Group: Ebill2ap-app) on System TS49SDAS2
2009/04/09 10:59:20 VCS WARNING V-16-55008-20158 (TS49SDAS2) WebLogic9:Web-ebill2app: online:<WebLogic9: DelayEntryPoint>:Server Process is not running.
2009/04/09 10:59:26 VCS NOTICE V-16-55008-20163 (TS49SDAS2) WebLogic9:Web-ebill2app: online:<WebLogic9: DelayEntryPoint>: Server not responding for first level check at [192.168.100.41:7002]. Sleeping for [5] seconds.
2009/04/09 10:59:31 VCS NOTICE V-16-55008-20163 (TS49SDAS2) WebLogic9:Web-ebill2app: online:<WebLogic9: DelayEntryPoint>: Server not responding for first level check at [192.168.100.41:7002]. Sleeping for [5] seconds.
2009/04/09 10:59:36 VCS NOTICE V-16-55008-20163 (TS49SDAS2) WebLogic9:Web-ebill2app: online:<WebLogic9: DelayEntryPoint>: Server not responding for first level check at [192.168.100.41:7002]. Sleeping for [5] seconds.
2009/04/09 10:59:41 VCS NOTICE V-16-55008-20163 (TS49SDAS2) WebLogic9:Web-ebill2app: online:<WebLogic9: DelayEntryPoint>: Server not responding for first level check at [192.168.100.41:7002]. Sleeping for [5] seconds.
2009/04/09 10:59:46 VCS NOTICE V-16-55008-20163 (TS49SDAS2) WebLogic9:Web-ebill2app: online:<WebLogic9: DelayEntryPoint>: Server not responding for first level check at [192.168.100.41:7002]. Sleeping for [5] seconds.
2009/04/09 10:59:51 VCS NOTICE V-16-55008-20163 (TS49SDAS2) WebLogic9:Web-ebill2app: online:<WebLogic9: DelayEntryPoint>: Server not responding for first level check at [192.168.100.41:7002]. Sleeping for [5] seconds.
2009/04/09 10:59:56 VCS NOTICE V-16-55008-20162 (TS49SDAS2) WebLogic9:Web-ebill2app: online:<WebLogic9: DelayEntryPoint>: First Level Check , server responding at [192.168.100.41:7002].
2009/04/09 10:59:56 VCS NOTICE V-16-55008-20255 (TS49SDAS2) WebLogic9:Web-ebill2app: online:<main::OnlineEntryPoint>:Resource [Web-ebill2app] started up successfully.
2009/04/09 10:59:58 VCS INFO V-16-1-10298 Resource Web-ebill2app (Owner: unknown, Group: Ebill2ap-app) is online on TS49SDAS2 (VCS initiated)
2009/04/09 10:59:58 VCS NOTICE V-16-1-10447 Group Ebill2ap-app is online on system TS49SDAS2
2009/04/09 10:59:58 VCS NOTICE V-16-1-10448 Group Ebill2ap-app failed over to system TS49SDAS2
2009/04/09 10:59:59 VCS INFO V-16-6-15051 (TS49SDAS2) nfs_restart:nfs_restart trigger did not do anything as there is no NFS/NFSLock/Share resource in the group
2009/04/09 10:59:59 VCS INFO V-16-6-15002 (TS49SDAS2) hatrigger:hatrigger executed /opt/VRTSvcs/bin/triggers/nfs_restart Ebill2ap-app successfully
2009/04/09 10:59:59 VCS INFO V-16-6-15004 (TS49SDAS2) hatrigger:Failed to send trigger for postonline; script doesn't exist
2009/04/09 15:46:39 VCS INFO V-16-1-50135 User root fired command: hagrp -clear Ebill2ap-app CS49PAPS2 from localhost
2009/04/09 15:46:39 VCS INFO V-16-1-10307 Resource Web-ebill2app (Owner: unknown, Group: Ebill2ap-app) is offline on CS49PAPS2 (Not initiated by VCS)

thanks

incredible · April 9, 2009, 1:42pm

the VCS log only shows that the resource grp became offline in one of the host , thus initiating a failover. But it wont explain WHY it went offline.

1jnike · April 9, 2009, 3:34pm

Hi ronny_nch

I am no expert in this field, but have a look at the highlighted areas:

<snip>

2009/04/09 10:59:12 VCS ERROR V-16-1-10205 Group Ebill2ap-app is faulted on system CS49PAPS2
2009/04/09 10:59:12 VCS NOTICE V-16-1-10446 Group Ebill2ap-app is offline on system CS49PAPS2
2009/04/09 10:59:12 VCS INFO V-16-1-10493 Evaluating CS49PAPS2 as potential target node for group Ebill2ap-app
2009/04/09 10:59:12 VCS INFO V-16-1-50010 Group Ebill2ap-app is online or faulted on system CS49PAPS2

2009/04/09 10:59:19 VCS NOTICE V-16-1-10301 Initiating Online of Resource Web-ebill2app (Owner: unknown, Group: Ebill2ap-app) on System TS49SDAS2
2009/04/09 10:59:20 VCS WARNING V-16-55008-20158 (TS49SDAS2) WebLogic9:Web-ebill2app: online:<WebLogic9: DelayEntryPoint>:Server Process is not running.2009/04/09 10:59:26 VCS NOTICE V-16-55008-20163 (TS49SDAS2) WebLogic9:Web-ebill2app: online:<WebLogic9: DelayEntryPoint>: Server not responding for first level

2009/04/09 10:59:59 VCS INFO V-16-6-15051 (TS49SDAS2) nfs_restart:nfs_restart trigger did not do anything as there is no NFS/NFSLock/Share resource in the group
2009/04/09 10:59:59 VCS INFO V-16-6-15002 (TS49SDAS2) hatrigger:hatrigger executed /opt/VRTSvcs/bin/triggers/nfs_restart Ebill2ap-app successfully
2009/04/09 10:59:59 VCS INFO V-16-6-15004 (TS49SDAS2) hatrigger:Failed to send trigger for postonline; script doesn't exist

CS49PAPS2 (Not initiated by VCS)

TS49SDAS2 (VCS initiated)

I hope this is of help.

Jnike

ronny_nch · April 13, 2009, 1:52am

Hi Guys,

Thanks for the reply. Could weblogic be the reason because the server hosts few other applications as well but they did not failover.

rgds,

incredible · April 13, 2009, 3:36am

Possible.. The cluster wont failover unnecessarily unless it hits issue with the applications like weblogic or oracle