I am working with a Sun StoredgeTek 6540 disk array connected to two Sun 490 servers. After taking one of the 490 nodes on the cluster down to single user mode I proceeded to install the latest cluster patch from Oracle. After the patch was completed the system rebooted, failed to rejoin the cluster, crashdumped and rebooted continuously. The second node which was not updated and was functioning fine beforehand is also no longer able to connect to all the drives or the main Veritas drive group. I was running Solaris 10 137137-09 Sun Cluster 3.2 and Veritas 5 on both 490s before the cluster patch. I have since removed all the patches from the cluster patch and now both systems can boot up and join the cluster but cannot access the main Veritas drive group.
I have tried vxconfigrestore which fails.
Here are some sample errors when booting:
Dec 22 10:59:22 server02 Cluster.Framework: stderr: VxVM vxdg WARNING V-5-1-560
Dec 22 10:59:22 server02 Cluster.Framework: stderr: Disk satadg08: Not found, last known location: FLEXLINE0_4
Dec 22 10:59:22 server02 VxVM vxdg WARNING V-5-1-560 Disk satadg31: Not found, last known location: FLEXLINE0_0
Dec 22 10:59:22 server02 VxVM vxdg WARNING V-5-1-560 Disk satadg10: Not found, last known location: FLEXLINE0_10
..............
Dec 22 10:55:35 server02 scsi: WARNING: /scsi_vhci/ssd@g500000e014adbd30 (ssd2):
Dec 22 10:55:35 server02 drive offline
Dec 22 10:56:18 server02 vxvm:vxconfigd: V-5-1-8646 Error in claiming /dev/rdsk/c6t600A0B8000470CC2000008A347E0DDD6d0s2: I/O error
Dec 22 10:56:18 server02 vxvm:vxconfigd: V-5-1-8646 Error in claiming /dev/rdsk/c6t600A0B80004702300000085147E0E215d0s2: I/O error
Dec 22 10:56:18 server02 vxvm:vxconfigd: V-5-1-8646 Error in claiming /dev/rdsk/c6t600A0B80004702300000082147E0D921d0s2: I/O error
..........
I have copies of vxprint and vxdisk output from when the system was functioning normally and have 3 months worth of full tape backups but do not know where to begin to try and restore the main disk group. Been doing tons of reading online and just cannot seem to find the identical situation to mine. Not yet familiar enough with Veritas and Sun Cluster to fix this but am trying to learn as much as possible from this experience before this server is later put into production.
Search Oracle/Sun support site with the panic details from your system. I have been reading through the last year's patch reports and have seen reports dealing with Vertas and Sun Cluster issues after applying patch clusters.
At this point I'll be happy just to get everything mounted again. I just attempted another restore and it looks like I may have made some progress except for the "argument expected" before it completed.When I do a vxdisk list all of the FLEXLINE devices are shown but if I try to mount one of the volumes it hangs for a long time without any errors. And I can't seem to import the disk group either. I am a little concerned about committing the changes unless there's some way I can verify this configuration for sure. Any ideas?
-bash-3.00# vxconfigrestore -p 1206045518.165.server02
Diskgroup 1206045518.165.server02 configuration restoration started ......
Installing volume manager disk header for FLEXLINE0_0 ...
Installing volume manager disk header for FLEXLINE0_1 ...
Installing volume manager disk header for FLEXLINE0_10 ...
Installing volume manager disk header for FLEXLINE0_11 ...
Installing volume manager disk header for FLEXLINE0_12 ...
Installing volume manager disk header for FLEXLINE0_13 ...
Installing volume manager disk header for FLEXLINE0_14 ...
Installing volume manager disk header for FLEXLINE0_2 ...
Installing volume manager disk header for FLEXLINE0_3 ...
Installing volume manager disk header for FLEXLINE0_4 ...
Installing volume manager disk header for FLEXLINE0_5 ...
Installing volume manager disk header for FLEXLINE0_6 ...
Installing volume manager disk header for FLEXLINE0_7 ...
Installing volume manager disk header for FLEXLINE0_8 ...
Installing volume manager disk header for FLEXLINE0_9 ...
Installing volume manager disk header for FLEXLINE0_15 ...
Installing volume manager disk header for FLEXLINE0_16 ...
Installing volume manager disk header for FLEXLINE0_17 ...
Installing volume manager disk header for FLEXLINE0_18 ...
Installing volume manager disk header for FLEXLINE0_19 ...
Installing volume manager disk header for FLEXLINE0_20 ...
Installing volume manager disk header for FLEXLINE0_21 ...
Installing volume manager disk header for FLEXLINE0_22 ...
Installing volume manager disk header for FLEXLINE0_23 ...
Installing volume manager disk header for FLEXLINE0_24 ...
Installing volume manager disk header for FLEXLINE0_25 ...
Installing volume manager disk header for FLEXLINE0_26 ...
Installing volume manager disk header for FLEXLINE0_27 ...
Installing volume manager disk header for FLEXLINE0_28 ...
Installing volume manager disk header for FLEXLINE0_29 ...