We have a Sun Server running Solaris 10 and Veritas Cluster Server. The RAID Volumes in the Server (/ , swap, opt, var, usr) are managed by VxVm and UFS is grown on all these volumes.
Lately the system has been crashing due to an inconsistency in the opt filesystem. Upon reboot we did a fsck on the the opt multiple times and booted the system to multiuser mode. But again the system is crashing once the cluster is ok. The following is the panic message:-
panic[cpu1]/thread=3000d19a6c0: alloccgblk: can't find blk in cyl, pos:0, i:377, fs:/opt bno: 300
000002a102c50fb0 ufs:real_panic_v+60 (0, 19017f8, 2a102c51250, 30003bea000, 0, 600080e7d40)
%l0-3: 000006000832a000 0000000000090000 000006000e5d64c0 0000000000000300
%l4-7: 0000000000000180 0000000000000000 0000000000000064 0000000001826c00
000002a102c51060 ufs:ufs_fault_v+c8 (600085c7180, 19017f8, 2a102c51250, 6000d957648, 60006b2a2a8, 0)
%l0-3: 000006000832a000 0000000000090000 000006000e5d64c0 0000000000000300
%l4-7: 0000000000000180 0000000000000000 0000060006b2a200 0000000000000000
000002a102c51110 ufs:ufs_fault+1c (600085c7180, 19017f8, 0, 179, 6000832a0d4, 300)
%l0-3: 000006000832a000 0000000000090000 000006000e5d64c0 0000000000000300
%l4-7: 0000000000000180 0000000000000479 0000000000000179 000006000832a560
000002a102c511c0 ufs:alloccgblk+4c8 (1901400, 6000e5d6000, 0, 6000d957648, 2188, 0)
%l0-3: 000006000832a000 0000000000090000 000006000e5d64c0 0000000000000300
%l4-7: 0000000000000180 0000000000000479 0000000000000179 000006000832a560
000002a102c51270 ufs:alloccg+144 (90000, 60006b2a2a8, 662188, 2000, 90255, 6000e5d64c0)
%l0-3: 000006000e5d6000 000006000d957648 000006000832a2d8 0000000000000880
%l4-7: 0000000000000088 0000060006b2a200 000006000832a000 0000000000090255
000002a102c51320 ufs:hashalloc+24 (6000e8db878, 88, 662188, 2000, 122e8d0, 2a102c51480)
%l0-3: 0000060006b2a200 000006000e8db878 000006000832a000 0000000000000003
%l4-7: 0000060006b2a200 0000000000002000 0000000000000088 0000000000000088
000002a102c513d0 ufs:alloc+128 (0, 662188, 34d4f00, 2a102c51690, 600004040b8, 6000832a000)
%l0-3: 0000060006b2a200 000006000e8db878 0000000001e6a130 0000000000000003
%l4-7: 0000000000000000 0000000000002000 0000000000002000 0000000000000010
000002a102c51490 ufs:bmap_write+c40 (0, 2000, 2a102c515e8, 10, 0, 6000e8db878)
%l0-3: 0000060006b2a200 0000000000000000 0000000000662188 000002a102c515e8
%l4-7: 000000000000001c 0000000000661f48 0000000000000007 000006000d237d10
000002a102c516a0 ufs:wrip+448 (0, 2a102c51a98, ffffffffff, 2000, 6000e8db878, 8000)
%l0-3: 0000000000026000 0000000000000001 0000000000000000 0000000000000000
%l4-7: 0000060006b2a2a8 0000000000028000 0000000000000000 0000000000002000
000002a102c51810 ufs:ufs_write+580 (6000e8cdb80, 2a102c51a98, 8, 60006b2a248, 1, 6000e8db878)
%l0-3: 000006000e8db898 000006000e8db958 000006000e8db960 0000000000000001
%l4-7: 00000000019004f4 000006000e8db9b8 0000060006b2a200 0000000000000000
000002a102c51930 genunix:fop_write+20 (6000e8cdb80, 2a102c51a98, 8, 600004040b8, 0, 123ed74)
%l0-3: 0000000000002000 000006000e8cdb80 0000000000000000 000000000104db10
%l4-7: 0000000000002000 0000000000026000 0000000000000008 000000000000210a
000002a102c519e0 genunix:write+268 (1, 8058, 600155cb008, 2000, 210a, 1)
%l0-3: 0000000000000000 000006000e8cdb80 0000000000000000 000000000104db10
%l4-7: 0000000000002000 0000000000026000 0000000000000008 000000000000210a
syncing file systems... [1] 34 [1] 28 [1] 5 [1] 5 [1] 5 [1] 5 [1] 5 [1] 5 [1] 5 [1] 5 [1] 5 [1] 5 [1] 5 [1] 5 [1] 5 [1] 5 [1] 5 [1] 5 [1] 5 [1] 5 [1] 5 [1] 5 [1] 5 done (not all i/o completed)
From the vxprint -th output i see some parameter of the 2 plexes of opt has some different value as shown below:-
dm rootdisk c0t0d0s2 auto 20351 143328960 -
dm rootmirror c0t1d0s2 auto 9919 143328960 -
v opt - ENABLED SYNC 110796288 ROUND - fsgen
pl opt-01 opt ENABLED ACTIVE 110796288 CONCAT - RW
sd rootdisk-03 opt-01 rootdisk 32532672 110796288 0 c0t0d0 ENA
pl opt-02 opt ENABLED ACTIVE 110796288 CONCAT - RW
sd rootmirror-05 opt-02 rootmirror 32512320 110796288 0 c0t1d0 ENA
Please help me with a fix.