Serial Split Brain detected in solaris10

upengan78 · April 2, 2010, 3:51pm

Hello,

I have connected storedge d2 to V440 running solaris 10. I had a raid 5 volume created using VXVM software. This morning when I rebooted the system, I get below error,

Apr  2 14:21:10 servername vxvm:vxconfigd: [ID 702911 daemon.notice] V-5-1-9576 Split Brain. da id is 0.1, while dm id is 0.0 for dm myapp-dg01
Apr  2 14:21:10 servername last message repeated 1 time
Apr  2 14:21:10 servername vxvm:vxconfigd: [ID 702911 daemon.notice] V-5-1-9576 Split Brain. da id is 0.1, while dm id is 0.0 for dm myapp-dg02
Apr  2 14:21:10 servername vxvm:vxconfigd: [ID 702911 daemon.error] V-5-1-569 Disk group myapp-dg, Disk c2t2d0s2: Cannot auto-import group:
Apr  2 14:21:10 servername vxvm:vxconfigd: [ID 702911 daemon.error]        Serial Split Brain detected. Run vxsplitlines to import the diskgroup

/usr/lib/vxvm/bin/vxsplitlines -g myapp-dg

VxVM vxdg listssbinfo NOTICE V-0-0-0 There are 2 pools

All the disks in the first pool have the same config copies 
All the disks in the second pool may not have the same config copies 

Number of disks in the first pool: 6 
Number of disks in the second pool: 2 

To import the diskgroup with config copy from the first pool issue the command
/usr/sbin/vxdg (-s) -o selectcp=1267475633.69.servername import myapp-dg

To import the diskgroup with config copy from the second pool issue the command
/usr/sbin/vxdg (-s) -o selectcp=1267475616.65.servername import myapp-dg

Verbose : /usr/lib/vxvm/bin/vxsplitlines -g myapp-dg -v

VxVM vxdg listssbinfo NOTICE V-0-0-0. There are 2 pools

All the disks in the first pool have the same config copies 
All the disks in the second pool may not have the same config copies 

To see the configuration copy from a disk issue the command
/etc/vx/diag.d/vxprivutil dumpconfig <private path>

To import the diskgroup with config copy from a disk issue the command
/usr/sbin/vxdg (-s) -o selectcp=<diskid> import myapp-dg

                                Pool 0
DEVICE          DISK         DISK ID                      DISK PRIVATE PATH        
myapp-dg08     c2t10d0s2    1267475633.69.servername        /dev/vx/rdmp/c2t10d0s2   
myapp-dg07     c2t9d0s2     1267475633.69.servername        /dev/vx/rdmp/c2t9d0s2    
myapp-dg06     c2t8d0s2     1267475633.69.servername        /dev/vx/rdmp/c2t8d0s2    
myapp-dg05     c2t2d0s2     1267475633.69.servername        /dev/vx/rdmp/c2t2d0s2    
myapp-dg04     c2t1d0s2     1267475633.69.servername        /dev/vx/rdmp/c2t1d0s2    
myapp-dg03     c2t0d0s2     1267475633.69.servername        /dev/vx/rdmp/c2t0d0s2    


                                Pool 1
DEVICE          DISK         DISK ID                      DISK PRIVATE PATH        
myapp-dg02     c1t3d0s2     1267475616.65.servername        /dev/vx/rdmp/c1t3d0s2    
myapp-dg01     c1t2d0s2     1267475608.63.servername        /dev/vx/rdmp/c1t2d0s2

/usr/sbin/vxdg -s -f -o selectcp=1267475624.67.servername import myapp-dg

VxVM vxdg ERROR V-5-1-10978 Disk group myapp-dg: import failed: 
Operation must be executed on master

vxdg import myapp-dg

VxVM vxconfigd NOTICE V-5-1-9576 Split Brain. da id is 0.1, while dm id is 0.0 for dm myapp-dg01
VxVM vxconfigd NOTICE V-5-1-9576 Split Brain. da id is 0.1, while dm id is 0.0 for dm myapp-dg01
VxVM vxconfigd NOTICE V-5-1-9576 Split Brain. da id is 0.1, while dm id is 0.0 for dm myapp-dg02
VxVM vxdg ERROR V-5-1-10978 Disk group myapp-dg: import failed: 
Serial Split Brain detected. Run vxsplitlines to import the diskgroup

vxdisk list

DEVICE       TYPE            DISK         GROUP        STATUS
c1t0d0s2     auto:none       -            -            online invalid
c1t1d0s2     auto:cdsdisk    -            -            online
c1t2d0s2     auto:cdsdisk    -            -            online
c1t3d0s2     auto:cdsdisk    -            -            online
c2t0d0s2     auto:cdsdisk    -            -            online
c2t1d0s2     auto:cdsdisk    -            -            online
c2t2d0s2     auto:cdsdisk    -            -            online
c2t8d0s2     auto:cdsdisk    -            -            online
c2t9d0s2     auto:cdsdisk    -            -            online
c2t10d0s2    auto:cdsdisk    -            -            online

Except the first 2 disks, every disk was part of raid 5 volume and 1 disk as spare and 1 for log.

Any idea how I can recover this?

Thanks in advance.

Edit 1
More Info :

vxconfigd -k

VxVM vxconfigd NOTICE V-5-1-9576 Split Brain. da id is 0.1, while dm id is 0.0 for dm myapp-dg01
VxVM vxconfigd NOTICE V-5-1-9576 Split Brain. da id is 0.1, while dm id is 0.0 for dm myapp-dg01
VxVM vxconfigd NOTICE V-5-1-9576 Split Brain. da id is 0.1, while dm id is 0.0 for dm myapp-dg02
VxVM vxconfigd ERROR V-5-1-569 Disk group myapp-dg, Disk c2t2d0s2: Cannot auto-import group:
        Serial Split Brain detected. Run vxsplitlines to import the diskgroup

/etc/vx/diag.d/vxprivutil dumpconfig /dev/vx/rdmp/c2t2d0s2 
VxVM vxprivutil ERROR V-5-1-595 Disk has no enabled config copies

reborg · April 2, 2010, 7:01pm

What kind of setup is this? A cluster or have you somehow connected two non-clustered nodes to the same storage?

---------- Post updated 2010-04-03 at 12:01 AM ---------- Previous update was 2010-04-02 at 11:58 PM ----------

Also can you post the output of:

vxdg -o alldgs list

upengan78 · April 2, 2010, 7:15pm

reborg:

What kind of setup is this? A cluster or have you somehow connected two non-clustered nodes to the same storage?

---------- Post updated 2010-04-03 at 12:01 AM ---------- Previous update was 2010-04-02 at 11:58 PM ----------

Also can you post the output of:
vxdg -o alldgs list

Hello,

Thanks for reply.

This is a single machine (v440) connected via scsi cable to only ESM on storedge D2. Other scsi port is terminated using LVD/SE terminator. No cluster software on the v440 as well as no such intention.

 vxdg -o alldgs list
VxVM vxdg ERROR V-5-1-776 Option alldgs not recognized, recognized options are:
        failfast - set the failfast timeout for the disk (msec)
        verify - verifies that the specified disk group task can be performed
                without violating any storage-specific guidelines,
                but does not perform the task
        override - performs the  disk  group  task  and  ignores  any
                storage-specific   guidelines
        remove - removes the objects involved in a move/split/join from
                thisdiskgroup
        complete - perform recovery by retrying original operation
        clean - clear any move/split/join-related internal flags
        expand  - select additional objects that reside on same disks
        dgtype - select dg type (private, shared, san)
        clearreserve - clear reservations before import
        groupreserve - import with group reserve
        listreserve - show group reserve information with list command
        ejectkey - eject host membership
        dgid - need to use dgid for the dg to be initialized
        selectcp - Select cp on this disk
        site - Site based import
        noreonline - Don't reonline disks
        coordinator - dg disks reserved as coordinator disks




        site - disk group site name
        tag -  disk group tag name[=value]
        source - disk group tag source
        useclonedev - import clone disks
        updateid - update ids when importing clone disks
        addmirror - Add mirror on the new site
        rmmirror - Remove mirror on the site being removed
        overridessb - Override SSB on reattaching a disk/site
        alldisks - Reattach a site only if all disks are online
        autotagging - Autotaging for newly added disks

reborg · April 2, 2010, 7:18pm

sorry, a lapse in concentration, the command I wanted was:

vxdisk -o alldgs list

upengan78 · April 2, 2010, 7:23pm

No Problem,

vxdisk -o alldgs list

DEVICE       TYPE            DISK         GROUP        STATUS
c1t0d0s2     auto:none       -            -            online invalid
c1t1d0s2     auto:cdsdisk    -            -            online
c1t2d0s2     auto:cdsdisk    -            (myapp-dg)  online
c1t3d0s2     auto:cdsdisk    -            (myapp-dg)  online
c2t0d0s2     auto:cdsdisk    -            (myapp-dg)  online
c2t1d0s2     auto:cdsdisk    -            (myapp-dg)  online
c2t2d0s2     auto:cdsdisk    -            (myapp-dg)  online
c2t8d0s2     auto:cdsdisk    -            (myapp-dg)  online
c2t9d0s2     auto:cdsdisk    -            (myapp-dg)  online
c2t10d0s2    auto:cdsdisk    -            (myapp-dg)  online

reborg · April 2, 2010, 7:42pm

Ok, it doesn't look as bad as it might have been.

Can you also post a dxdisk list for each disk eg:

vxdg list c2t2d0s2

upengan78 · April 2, 2010, 7:55pm

okay, vxdisk list {}

Device:    c1t2d0s2
devicetag: c1t2d0
type:      auto
hostid:    servername
disk:      name= id=1267475608.63.servername
group:     name=myapp-dg id=1267475668.79.servername
info:      format=cdsdisk,privoffset=256,pubslice=2,privslice=2
flags:     online ready private autoconfig autoimport
pubpaths:  block=/dev/vx/dmp/c1t2d0s2 char=/dev/vx/rdmp/c1t2d0s2
guid:      {b132cae8-1dd1-11b2-9c85-0003ba3376dd}
udid:      SEAGATE%5FST373207LSUN72G%5FDISKS%5F303630393334335850480000
site:      -
version:   3.1
iosize:    min=512 (bytes) max=2048 (blocks)
public:    slice=2 offset=65792 len=143283520 disk_offset=0
private:   slice=2 offset=256 len=65536 disk_offset=0
update:    time=1270238982 seqno=0.33
ssb:       actual_seqno=0.1
headers:   0 240
configs:   count=1 len=48144
logs:      count=1 len=7296
Defined regions:
 config   priv 000048-000239[000192]: copy=01 offset=000000 enabled
 config   priv 000256-048207[047952]: copy=01 offset=000192 enabled
 log      priv 048208-055503[007296]: copy=01 offset=000000 enabled
 lockrgn  priv 055504-055647[000144]: part=00 offset=000000
Multipathing information:
numpaths:   1
c1t2d0s2        state=enabled

Device:    c1t3d0s2
devicetag: c1t3d0
type:      auto
hostid:    servername
disk:      name= id=1267475616.65.servername
group:     name=myapp-dg id=1267475668.79.servername
info:      format=cdsdisk,privoffset=256,pubslice=2,privslice=2
flags:     online ready private autoconfig autoimport
pubpaths:  block=/dev/vx/dmp/c1t3d0s2 char=/dev/vx/rdmp/c1t3d0s2
guid:      {b5d6e6b0-1dd1-11b2-9c85-0003ba3376dd}
udid:      SEAGATE%5FST373207LSUN72G%5FDISKS%5F303630393333585242540000
site:      -
version:   3.1
iosize:    min=512 (bytes) max=2048 (blocks)
public:    slice=2 offset=65792 len=143283520 disk_offset=0
private:   slice=2 offset=256 len=65536 disk_offset=0
update:    time=1270238982 seqno=0.32
ssb:       actual_seqno=0.1
headers:   0 240
configs:   count=1 len=48144
logs:      count=1 len=7296
Defined regions:
 config   priv 000048-000239[000192]: copy=01 offset=000000 enabled
 config   priv 000256-048207[047952]: copy=01 offset=000192 enabled
 log      priv 048208-055503[007296]: copy=01 offset=000000 enabled
 lockrgn  priv 055504-055647[000144]: part=00 offset=000000
Multipathing information:
numpaths:   1
c1t3d0s2        state=enabled

Device:    c2t0d0s2
devicetag: c2t0d0
type:      auto
hostid:    servername
disk:      name= id=1267475624.67.servername
group:     name=myapp-dg id=1267475668.79.servername
info:      format=cdsdisk,privoffset=256,pubslice=2,privslice=2
flags:     online ready private autoconfig autoimport
pubpaths:  block=/dev/vx/dmp/c2t0d0s2 char=/dev/vx/rdmp/c2t0d0s2
guid:      {bacffc74-1dd1-11b2-9c85-0003ba3376dd}
udid:      FUJITSU%5FMAP3367N%20SUN36G%5FDISKS%5F00N05W5D%20%20%20%20
site:      -
version:   3.1
iosize:    min=512 (bytes) max=2048 (blocks)
public:    slice=2 offset=65792 len=71061376 disk_offset=0
private:   slice=2 offset=256 len=65536 disk_offset=0
update:    time=1270238982 seqno=0.31
ssb:       actual_seqno=0.0
headers:   0 240
configs:   count=1 len=48144
logs:      count=1 len=7296
Defined regions:
 config   priv 000048-000239[000192]: copy=01 offset=000000 enabled
 config   priv 000256-048207[047952]: copy=01 offset=000192 enabled
 log      priv 048208-055503[007296]: copy=01 offset=000000 enabled
 lockrgn  priv 055504-055647[000144]: part=00 offset=000000
Multipathing information:
numpaths:   1
c2t0d0s2        state=enabled

Device:    c2t1d0s2
devicetag: c2t1d0
type:      auto
hostid:    servername
disk:      name= id=1267475633.69.servername
group:     name=myapp-dg id=1267475668.79.servername
info:      format=cdsdisk,privoffset=256,pubslice=2,privslice=2
flags:     online ready private autoconfig autoimport
pubpaths:  block=/dev/vx/dmp/c2t1d0s2 char=/dev/vx/rdmp/c2t1d0s2
guid:      {c003b6f4-1dd1-11b2-9c85-0003ba3376dd}
udid:      FUJITSU%5FMAP3367N%20SUN36G%5FDISKS%5F00N05W67%20%20%20%20
site:      -
version:   3.1
iosize:    min=512 (bytes) max=2048 (blocks)
public:    slice=2 offset=65792 len=71061376 disk_offset=0
private:   slice=2 offset=256 len=65536 disk_offset=0
update:    time=1270238982 seqno=0.36
ssb:       actual_seqno=0.0
headers:   0 240
configs:   count=1 len=48144
logs:      count=1 len=7296
Defined regions:
 config   priv 000048-000239[000192]: copy=01 offset=000000 enabled
 config   priv 000256-048207[047952]: copy=01 offset=000192 enabled
 log      priv 048208-055503[007296]: copy=01 offset=000000 disabled
 lockrgn  priv 055504-055647[000144]: part=00 offset=000000
Multipathing information:
numpaths:   1
c2t1d0s2        state=enabled

Device:    c2t2d0s2
devicetag: c2t2d0
type:      auto
hostid:    servername
disk:      name= id=1267475642.71.servername
group:     name=myapp-dg id=1267475668.79.servername
info:      format=cdsdisk,privoffset=256,pubslice=2,privslice=2
flags:     online ready private autoconfig autoimport
pubpaths:  block=/dev/vx/dmp/c2t2d0s2 char=/dev/vx/rdmp/c2t2d0s2
guid:      {c531f974-1dd1-11b2-9c85-0003ba3376dd}
udid:      FUJITSU%5FMAP3367N%20SUN36G%5FDISKS%5F00N05W0R%20%20%20%20
site:      -
version:   3.1
iosize:    min=512 (bytes) max=2048 (blocks)
public:    slice=2 offset=65792 len=71061376 disk_offset=0
private:   slice=2 offset=256 len=65536 disk_offset=0
update:    time=1270238982 seqno=0.36
ssb:       actual_seqno=0.0
headers:   0 240
configs:   count=1 len=48144
logs:      count=1 len=7296
Defined regions:
 config   priv 000048-000239[000192]: copy=01 offset=000000 disabled
 config   priv 000256-048207[047952]: copy=01 offset=000192 disabled
 log      priv 048208-055503[007296]: copy=01 offset=000000 enabled
 lockrgn  priv 055504-055647[000144]: part=00 offset=000000
Multipathing information:
numpaths:   1
c2t2d0s2        state=enabled

Device:    c2t8d0s2
devicetag: c2t8d0
type:      auto
hostid:    servername
disk:      name= id=1267475649.73.servername
group:     name=myapp-dg id=1267475668.79.servername
info:      format=cdsdisk,privoffset=256,pubslice=2,privslice=2
flags:     online ready private autoconfig autoimport
pubpaths:  block=/dev/vx/dmp/c2t8d0s2 char=/dev/vx/rdmp/c2t8d0s2
guid:      {ca0db7f8-1dd1-11b2-9c85-0003ba3376dd}
udid:      HITACHI%5FDK32EJ36NSUN36G%5FDISKS%5FHITACHI%20DK32EJ-36NC%20%20%20%20%20435J6553
site:      -
version:   3.1
iosize:    min=512 (bytes) max=2048 (blocks)
public:    slice=2 offset=65792 len=71061376 disk_offset=0
private:   slice=2 offset=256 len=65536 disk_offset=0
update:    time=1270238982 seqno=0.22
ssb:       actual_seqno=0.0
headers:   0 240
configs:   count=1 len=48144
logs:      count=1 len=7296
Defined regions:
 config   priv 000048-000239[000192]: copy=01 offset=000000 enabled
 config   priv 000256-048207[047952]: copy=01 offset=000192 enabled
 log      priv 048208-055503[007296]: copy=01 offset=000000 enabled
 lockrgn  priv 055504-055647[000144]: part=00 offset=000000
Multipathing information:
numpaths:   1
c2t8d0s2        state=enabled

Device:    c2t9d0s2
devicetag: c2t9d0
type:      auto
hostid:    servername
disk:      name= id=1267475658.75.servername
group:     name=myapp-dg id=1267475668.79.servername
info:      format=cdsdisk,privoffset=256,pubslice=2,privslice=2
flags:     online ready private autoconfig autoimport
pubpaths:  block=/dev/vx/dmp/c2t9d0s2 char=/dev/vx/rdmp/c2t9d0s2
guid:      {ced888c6-1dd1-11b2-9c85-0003ba3376dd}
udid:      HITACHI%5FDK32EJ36NSUN36G%5FDISKS%5FHITACHI%20DK32EJ-36NC%20%20%20%20%20435B6497
site:      -
version:   3.1
iosize:    min=512 (bytes) max=2048 (blocks)
public:    slice=2 offset=65792 len=71061376 disk_offset=0
private:   slice=2 offset=256 len=65536 disk_offset=0
update:    time=1270238982 seqno=0.21
ssb:       actual_seqno=0.0
headers:   0 240
configs:   count=1 len=48144
logs:      count=1 len=7296
Defined regions:
 config   priv 000048-000239[000192]: copy=01 offset=000000 enabled
 config   priv 000256-048207[047952]: copy=01 offset=000192 enabled
 log      priv 048208-055503[007296]: copy=01 offset=000000 enabled
 lockrgn  priv 055504-055647[000144]: part=00 offset=000000
Multipathing information:
numpaths:   1
c2t9d0s2        state=enabled

Device:    c2t10d0s2
devicetag: c2t10d0
type:      auto
hostid:    servername
disk:      name= id=1267475665.77.servername
group:     name=myapp-dg id=1267475668.79.servername
info:      format=cdsdisk,privoffset=256,pubslice=2,privslice=2
flags:     online ready private autoconfig autoimport
pubpaths:  block=/dev/vx/dmp/c2t10d0s2 char=/dev/vx/rdmp/c2t10d0s2
guid:      {d394af98-1dd1-11b2-9c85-0003ba3376dd}
udid:      HITACHI%5FDK32EJ36NSUN36G%5FDISKS%5FHITACHI%20DK32EJ-36NC%20%20%20%20%2043536259
site:      -
version:   3.1
iosize:    min=512 (bytes) max=2048 (blocks)
public:    slice=2 offset=65792 len=71061376 disk_offset=0
private:   slice=2 offset=256 len=65536 disk_offset=0
update:    time=1270238982 seqno=0.26
ssb:       actual_seqno=0.0
headers:   0 240
configs:   count=1 len=48144
logs:      count=1 len=7296
Defined regions:
 config   priv 000048-000239[000192]: copy=01 offset=000000 enabled
 config   priv 000256-048207[047952]: copy=01 offset=000192 enabled
 log      priv 048208-055503[007296]: copy=01 offset=000000 enabled
 lockrgn  priv 055504-055647[000144]: part=00 offset=000000
Multipathing information:
numpaths:   1
c2t10d0s2       state=enabled

reborg · April 2, 2010, 7:58pm

Actually what you have already provided looks good enough. It looks like you lost conatct to these disks:

myapp-dg02     c1t3d0s2     1267475616.65.servername        /dev/vx/rdmp/c1t3d0s2    
myapp-dg01     c1t2d0s2     1267475608.63.servername        /dev/vx/rdmp/c1t2d0s2

Which I am guessing are internal disks judging by the naming. I would guess there was something ongoing and you didn't reboot with 'init 6'.

This should work for you:

/usr/sbin/vxdg -Cf -o selectcp=1267475633.69.servername import myapp-dg

upengan78 · April 2, 2010, 8:07pm

Actually I did init 6 only but while booting system was showing lot of scsi disconnect errors or similar scsi messges when I decided to send break but that did not work so powered off from sc and rebooted again by reinserting scsi cable again.

Your command actually worked successfully,

/usr/sbin/vxdg -Cf -o selectcp=1267475633.69.servername import myapp-dg

Now,
-bash-3.00# vxdisk list

DEVICE       TYPE            DISK         GROUP        STATUS
c1t0d0s2     auto:none       -            -            online invalid
c1t1d0s2     auto:cdsdisk    -            -            online
c1t2d0s2     auto:cdsdisk    myapp-dg01  myapp-dg    online spare
c1t3d0s2     auto:cdsdisk    myapp-dg02  myapp-dg    online
c2t0d0s2     auto:cdsdisk    myapp-dg03  myapp-dg    online
c2t1d0s2     auto:cdsdisk    myapp-dg04  myapp-dg    online
c2t2d0s2     auto:cdsdisk    myapp-dg05  myapp-dg    online
c2t8d0s2     auto:cdsdisk    myapp-dg06  myapp-dg    online
c2t9d0s2     auto:cdsdisk    myapp-dg07  myapp-dg    online failing
c2t10d0s2    auto:cdsdisk    myapp-dg08  myapp-dg    online failing

-bash-3.00# vxprint -htr

Disk group: myapp-dg

DG NAME         NCONFIG      NLOG     MINORS   GROUP-ID
ST NAME         STATE        DM_CNT   SPARE_CNT         APPVOL_CNT
DM NAME         DEVICE       TYPE     PRIVLEN  PUBLEN   STATE
RV NAME         RLINK_CNT    KSTATE   STATE    PRIMARY  DATAVOLS  SRL
RL NAME         RVG          KSTATE   STATE    REM_HOST REM_DG    REM_RLNK
CO NAME         CACHEVOL     KSTATE   STATE
VT NAME         RVG          KSTATE   STATE    NVOLUME
V  NAME         RVG/VSET/CO  KSTATE   STATE    LENGTH   READPOL   PREFPLEX UTYPE
PL NAME         VOLUME       KSTATE   STATE    LENGTH   LAYOUT    NCOL/WID MODE
SD NAME         PLEX         DISK     DISKOFFS LENGTH   [COL/]OFF DEVICE   MODE
SV NAME         PLEX         VOLNAME  NVOLLAYR LENGTH   [COL/]OFF AM/NM    MODE
SC NAME         PLEX         CACHE    DISKOFFS LENGTH   [COL/]OFF DEVICE   MODE
DC NAME         PARENTVOL    LOGVOL
SP NAME         SNAPVOL      DCO
EX NAME         ASSOC        VC                       PERMS    MODE     STATE
SR NAME         KSTATE

dg myapp-dg    default      default  95000    1267475668.79.servername

dm myapp-dg01  c1t2d0s2     auto     65536    143283520 SPARE
dm myapp-dg02  c1t3d0s2     auto     65536    143283520 -
dm myapp-dg03  c2t0d0s2     auto     65536    71061376 -
dm myapp-dg04  c2t1d0s2     auto     65536    71061376 -
dm myapp-dg05  c2t2d0s2     auto     65536    71061376 -
dm myapp-dg06  c2t8d0s2     auto     65536    71061376 -
dm myapp-dg07  c2t9d0s2     auto     65536    71061376 FAILING
dm myapp-dg08  c2t10d0s2    auto     65536    71061376 FAILING

v  myapp-vxvol -            DISABLED ACTIVE   355305600 RAID     -        raid5
pl myapp-vxvol-01 myapp-vxvol DISABLED ACTIVE 355305600 RAID    6/32     RW
sd myapp-dg02-01 myapp-vxvol-01 myapp-dg02 0 71061120 0/0      c1t3d0   ENA
sd myapp-dg03-01 myapp-vxvol-01 myapp-dg03 0 71061120 1/0      c2t0d0   FAIL
sd myapp-dg04-01 myapp-vxvol-01 myapp-dg04 0 71061120 2/0      c2t1d0   FAIL
sd myapp-dg05-01 myapp-vxvol-01 myapp-dg05 0 71061120 3/0      c2t2d0   FAIL
sd myapp-dg06-01 myapp-vxvol-01 myapp-dg06 0 71061120 4/0      c2t8d0   FAIL
sd myapp-dg07-01 myapp-vxvol-01 myapp-dg07 0 71061120 5/0      c2t9d0   ENA
pl myapp-vxvol-03 myapp-vxvol DISABLED LOG   5760     CONCAT    -        RW
sd myapp-dg01-01 myapp-vxvol-03 myapp-dg01 0 5760    0         c1t2d0   ENA

reborg · April 2, 2010, 8:17pm

Ok, that makes sense for the failure but it looks like you internal disk controller could be on the way out.

You now need to clear the failing state on these disks since you know what caused them.

vxedit -g myapp set failing=off myapp-dg07
vxedit -g myapp set failing=off myapp-dg08

then start the volumes:

vxvol -g myapp-dg startall

upengan78 · April 2, 2010, 8:25pm

reborg:

Ok, that makes sense for the failure but it looks like you internal disk controller could be on the way out.

You now need to clear the failing state on these disks since you know what caused them.
vxedit -g myapp set failing=off myapp-dg07
vxedit -g myapp set failing=off myapp-dg08
then start the volumes:
vxvol -g myapp-dg startall 

Thanks for helping me out here . appreciate it.

Failing flag on disks is cleared.

Got below message after vxvol start command,

VxVM vxvol ERROR V-5-1-10127 changing volume myapp-vxvol:
        Raid volume would be unusable

vxprint -htr

vxprint -htr
Disk group: myapp-dg

DG NAME         NCONFIG      NLOG     MINORS   GROUP-ID
ST NAME         STATE        DM_CNT   SPARE_CNT         APPVOL_CNT
DM NAME         DEVICE       TYPE     PRIVLEN  PUBLEN   STATE
RV NAME         RLINK_CNT    KSTATE   STATE    PRIMARY  DATAVOLS  SRL
RL NAME         RVG          KSTATE   STATE    REM_HOST REM_DG    REM_RLNK
CO NAME         CACHEVOL     KSTATE   STATE
VT NAME         RVG          KSTATE   STATE    NVOLUME
V  NAME         RVG/VSET/CO  KSTATE   STATE    LENGTH   READPOL   PREFPLEX UTYPE
PL NAME         VOLUME       KSTATE   STATE    LENGTH   LAYOUT    NCOL/WID MODE
SD NAME         PLEX         DISK     DISKOFFS LENGTH   [COL/]OFF DEVICE   MODE
SV NAME         PLEX         VOLNAME  NVOLLAYR LENGTH   [COL/]OFF AM/NM    MODE
SC NAME         PLEX         CACHE    DISKOFFS LENGTH   [COL/]OFF DEVICE   MODE
DC NAME         PARENTVOL    LOGVOL
SP NAME         SNAPVOL      DCO
EX NAME         ASSOC        VC                       PERMS    MODE     STATE
SR NAME         KSTATE

dg myapp-dg    default      default  95000    1267475668.79.servername

dm myapp-dg01  c1t2d0s2     auto     65536    143283520 SPARE
dm myapp-dg02  c1t3d0s2     auto     65536    143283520 -
dm myapp-dg03  c2t0d0s2     auto     65536    71061376 -
dm myapp-dg04  c2t1d0s2     auto     65536    71061376 -
dm myapp-dg05  c2t2d0s2     auto     65536    71061376 -
dm myapp-dg06  c2t8d0s2     auto     65536    71061376 -
dm myapp-dg07  c2t9d0s2     auto     65536    71061376 -
dm myapp-dg08  c2t10d0s2    auto     65536    71061376 -

v  myapp-vxvol -            DETACHED CLEAN    355305600 RAID     -        raid5
pl myapp-vxvol-01 myapp-vxvol ENABLED(SPARSE) ACTIVE 355305600 RAID 6/32 RW
sd myapp-dg02-01 myapp-vxvol-01 myapp-dg02 0 71061120 0/0      c1t3d0   ENA
sd myapp-dg03-01 myapp-vxvol-01 myapp-dg03 0 71061120 1/0      c2t0d0   RCOV
sd myapp-dg04-01 myapp-vxvol-01 myapp-dg04 0 71061120 2/0      c2t1d0   RCOV
sd myapp-dg05-01 myapp-vxvol-01 myapp-dg05 0 71061120 3/0      c2t2d0   RCOV
sd myapp-dg06-01 myapp-vxvol-01 myapp-dg06 0 71061120 4/0      c2t8d0   RCOV
sd myapp-dg07-01 myapp-vxvol-01 myapp-dg07 0 71061120 5/0      c2t9d0   ENA
pl myapp-vxvol-03 myapp-vxvol ENABLED LOG    5760     CONCAT    -        RW
sd myapp-dg01-01 myapp-vxvol-03 myapp-dg01 0 5760    0         c1t2d0   ENA

And, you could be right about scsi controller. but I think both storedge as well as internal disks use same onboard controller?

reborg · April 2, 2010, 8:27pm

Looks like vxrecover kicked in correctly.

What does

vxtask list

show?

upengan78 · April 2, 2010, 8:28pm

vxtask list

TASKID  PTID TYPE/STATE    PCT   PROGRESS
-bash-3.00#

reborg · April 2, 2010, 8:38pm

ok, that shows nothing but all the subdisks are recoverable so that is good.

First try the safest option:

vxrecover -g myapp-dg myapp-vxvol

upengan78 · April 2, 2010, 8:40pm

Ok, ran that command,

VxVM vxvol ERROR V-5-1-1367 Volume myapp-vxvol: not ENABLED, cannot recover subdisks

reborg · April 2, 2010, 8:42pm

I changed my mind on the command to use knowing that one would fail, see previous post(updated)

upengan78 · April 2, 2010, 8:47pm

vxrecover -g myapp-dg myapp-vxvol

went without any error but the output of vxprint -htr is same in my earlier post where I posted vxprint -htr

---------- Post updated at 07:47 PM ---------- Previous update was at 07:46 PM ----------

And, no task in vxtask list this time as well.

reborg · April 2, 2010, 8:48pm

That's ok.

vxvol -g myapp-dg -o delayrecover start myapp-vxvol

upengan78 · April 2, 2010, 8:50pm

O/p for last command is,

VxVM vxvol ERROR V-5-1-1236 Volume myapp-vxvol is not startable; Raid5 plex does not map the entire volume length

reborg · April 2, 2010, 9:04pm

ok, give me a few minutes to go through the current states.

---------- Post updated at 02:04 AM ---------- Previous update was at 02:01 AM ----------

Can I see the output of:

vxrecover -V