Why didn't she panic? (Sol 10 + SVM + HDS)

UNIX Solaris
1

Hi folks,

the following incident occured today:

by mistake one of our renowned administrators deleted the complete zoning for a 25K domain running solaris 10.

Thus the system lost all of it's external disks.

We've got oracle datafiles and oracle software residing on those lost disks.

The system logged read and write errors to /var/adm/messages. But it did not panic, because the write errors were qualified "retryable".

The external disks were mounted as metadevices in metasets.

Does SVM keep the system from panicing?

Background information:

uname -a:
SunOS <servername> 5.10 Generic_127111-11 sun4u sparc SUNW,Sun-Fire-15000

cat /etc/release:
Solaris 10 8/07 s10s_u4wos_12b SPARC
Copyright 2007 Sun Microsystems, Inc. All Rights Reserved.
Use is subject to license terms.
Assembled 16 August 2007

/var/adm/messages:

Nov 13 10:28:47 sv0703 md_stripe: [ID 641072 kern.warning] WARNING: md: 0703m1/d101: write error on /dev/dsk/c6t60060E801526C300000126C300003265d0s0
Nov 13 10:28:47 sv0703 md_stripe: [ID 641072 kern.warning] WARNING: md: 0703m1/d80: write error on /dev/dsk/c6t60060E801526C300000126C300002265d0s0
Nov 13 10:28:47 sv0703 md_sp: [ID 641072 kern.warning] WARNING: md: 0703m1/d101: write error on /dev/md/0703m1/dsk/d90
Nov 13 10:56:09 sv0703 Error for Command: write(10) Error Level: Retryable
...and so on...

df -h /u02:
/dev/md/<metasetname>/dsk/d100 103G 85G 17G 83% /u02

metaset -s <metasetname>
Set name = <metasetname>, Set number = 4
Host Owner
<hostname> Yes (auto)
Drive Dbase
/dev/dsk/c6t60060E801526C300000126C3000022A2d0 Yes
/dev/dsk/c6t60060E801526C300000126C3000032A2d0 Yes
/dev/dsk/c6t50060E80000000000000F8FE000000A2d0 Yes
/dev/dsk/c6t50060E80000000000000F8FE000004A2d0 Yes

For each FS we've got:

submirror-> mirror-> soft partition on metaset

Thanks & Regards

Mika

###

2

in this case i would suspect a bug in your san firmware or the leadville driver stack. this is something to try on another mashine with the same software setup. if the error can be reproduced you can try to add newer patches or a newer san firmware... maybe this fixes your "problem".

greets,
DN2

3

isn't that normal? ... the panic option you can set with mount (onerror=), specifies the action about inconsistency filesystems, but not if the box can't reach the fs... i tried that some times and it did nothing, also saw retries in the log... there is a new option in sun cluster 3.2 called "reboot_on_path_failure" to prevent such issues....

# clnode set -p reboot_on_path_failure=enabled <node1> <node2>

rgds

  • pressy
4

My understanding is, that a system should panic whenever cached data cannot be written to a disk device.

This is why I think the "retryable" qualifier in /var/adm/messages is the culprit. :confused:

Anyhow - thanks for all replies!

Groetjes

Mika

###

5

Absolutely not. A system should panic when it is so confused that attempting to write cached data may cause further damage. Imagine a system with external disks and you bump your knee into a power button, turning off the disk. All you need to do is power the drive back on. And yes, that really happened to me and I was grateful that the HP-UX system did not panic.

6

Perderabo - you are either really tall or that power button is super low!!! :smiley:

7

I was sitting in a chair using the top of the drive cabinet as a desk. There were two drives in it each was 571 MB witha separate power button for each drive. I mentioned this pair of drives before: How to create file of fixed size?