Hi All,
I am getting zpool corrupted message under zpool status command.What could be the reason for this.I had observed this zpool was full this morning after this i am seeing this error.
[Sun1 root ] $PWD>zpool status -xv R5
pool: R5
state: DEGRADED
status: One or more devices has experienced an error resulting in data
corruption. Applications may be affected.
action: Restore the file in question if possible. Otherwise restore the
entire pool from backup.
see: http://www.sun.com/msg/ZFS-8000-8A
scan: scrub in progress since Mon May 6 06:21:11 2013
358G scanned out of 1.26T at 27.8M/s, 9h30m to go
358G scanned out of 1.26T at 27.8M/s, 9h30m to go
0 repaired, 27.81% done
config:
NAME STATE READ WRITE CKSUM
R5 DEGRADED 0 0 3.56M
raidz1-0 DEGRADED 0 0 21.1M
c3t600A0B80006872FF000003104B8BE555d0 DEGRADED 0 0 0 too many errors
c3t600A0B80006872FF000003134B8BE5C3d0 DEGRADED 0 0 0 too many errors
c3t600A0B800068729A000002DC4B8BE524d0 DEGRADED 0 0 0 too many errors
c3t600A0B800068729A000002DF4B8BE59Ed0 DEGRADED 0 0 0 too many errors
c3t600A0B800068729A000002E24B8BE618d0 DEGRADED 0 0 0 too many errors
errors: Permanent errors have been detected in the following files:
<metadata>:<0x37>
<metadata>:<0x38>
<metadata>:<0xa5>
<metadata>:<0x10c>
<metadata>:<0x192>
R5/zones:<0x0>
R5/cat-test-db/projects:<0x0>
R5/cat-trn-db/export:<0x0>
R5/cat-trn-oas/export:<0x0>
/R5/cat-11gr2-db/app/oracle/oradata/testdb2/datafile/undotbs1_01.dbf
R5/was801/export:<0x0>
/R5/oas11g-eval/export/app/oracle/weblogic/10.3.5/user_projects/domains/MST/servers/AdminServer/logs/access.log
<0x289>:<0x0>
I had a look to check if any of disk has failed.But nothing seems to appear under format/iostat commands.
Format
Searching for disks...done
AVAILABLE DISK SELECTIONS:
0. c0t0d0 <SUN72G cyl 14087 alt 2 hd 24 sec 424>
/pci@0,600000/pci@0/pci@8/pci@0/scsi@1/sd@0,0
1. c0t1d0 <SUN72G cyl 14087 alt 2 hd 24 sec 424>
/pci@0,600000/pci@0/pci@8/pci@0/scsi@1/sd@1,0
2. c3t600A0B80006872FF000003104B8BE555d0 <SUN-LCSM100_F-0735-278.90GB>
/scsi_vhci/ssd@g600a0b80006872ff000003104b8be555
3. c3t600A0B80006872FF000003134B8BE5C3d0 <SUN-LCSM100_F-0735-278.90GB>
/scsi_vhci/ssd@g600a0b80006872ff000003134b8be5c3
4. c3t600A0B800068729A000002AB4B87E71Dd0 <SUN-LCSM100_F-0735 cyl 65533 alt 2 hd 128 sec 139>
/scsi_vhci/ssd@g600a0b800068729a000002ab4b87e71d
5. c3t600A0B800068729A000002DC4B8BE524d0 <SUN-LCSM100_F-0735-278.90GB>
/scsi_vhci/ssd@g600a0b800068729a000002dc4b8be524
6. c3t600A0B800068729A000002DF4B8BE59Ed0 <SUN-LCSM100_F-0735-278.90GB>
/scsi_vhci/ssd@g600a0b800068729a000002df4b8be59e
7. c3t600A0B800068729A000002E24B8BE618d0 <SUN-LCSM100_F-0735-278.90GB>
/scsi_vhci/ssd@g600a0b800068729a000002e24b8be618
8. c3t600000E00D0000000001277400020000d0 <FUJITSU-ETERNUS_DXL-0000-1.79TB>
/scsi_vhci/ssd@g600000e00d0000000001277400020000
9. c3t600000E00D0000000001277400000000d0 <FUJITSU-ETERNUS_DXL-0000-272.50GB>
/scsi_vhci/ssd@g600000e00d0000000001277400000000
10. c3t600000E00D0000000001277400010000d0 <FUJITSU-ETERNUS_DXL-0000-1.86TB>
/scsi_vhci/ssd@g600000e00d0000000001277400010000
11. c6t2d31 <SUN-UniversalXport-0735 cyl 8 alt 2 hd 64 sec 64>
/pci@0,600000/pci@0/pci@9/SUNW,emlxs@0,1/fp@0,0/ssd@w202500a0b86872ff,1f
12. c7t0d31 <SUN-UniversalXport-0735 cyl 8 alt 2 hd 64 sec 64>
/pci@0,600000/pci@0/pci@9/SUNW,emlxs@0/fp@0,0/ssd@w202400a0b86872ff,1f
Specify disk (enter its number): Specify disk (enter its number):
Iostat output
c0t1d0 Soft Errors: 0 Hard Errors: 0 Transport Errors: 0
Vendor: SEAGATE Product: ST973402SSUN72G Revision: 0603 Serial No: 080522G5KP
Size: 73.41GB <73407865856 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0
Illegal Request: 3 Predictive Failure Analysis: 0
c0t0d0 Soft Errors: 0 Hard Errors: 0 Transport Errors: 0
Vendor: SEAGATE Product: ST973402SSUN72G Revision: 0603 Serial No: 080522FVZH
Size: 73.41GB <73407865856 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0
Illegal Request: 3 Predictive Failure Analysis: 0
c0t3d0 Soft Errors: 0 Hard Errors: 0 Transport Errors: 0
Vendor: TSSTcorp Product: CD/DVDW TS-L632D Revision: SR02 Serial No:
Size: 0.00GB <0 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0
Illegal Request: 2 Predictive Failure Analysis: 0
c6t2d31 Soft Errors: 0 Hard Errors: 0 Transport Errors: 0
Vendor: SUN Product: Universal Xport Revision: 0735 Serial No:
Size: 0.02GB <20971520 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0
Illegal Request: 53 Predictive Failure Analysis: 0
c7t0d31 Soft Errors: 0 Hard Errors: 1 Transport Errors: 0
Vendor: SUN Product: Universal Xport Revision: 0735 Serial No:
Size: 0.02GB <20971520 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0
Illegal Request: 5 Predictive Failure Analysis: 0
c3t600A0B800068729A000002AB4B87E71Dd0 Soft Errors: 0 Hard Errors: 0 Transport Errors: 0
Vendor: SUN Product: LCSM100_F Revision: 0735 Serial No:
Size: 598.93GB <598925639680 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0
Illegal Request: 2 Predictive Failure Analysis: 0
c3t600A0B800068729A000002E24B8BE618d0 Soft Errors: 0 Hard Errors: 0 Transport Errors: 0
Vendor: SUN Product: LCSM100_F Revision: 0735 Serial No:
Size: 299.46GB <299462819840 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0
Illegal Request: 2 Predictive Failure Analysis: 0
c3t600A0B80006872FF000003134B8BE5C3d0 Soft Errors: 0 Hard Errors: 0 Transport Errors: 0
Vendor: SUN Product: LCSM100_F Revision: 0735 Serial No:
Size: 299.46GB <299462819840 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0
Illegal Request: 0 Predictive Failure Analysis: 0
c3t600A0B800068729A000002DF4B8BE59Ed0 Soft Errors: 0 Hard Errors: 0 Transport Errors: 0
Vendor: SUN Product: LCSM100_F Revision: 0735 Serial No:
Size: 299.46GB <299462819840 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0
Illegal Request: 2 Predictive Failure Analysis: 0
c3t600A0B80006872FF000003104B8BE555d0 Soft Errors: 0 Hard Errors: 0 Transport Errors: 0
Vendor: SUN Product: LCSM100_F Revision: 0735 Serial No:
Size: 299.46GB <299462819840 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0
Illegal Request: 0 Predictive Failure Analysis: 0
c3t600A0B800068729A000002DC4B8BE524d0 Soft Errors: 0 Hard Errors: 0 Transport Errors: 0
Vendor: SUN Product: LCSM100_F Revision: 0735 Serial No:
Size: 299.46GB <299462819840 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0
Illegal Request: 2 Predictive Failure Analysis: 0
c3t600000E00D0000000001277400000000d0 Soft Errors: 0 Hard Errors: 0 Transport Errors: 0
Vendor: FUJITSU Product: ETERNUS_DXL Revision: 0000 Serial No:
Size: 292.59GB <292594647040 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0
Illegal Request: 1 Predictive Failure Analysis: 0
c3t600000E00D0000000001277400010000d0 Soft Errors: 0 Hard Errors: 0 Transport Errors: 0
Vendor: FUJITSU Product: ETERNUS_DXL Revision: 0000 Serial No:
Size: 2048.16GB <2048162529280 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0
Illegal Request: 1 Predictive Failure Analysis: 0
c3t600000E00D0000000001277400020000d0 Soft Errors: 0 Hard Errors: 0 Transport Errors: 0
Vendor: FUJITSU Product: ETERNUS_DXL Revision: 0000 Serial No:
Size: 1966.56GB <1966558150656 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0
Illegal Request: 2 Predictive Failure Analysis: 0
rmt/1 Soft Errors: 0 Hard Errors: 0 Transport Errors: 0
Vendor: HP Product: Ultrium 3-SCSI Revision: G57Z Serial No:
rmt/0 Soft Errors: 137 Hard Errors: 46 Transport Errors: 141118
Vendor: HP Product: Ultrium 3-SCSI Revision: G56Z Serial No:
Dmesg logs
May 4 09:10:09 sun mpt_handle_event: IOCLogInfo=0x110a0000
May 4 09:10:09 sun scsi: [ID 243001 kern.info] /pci@3,700000/pci@0/scsi@8 (mpt1):
May 4 09:10:09 sun mpt_handle_event_sync: IOCLogInfo=0x110a0000
May 4 09:10:09 sun scsi: [ID 243001 kern.info] /pci@3,700000/pci@0/scsi@8 (mpt1):
May 4 09:10:09 sun mpt_handle_event: IOCLogInfo=0x110a0000
May 4 09:10:09 sun scsi: [ID 243001 kern.info] /pci@3,700000/pci@0/scsi@8 (mpt1):
May 4 09:10:09 sun mpt_handle_event_sync: IOCLogInfo=0x110a0000
May 4 09:10:09 sun scsi: [ID 243001 kern.info] /pci@3,700000/pci@0/scsi@8 (mpt1):
May 4 09:10:09 sun mpt_handle_event: IOCLogInfo=0x110a0000
May 4 09:10:09 sun scsi: [ID 243001 kern.info] /pci@3,700000/pci@0/scsi@8 (mpt1):
May 4 09:10:09 sun mpt_handle_event_sync: IOCLogInfo=0x110a0000
May 4 09:10:09 sun scsi: [ID 243001 kern.info] /pci@3,700000/pci@0/scsi@8 (mpt1):
May 4 09:10:09 sun mpt_handle_event: IOCLogInfo=0x110a0000
May 4 09:10:09 sun scsi: [ID 243001 kern.info] /pci@3,700000/pci@0/scsi@8 (mpt1):
May 4 09:10:09 sun mpt_handle_event_sync: IOCLogInfo=0x110a0000
May 4 09:10:09 sun scsi: [ID 243001 kern.info] /pci@3,700000/pci@0/scsi@8 (mpt1):