zpool status shows things NOT OK, but 3rd party raid says all is well

UNIX Solaris
1

Hi,
I've gone around with this on Oracle's site (and tech support) and ended up empty handed and without ideas of what to do to fix the problem.
Background:
V245, Solaris 10, has 2 12-disk infortrend RAIDs attached.
Have replaced faulty disks many times - familiar with the routine. However, this
time didn't go as routine, and didn't go as "replacement disk was faulty" - or anything else that would be normal and logical. This time, things went haywire.
Now, the RAID software says everything is OK - LUNs, logical disks, etc - no errors, no red lights, nothing in its log...
But,

idadcc# zpool status -v dp1
  pool: dp1
 state: DEGRADED
status: One or more devices has experienced an error resulting in data
        corruption.  Applications may be affected.
action: Restore the file in question if possible.  Otherwise restore the
        entire pool from backup.
   see: 
 scrub: scrub completed after 14h14m with 65 errors on Thu Apr  5 05:25:13 2012
config:

        NAME           STATE     READ WRITE CKSUM
        dp1            DEGRADED     0     0   154
          raidz1-0     DEGRADED     0     0   308
            spare-0    DEGRADED     0     0     0
              c2t0d0   DEGRADED     0     0     0  too many errors
              c2t0d11  ONLINE       0     0     0
            spare-1    DEGRADED     0     0     0
              c2t0d1   DEGRADED     0     0     0  too many errors
              c2t0d10  ONLINE       0     0     0
            c2t0d2     DEGRADED     0     0     0  too many errors
            c2t0d3     DEGRADED     0     0     0  too many errors
            c2t0d4     DEGRADED     0     0     0  too many errors
          raidz1-1     ONLINE       0     0     0
            c2t0d5     ONLINE       0     0     0
            c2t0d6     ONLINE       0     0     0
            c2t0d7     ONLINE       0     0     0
            c2t0d8     ONLINE       0     0     0
            c2t0d9     ONLINE       0     0     0
        spares
          c2t0d10      INUSE     currently in use
          c2t0d11      INUSE     currently in use

errors: Permanent errors have been detected in the following files...

(and a list of -- files that are actually my snapshots)....

I've done the "clear" and the "scrub" - all the usual tricks. Same result.

idadcc# echo | format
Searching for disks...done


AVAILABLE DISK SELECTIONS:
       0. c0t0d0 <SUN72G cyl 14087 alt 2 hd 24 sec 424>
          /pci@1e,600000/pci@0/pci@a/pci@0/pci@8/scsi@1/sd@0,0
       1. c0t1d0 <SUN72G cyl 14087 alt 2 hd 24 sec 424>
          /pci@1e,600000/pci@0/pci@a/pci@0/pci@8/scsi@1/sd@1,0
       2. c0t2d0 <SUN72G cyl 14087 alt 2 hd 24 sec 424>
          /pci@1e,600000/pci@0/pci@a/pci@0/pci@8/scsi@1/sd@2,0
       3. c0t3d0 <SUN72G cyl 14087 alt 2 hd 24 sec 424>
          /pci@1e,600000/pci@0/pci@a/pci@0/pci@8/scsi@1/sd@3,0
       4. c1t0d0 <IFT-A12U-G2421-347R-931.26GB>
          /pci@1f,700000/pci@0/LSILogic,scsi@2/sd@0,0
       5. c1t0d1 <IFT-A12U-G2421-347R-931.26GB>
          /pci@1f,700000/pci@0/LSILogic,scsi@2/sd@0,1
       6. c1t0d2 <IFT-A12U-G2421-347R-931.26GB>
          /pci@1f,700000/pci@0/LSILogic,scsi@2/sd@0,2
       7. c1t0d3 <IFT-A12U-G2421-347R-931.26GB>
          /pci@1f,700000/pci@0/LSILogic,scsi@2/sd@0,3
       8. c1t0d4 <IFT-A12U-G2421-347R-931.26GB>
          /pci@1f,700000/pci@0/LSILogic,scsi@2/sd@0,4
       9. c1t0d5 <IFT-A12U-G2421-347R-931.26GB>
          /pci@1f,700000/pci@0/LSILogic,scsi@2/sd@0,5
      10. c1t0d6 <IFT-A12U-G2421-347R-931.26GB>
          /pci@1f,700000/pci@0/LSILogic,scsi@2/sd@0,6
      11. c1t0d7 <IFT-A12U-G2421-347R-931.26GB>
          /pci@1f,700000/pci@0/LSILogic,scsi@2/sd@0,7
      12. c1t0d8 <IFT-A12U-G2421-347R-931.26GB>
          /pci@1f,700000/pci@0/LSILogic,scsi@2/sd@0,8
      13. c1t0d9 <IFT-A12U-G2421-347R-931.26GB>
          /pci@1f,700000/pci@0/LSILogic,scsi@2/sd@0,9
      14. c1t0d10 <IFT-A12U-G2421-347R-931.26GB>
          /pci@1f,700000/pci@0/LSILogic,scsi@2/sd@0,a
      15. c1t0d11 <IFT-A12U-G2421-347R-931.26GB>
          /pci@1f,700000/pci@0/LSILogic,scsi@2/sd@0,b
      16. c2t0d0 <IFT-A12U-G2421-347R-931.26GB>
          /pci@1f,700000/pci@0,2/scsi@1,1/sd@0,0
      17. c2t0d1 <IFT-A12U-G2421-347R-931.26GB>
          /pci@1f,700000/pci@0,2/scsi@1,1/sd@0,1
      18. c2t0d2 <IFT-A12U-G2421-347R-931.26GB>
          /pci@1f,700000/pci@0,2/scsi@1,1/sd@0,2
      19. c2t0d3 <IFT-A12U-G2421-347R-931.26GB>
          /pci@1f,700000/pci@0,2/scsi@1,1/sd@0,3
      20. c2t0d4 <IFT-A12U-G2421-347R-931.26GB>
          /pci@1f,700000/pci@0,2/scsi@1,1/sd@0,4
      21. c2t0d5 <IFT-A12U-G2421-347R-931.26GB>
          /pci@1f,700000/pci@0,2/scsi@1,1/sd@0,5
      22. c2t0d6 <IFT-A12U-G2421-347R-931.26GB>
          /pci@1f,700000/pci@0,2/scsi@1,1/sd@0,6
      23. c2t0d7 <IFT-A12U-G2421-347R-931.26GB>
          /pci@1f,700000/pci@0,2/scsi@1,1/sd@0,7
      24. c2t0d8 <IFT-A12U-G2421-347R-931.26GB>
          /pci@1f,700000/pci@0,2/scsi@1,1/sd@0,8
      25. c2t0d9 <IFT-A12U-G2421-347R-931.26GB>
          /pci@1f,700000/pci@0,2/scsi@1,1/sd@0,9
      26. c2t0d10 <IFT-A12U-G2421-347R-931.26GB>
          /pci@1f,700000/pci@0,2/scsi@1,1/sd@0,a
      27. c2t0d11 <IFT-A12U-G2421-347R-931.26GB>
          /pci@1f,700000/pci@0,2/scsi@1,1/sd@0,b

FWIW, it was the slot 1 disk that I replaced. I even replaced it with a different disk. I've gone through the delete/remake LUN/logical drive on the RAID several times. The raid looks happy. Solaris doesn't.
ANY suggestions welcome. Suggestions on Solaris-friendly vendors of RAIDs welcome - I'd like to get a system that doesn't end up with finger-pointing to "the other guy" whenever I have questions that I can't find answers to!
Thank you!!!

2

Here is why you're stuck between oracle zfs and the vendor:

Should ZFS Have a fsck Tool?

It doesn't help your situation. I have no good answer because zfs is SUPPOSED to be self healing. Thers is only zfs scrub, which I guess you've tried. It does sound like whatever the SAN software does to replicate/snapshot a lun had some issues.

3

An fsck would be of no use. The file systems are mountable here. Whatever the file system, fsck is a tool that fixes the structure, not the file content which is the issue here.
I would do a snapshot to lock the faulty blocks and restore the problem files (and only them) from a backup.

4

zpool clear

5

as stated in the first post, that doesn't work!

6

you can check the disk errors with iostat -En

Even your controller says everything is ok... it could be wrong.

Juan