lto4 devices on mpt fusion driver

We are seeing an odd problem on an HP dl380 running Redhat es5 (2.6.18-52+.el5). we have 3 dual channel HBA's with a single tape drive on each channel totaling six tape drives.(2 x lto3, and 4 xlto4). The problem is only a two of the lto-4s are working. At first we thought it might be hardware, but we have swapped everything out with new parts and we still see the same issue.

We have used the sg3_utils package to look at the devices. If you use the sg devices for each, they all can take commands and respond to the sg_senddiag command. Once you start trying to use the /dev/st or /dev/nst the devices the work respond ok, the others give an IO error(EIO). In fact, if you run a sginfo -l you can see that it thinks only some of the devices are tape devices and the two drives that aren't working are just generic scsi devices. sginfo can see they are ultrium drives, but have no /dev/st or /dev/nst associated devices.

Since they are really all the same tape drives, we can't figure out why they aren't all working or all not working. To make matters worse, if you use the Fusion Redhat drivers, each time you boot, the drives that aren't working will be different drives. Upgrading to the latest LSI mpt drivers seemed to have fixed this however. We also along the way completely reinstalled the OS from media and updated it.

The server had been working in the configuration for almost a year. It had a network problem that we rebooted it for, and since then the drives have been in this state. Just really not sure where to look from here.

I am assuming that there might be something wrong in the st.ko module. the fact the /dev/st devices don't work but the sg devices do seems to point to some kind of weird thing going on one layer up. I have been all over the linux scsi forums and have tickets open with RH but have not resolution as yet, so I thought I would post here.

Hope someone else has seen something like this.

thanks,
Bil