DELL M910 with RedHat Linux 5.5 getting "Buffer I/O error on device sdd"

devyfong · August 14, 2011, 11:49am

Dear Guru,

IHAC who is using Redhat Linux 5 on DELL M910.

When system bootup, it show "Buffer I/O error on device sdd" error message. Below is the "dmesg" output.

The scenario is:
If more external FC SAN disk LUN add, the device would change other name. So it will impact the raw device mapping which they set in "/etc/udev/rules.d/60-raw.rules".

Anybody met this issue before? Any tips for fixing this issue?

Thanks a lot for your help!

Best regards,
David Feng

===================================
#dmesg | grep sdd
SCSI device sdd: 2 512-byte hdwr sectors (0 MB)
sdd: Write Protect is off
sdd: Mode Sense: 97 00 00 08
SCSI device sdd: drive cache: write through
SCSI device sdd: 2 512-byte hdwr sectors (0 MB)
sdd: Write Protect is off
sdd: Mode Sense: 97 00 00 08
SCSI device sdd: drive cache: write through
sdd:[0x000053420001015e][INFO][UP][__mppLnx_scsi_done, 2682][4294689543] UP_done: A0C0P0L0,r=8000002,MPP_CHECK_CONDITION,sk=5,ASC/ASCQ=20/0,SN:13.
end_request: I/O error, dev sdd, sector 0
Buffer I/O error on device sdd, logical block 0
end_request: I/O error, dev sdd, sector 0
Buffer I/O error on device sdd, logical block 0
end_request: I/O error, dev sdd, sector 0
Buffer I/O error on device sdd, logical block 0
end_request: I/O error, dev sdd, sector 0
Buffer I/O error on device sdd, logical block 0
end_request: I/O error, dev sdd, sector 0
Buffer I/O error on device sdd, logical block 0
end_request: I/O error, dev sdd, sector 0
Buffer I/O error on device sdd, logical block 0
end_request: I/O error, dev sdd, sector 0
Buffer I/O error on device sdd, logical block 0
Dev sdd: unable to read RDB block 0
end_request: I/O error, dev sdd, sector 0
Buffer I/O error on device sdd, logical block 0
end_request: I/O error, dev sdd, sector 0
Buffer I/O error on device sdd, logical block 0
sd 8:0:0:0: Attached scsi disk sdd
end_request: I/O error, dev sdd, sector 0
Buffer I/O error on device sdd, logical block 0
end_request: I/O error, dev sdd, sector 0
Buffer I/O error on device sdd, logical block 0
end_request: I/O error, dev sdd, sector 0
Buffer I/O error on device sdd, logical block 0
end_request: I/O error, dev sdd, sector 0
Buffer I/O error on device sdd, logical block 0
end_request: I/O error, dev sdd, sector 0
Buffer I/O error on device sdd, logical block 0
end_request: I/O error, dev sdd, sector 0
end_request: I/O error, dev sdd, sector 0
end_request: I/O error, dev sdd, sector 0
end_request: I/O error, dev sdd, sector 0
end_request: I/O error, dev sdd, sector 0
end_request: I/O error, dev sdd, sector 0
end_request: I/O error, dev sdd, sector 0
end_request: I/O error, dev sdd, sector 0
end_request: I/O error, dev sdd, sector 0
end_request: I/O error, dev sdd, sector 0
end_request: I/O error, dev sdd, sector 0
end_request: I/O error, dev sdd, sector 0
end_request: I/O error, dev sdd, sector 0
end_request: I/O error, dev sdd, sector 0
end_request: I/O error, dev sdd, sector 0
end_request: I/O error, dev sdd, sector 0
end_request: I/O error, dev sdd, sector 0
end_request: I/O error, dev sdd, sector 0
end_request: I/O error, dev sdd, sector 0
end_request: I/O error, dev sdd, sector 0
end_request: I/O error, dev sdd, sector 0
end_request: I/O error, dev sdd, sector 0
end_request: I/O error, dev sdd, sector 0
end_request: I/O error, dev sdd, sector 0
end_request: I/O error, dev sdd, sector 0
end_request: I/O error, dev sdd, sector 0
end_request: I/O error, dev sdd, sector 0
end_request: I/O error, dev sdd, sector 0
end_request: I/O error, dev sdd, sector 0
end_request: I/O error, dev sdd, sector 0
end_request: I/O error, dev sdd, sector 0
Buffer I/O error on device sdd, logical block 0
end_request: I/O error, dev sdd, sector 0
Buffer I/O error on device sdd, logical block 0
end_request: I/O error, dev sdd, sector 0
Buffer I/O error on device sdd, logical block 0
end_request: I/O error, dev sdd, sector 0
Buffer I/O error on device sdd, logical block 0
end_request: I/O error, dev sdd, sector 0
end_request: I/O error, dev sdd, sector 0
end_request: I/O error, dev sdd, sector 0
end_request: I/O error, dev sdd, sector 0
end_request: I/O error, dev sdd, sector 0
end_request: I/O error, dev sdd, sector 0
end_request: I/O error, dev sdd, sector 0
end_request: I/O error, dev sdd, sector 0
end_request: I/O error, dev sdd, sector 0
end_request: I/O error, dev sdd, sector 0
end_request: I/O error, dev sdd, sector 0
end_request: I/O error, dev sdd, sector 0
end_request: I/O error, dev sdd, sector 0
Buffer I/O error on device sdd, logical block 0
end_request: I/O error, dev sdd, sector 0
Buffer I/O error on device sdd, logical block 0
SCSI device sdd: 2 512-byte hdwr sectors (0 MB)
sdd: Write Protect is off
sdd: Mode Sense: 97 00 00 08
SCSI device sdd: drive cache: write through
sdd:[0x000053420001015e][INFO][UP][__mppLnx_scsi_done, 2682][4294757946] UP_done: A0C0P3L0,r=8000002,MPP_CHECK_CONDITION,sk=5,ASC/ASCQ=20/0,SN:1915.
end_request: I/O error, dev sdd, sector 0
Buffer I/O error on device sdd, logical block 0
end_request: I/O error, dev sdd, sector 0
Buffer I/O error on device sdd, logical block 0
end_request: I/O error, dev sdd, sector 0
end_request: I/O error, dev sdd, sector 0
end_request: I/O error, dev sdd, sector 0
end_request: I/O error, dev sdd, sector 0
end_request: I/O error, dev sdd, sector 0
Dev sdd: unable to read RDB block 0
end_request: I/O error, dev sdd, sector 0
end_request: I/O error, dev sdd, sector 0
end_request: I/O error, dev sdd, sector 0
[root@rac1 log]# 001015e][INFO][UP][__mppLnx_scsi_done, 2682][4294757946] UP_done: A0C0P3L0,r=8000002,MPP_CHECK_CONDITION,sk=5,ASC/ASCQ=20/0,SN:1915.
-bash: 001015e][INFO][UP][__mppLnx_scsi_done,: command not found

dude2cool · August 14, 2011, 11:52am

Please post output from:

fdisk -l

and

df -h

and

cat /etc/fstab

devyfong · August 14, 2011, 11:59am

Thanks for quick response!

But I cannot get the output at the moment, would post the output tomorrow morning.

Do you have any work around for me to do some testing since tomorrow morning in my time is your midnight.

Regards,
David Feng

dude2cool · August 14, 2011, 12:06pm

I would hate to take a guess, without looking at the output of those commands. Also, what kind of SAN Storage array do you have and are you using any volume manager? Lvm is the native volume manager or you could use veritas.
Also these forums has always someone who can help in your timezone. Don'tworry about that :). If I am not online,someone else will be able to help.

devyfong · August 14, 2011, 12:15pm

Sure understood! Thanks!

The customer is using the Linux native LVM.
The SAN storage is HuaWei Symantec 5600T.

Will post more detail information tomorrow.

Regards,
David Feng

dude2cool · August 14, 2011, 12:26pm

Another useful output will be from sg3-utils, install these through yum if you are subscribed to RHN.
Once installed, paste output from :

sg_map -i

mark54g · August 15, 2011, 11:25am

Also, are you using multipathing drivers?

devyfong · August 19, 2011, 3:07am

Hi All Friends,

After onsite checked, I found the problem was caused by external storage. There is a virtual device which would not be assigned a LUN to it. When system boot, it will generated the Buffer IO error message.

Thanks all for your support and tips.

The solution is:

Remove the virtual device from the storage configuration tools. rescan the devices.

Problem fixed.

Have a day!

David Feng