Hi all
we have two Rac cluster in our environment. we have asked our storage team to provide luns for adding space to our rac. Right now in one our server luns are visible but not in another. But if i see dmesg in servers i can see disk attached but not able to see them. even iostat showing disks as well, multipath -ll not showing those disk. all online rescan has been done and multipath.conf blacklist has been removed.
env- redhat 5.5 - 2.6.18-371.9.1.el5PAE
lsscsi output -
[3:0:2:27] disk IBM 2107900 1.32 -
[3:0:2:28] disk IBM 2107900 1.32 -
[3:0:2:29] disk IBM 2107900 1.32 -
[3:0:2:30] disk IBM 2107900 1.32 -
-----------------------------------------------------------
[4:0:2:27] disk IBM 2107900 1.32 -
[4:0:2:28] disk IBM 2107900 1.32 -
[4:0:2:29] disk IBM 2107900 1.32 -
[4:0:2:30] disk IBM 2107900 1.32 -
dmesg error
SCSI device sdcz: 285212672 512-byte hdwr sectors (146029 MB)
sdcz: Write Protect is off
sdcz: Mode Sense: ed 00 00 08
SCSI device sdcz: drive cache: write back
sdcz: detected capacity change from 0 to 146028888064
SCSI device sdda: 285212672 512-byte hdwr sectors (146029 MB)
sdda: Write Protect is off
sdda: Mode Sense: ed 00 00 08
SCSI device sdda: drive cache: write back
sdda: detected capacity change from 0 to 146028888064
SCSI device sddb: 285212672 512-byte hdwr sectors (146029 MB)
sddb: Write Protect is off
sddb: Mode Sense: ed 00 00 08
SCSI device sddb: drive cache: write back
sddb: detected capacity change from 0 to 146028888064
SCSI device sddc: 285212672 512-byte hdwr sectors (146029 MB)
sddc: Write Protect is off
sddc: Mode Sense: ed 00 00 08
SCSI device sddc: drive cache: write back
sddc: detected capacity change from 0 to 146028888064
SCSI device sddd: 285212672 512-byte hdwr sectors (146029 MB)
sddd: Write Protect is off
sddd: Mode Sense: ed 00 00 08
SCSI device sddd: drive cache: write back
sddd: detected capacity change from 0 to 146028888064
SCSI device sdde: 285212672 512-byte hdwr sectors (146029 MB)
sdde: Write Protect is off
sdde: Mode Sense: ed 00 00 08
SCSI device sdde: drive cache: write back
sdde: detected capacity change from 0 to 146028888064
SCSI device sddf: 285212672 512-byte hdwr sectors (146029 MB)
sddf: Write Protect is off
sddf: Mode Sense: ed 00 00 08
SCSI device sddf: drive cache: write back
sddf: detected capacity change from 0 to 146028888064
SCSI device sddg: 285212672 512-byte hdwr sectors (146029 MB)
sddg: Write Protect is off
sddg: Mode Sense: ed 00 00 08
SCSI device sddg: drive cache: write back
sddg: detected capacity change from 0 to 146028888064
iostat -
sddd 0.00 0.00 0.00 16 0
sdde 0.00 0.00 0.00 16 0
sddf 0.00 0.00 0.00 16 0
sddg 0.00 0.00 0.00 16 0
sdcz 0.00 0.00 0.00 16 0
sdda 0.00 0.00 0.00 16 0
sddb 0.00 0.00 0.00 16 0
sddc 0.00 0.00 0.00 16 0
any pointers would be helpful.
To keep the forums high quality for all users, please take the time to format your posts correctly.
First of all, use Code Tags when you post any code or data samples so others can easily read your code. You can easily do this by highlighting your code and then clicking on the # in the editing menu. (You can also type code tags
```text
and
```
by hand.)
Second, avoid adding color or different fonts and font size to your posts. Selective use of color to highlight a single word or phrase can be useful at times, but using color, in general, makes the forums harder to read, especially bright colors like red.
Third, be careful when you cut-and-paste, edit any odd characters and make sure all links are working property.
Thank You.
The UNIX and Linux Forums
1 Like
So it is not a rescan issue, but a multipath issue ? Is that it? What does multipath -ll show and what is in multipath.conf ? Did you reload multipath ?
Hi Scrutinizer
I'm not sure its multipath issue or not as multipath.conf for cluster is same for both nodes and multipath -ll not showing disk for this rac server as lsscsi not showing disk. I have reload multipath as well.
Multipath -ll
mpath38 (36005076304ffd7430000000000003107) dm-24 IBM,2107900
[features=1 queue_if_no_path][hwhandler=0][rw]
\_ round-robin 0 [prio=2][active]
\_ 4:0:2:13 sdap 66:144 [active][ready]
\_ 3:0:2:13 sdo 8:224 [active][ready]
mpath72 (36005076304ffd7430000000000001400) dm-31 IBM,2107900
[features=1 queue_if_no_path][hwhandler=0][rw]
\_ round-robin 0 [prio=2][active]
\_ 4:0:2:20 sdaw 67:0 [active][ready]
\_ 3:0:2:20 sdv 65:80 [active][ready]
mpath37 (36005076304ffd7430000000000003106) dm-23 IBM,2107900
[features=1 queue_if_no_path][hwhandler=0][rw]
\_ round-robin 0 [prio=2][active]
\_ 4:0:2:12 sdao 66:128 [active][ready]
\_ 3:0:2:12 sdn 8:208 [active][ready]
mpath69 (36005076304ffd7430000000000002354) dm-28 IBM,2107900
[features=1 queue_if_no_path][hwhandler=0][rw]
\_ round-robin 0 [prio=2][active]
\_ 4:0:2:17 sdat 66:208 [active][ready]
\_ 3:0:2:17 sds 65:32 [active][ready]
mpath71 (36005076304ffd7430000000000002254) dm-30 IBM,2107900
[features=1 queue_if_no_path][hwhandler=0][rw]
\_ round-robin 0 [prio=2][active]
\_ 4:0:2:19 sdav 66:240 [active][ready]
\_ 3:0:2:19 sdu 65:64 [active][ready]
mpath36 (36005076304ffd7430000000000003105) dm-22 IBM,2107900
[features=1 queue_if_no_path][hwhandler=0][rw]
\_ round-robin 0 [prio=2][active]
\_ 4:0:2:11 sdan 66:112 [active][ready]
\_ 3:0:2:11 sdm 8:192 [active][ready]
mpath68 (36005076304ffd7430000000000002210) dm-27 IBM,2107900
[features=1 queue_if_no_path][hwhandler=0][rw]
\_ round-robin 0 [prio=2][active]
\_ 4:0:2:16 sdas 66:192 [active][ready]
\_ 3:0:2:16 sdr 65:16 [active][ready]
mpath70 (36005076304ffd7430000000000002355) dm-29 IBM,2107900
[features=1 queue_if_no_path][hwhandler=0][rw]
\_ round-robin 0 [prio=2][active]
\_ 4:0:2:18 sdau 66:224 [active][ready]
\_ 3:0:2:18 sdt 65:48 [active][ready]
mpath35 (36005076304ffd7430000000000003104) dm-21 IBM,2107900
[features=1 queue_if_no_path][hwhandler=0][rw]
\_ round-robin 0 [prio=2][active]
\_ 4:0:2:10 sdam 66:96 [active][ready]
\_ 3:0:2:10 sdl 8:176 [active][ready]
mpath67 (36005076304ffd7430000000000002485) dm-26 IBM,2107900
[features=1 queue_if_no_path][hwhandler=0][rw]
\_ round-robin 0 [prio=2][active]
\_ 4:0:2:15 sdar 66:176 [active][ready]
\_ 3:0:2:15 sdq 65:0 [active][ready]
mpath66 (36005076304ffd74300000000000021b4) dm-20 IBM,2107900
[features=1 queue_if_no_path][hwhandler=0][rw]
\_ round-robin 0 [prio=2][active]
\_ 4:0:2:9 sdal 66:80 [active][ready]
\_ 3:0:2:9 sdk 8:160 [active][ready]
mpath65 (36005076304ffd743000000000000206f) dm-19 IBM,2107900
[features=1 queue_if_no_path][hwhandler=0][rw]
\_ round-robin 0 [prio=2][active]
\_ 4:0:2:8 sdak 66:64 [active][ready]
\_ 3:0:2:8 sdj 8:144 [active][ready]
mpath64 (36005076304ffd7430000000000002070) dm-18 IBM,2107900
[features=1 queue_if_no_path][hwhandler=0][rw]
\_ round-robin 0 [prio=2][active]
\_ 4:0:2:7 sdaj 66:48 [active][ready]
\_ 3:0:2:7 sdi 8:128 [active][ready]
mpath63 (36005076304ffd7430000000000002170) dm-17 IBM,2107900
[features=1 queue_if_no_path][hwhandler=0][rw]
\_ round-robin 0 [prio=2][active]
\_ 4:0:2:6 sdai 66:32 [active][ready]
\_ 3:0:2:6 sdh 8:112 [active][ready]
mpath78 (36005076304ffd743000000000000207d) dm-10 IBM,2107900
[features=1 queue_if_no_path][hwhandler=0][rw]
\_ round-robin 0 [prio=2][active]
\_ 3:0:2:26 sdab 65:176 [active][ready]
\_ 4:0:2:26 sdbc 67:96 [active][ready]
mpath62 (36005076304ffd743000000000000216f) dm-16 IBM,2107900
[features=1 queue_if_no_path][hwhandler=0][rw]
\_ round-robin 0 [prio=2][active]
\_ 4:0:2:5 sdah 66:16 [active][ready]
\_ 3:0:2:5 sdg 8:96 [active][ready]
mpath77 (36005076304ffd7430000000000002038) dm-9 IBM,2107900
[features=1 queue_if_no_path][hwhandler=0][rw]
\_ round-robin 0 [prio=2][active]
\_ 3:0:2:25 sdaa 65:160 [active][ready]
\_ 4:0:2:25 sdbb 67:80 [active][ready]
mpath59 (36005076304ffd743000000000000212b) dm-13 IBM,2107900
[features=1 queue_if_no_path][hwhandler=0][rw]
\_ round-robin 0 [prio=2][active]
\_ 4:0:2:2 sdae 65:224 [active][ready]
\_ 3:0:2:2 sdd 8:48 [active][ready]
mpath76 (36005076304ffd7430000000000002039) dm-35 IBM,2107900
[features=1 queue_if_no_path][hwhandler=0][rw]
\_ round-robin 0 [prio=2][active]
\_ 4:0:2:24 sdba 67:64 [active][ready]
\_ 3:0:2:24 sdz 65:144 [active][ready]
mpath61 (36005076304ffd743000000000000202a) dm-15 IBM,2107900
[features=1 queue_if_no_path][hwhandler=0][rw]
\_ round-robin 0 [prio=2][active]
\_ 4:0:2:4 sdag 66:0 [active][ready]
\_ 3:0:2:4 sdf 8:80 [active][ready]
mpath58 (36005076304ffd743000000000000212a) dm-12 IBM,2107900
[features=1 queue_if_no_path][hwhandler=0][rw]
\_ round-robin 0 [prio=2][active]
\_ 4:0:2:1 sdad 65:208 [active][ready]
\_ 3:0:2:1 sdc 8:32 [active][ready]
mpath75 (36005076304ffd743000000000000274a) dm-34 IBM,2107900
[features=1 queue_if_no_path][hwhandler=0][rw]
\_ round-robin 0 [prio=2][active]
\_ 4:0:2:23 sdaz 67:48 [active][ready]
\_ 3:0:2:23 sdy 65:128 [active][ready]
mpath60 (36005076304ffd743000000000000202b) dm-14 IBM,2107900
[features=1 queue_if_no_path][hwhandler=0][rw]
\_ round-robin 0 [prio=2][active]
\_ 4:0:2:3 sdaf 65:240 [active][ready]
\_ 3:0:2:3 sde 8:64 [active][ready]
mpath74 (36005076304ffd7430000000000001402) dm-33 IBM,2107900
[features=1 queue_if_no_path][hwhandler=0][rw]
\_ round-robin 0 [prio=2][active]
\_ 4:0:2:22 sday 67:32 [active][ready]
\_ 3:0:2:22 sdx 65:112 [active][ready]
mpath39 (36005076304ffd7430000000000003108) dm-25 IBM,2107900
[features=1 queue_if_no_path][hwhandler=0][rw]
\_ round-robin 0 [prio=2][active]
\_ 4:0:2:14 sdaq 66:160 [active][ready]
\_ 3:0:2:14 sdp 8:240 [active][ready]
mpath56 (36005076304ffd7430000000000002723) dm-11 IBM,2107900
[features=1 queue_if_no_path][hwhandler=0][rw]
\_ round-robin 0 [prio=2][active]
\_ 4:0:2:0 sdac 65:192 [active][ready]
\_ 3:0:2:0 sdb 8:16 [active][ready]
mpath73 (36005076304ffd7430000000000001401) dm-32 IBM,2107900
[features=1 queue_if_no_path][hwhandler=0][rw]
\_ round-robin 0 [prio=2][active]
\_ 4:0:2:21 sdax 67:16 [active][ready]
\_ 3:0:2:21 sdw 65:96 [active][ready]
multipath.conf -
# Blacklist all devices by default. Remove this to enable multipathing
# on the default devices.
#blacklist {
#devnode "*"
#}
blacklist {
device {
vendor "IBM"
product "*"
}
}
######
blacklist_exceptions {
device {
vendor "IBM"
product "2107900"
}
}
#blacklist {
# device {
# vendor "IBM"
# product "*"
# }
#}
## By default, devices with vendor = "IBM" and product = "S/390.*" are
## blacklisted. To enable mulitpathing on these devies, uncomment the
## following lines.
#blacklist_exceptions {
# device {
# vendor "IBM"
# product "2107900"
# }
#}
## Use user friendly names, instead of using WWIDs as names.
defaults {
user_friendly_names yes
# path_grouping_policy multibus
}
##
## Here is an example of how to configure some standard options.
##
#
##
## The wwid line in the following blacklist section is shown as an example
## of how to blacklist devices by wwid. The 2 devnode lines are the
## compiled in default blacklist. If you want to blacklist entire types
## of devices, such as all scsi devices, you should use a devnode line.
## However, if you want to blacklist specific devices, you should use
## a wwid line. Since there is no guarantee that a specific device will
## not change names on reboot (from /dev/sda to /dev/sdb for example)
## devnode lines are not recommended for blacklisting specific devices.
##
#blacklist {
# wwid 26353900f02796769
# devnode "^(ram|raw|loop|fd|md|dm-|sr|scd|st)[0-9]*"
# devnode "^hd[a-z]"
# devnode "^hd[a-z][[0-9]*]"
# devnode "^vd[a-z]"
# devnode "^cciss!c[0-9]d[0-9]*[p[0-9]*]"
#
#}
#multipaths {
# multipath {
# wwid 3600508b4000156d700012000000b0000
# alias yellow
# path_grouping_policy multibus
# path_checker readsector0
# path_selector "round-robin 0"
# failback manual # rr_weight priorities # no_path_retry 5
# }
# multipath {
# wwid 1DEC_____321816758474
# alias red
# }
#}
devices {
device {
vendor "IBM"
product "2107900"
getuid_callout "/sbin/scsi_id -g -u -s /block/%n"
features "1 queue_if_no_path"
hardware_handler "0"
path_grouping_policy multibus
rr_weight uniform
rr_min_io 1000
path_checker tur
}
}
Has Lun Masking and zoning been set correctly by the storage team for all LUNs and for all 4 HBA's (2 per RAC node).?
Did you (or someone else) removed or resized any SAN luns from non-working host recently ?
Or did you present a disk with same WWID as in some point in the past (same storage and host lun number, but different size) ?
Did you diff the output from multipath -ll from both hosts ?
If everything is okay (FC/host wise), you can always try rebooting the host that doesn't see the disks, rather then spending hours on fixing stuff like kernel sees lun sized xy and on storage is setuped and presented with different size and similar situations.
Couple of suggestions for future notice :
- Have names in multipath.conf, it is more user friendly then 'mpathxy' generic name.
If you are creating documentation (you should 
), it will be easier to write and maintain it.
- Create one primary partition when giving disks to services, in this case ASM, don't give whole disks (first sector is for OS usage).
Hi Scrutinizer & Peasant,
@ Scrutinizer - as per storage team lun masking and zoning is correct as per them.
@Peasant- Yes it might be the case that disk might be present to server with same wwid as used earlier for any other server.
yes i have diff the output for both multipath -ll, in one rac disk are showing in multipath and not in another.
and Guys thanks for your suggestions but it seems we have downtime now with no other options , we might reboot the server for this issue.
i'm not sure its a bug or what. 
will guys posted u on this if issue has been resolved after reboot or not..
thanks all for your support.
