I'm setting up a CentOS 5 server that will be connected to an HP EVA4000. My previous experience is with an HP VA7400 and HP-UX. In HP-UX I can add "alternate paths" to a volume group in order to have redundant paths to the SAN via dual fiber channel HBAs just by either adding them with 'vgextend' or appending the appropriate PV paths when using 'vgcreate'. I tried doing something similar in CentOS with 'vgextend' and got some errors about duplicate PV ids. The LUN (a single one) I create on the EVA4000 show up on the CentOS box as /dev/sda, /dev/sdb, /dev/sdc, /dev/sdd. And when I partitioned /dev/sda so that I had /dev/sda1, I also saw that /dev/sdb1, /dev/sdc1 and /dev/sdd1 were automagically there. So I know that these extra devices reflect the dual fiber channel HBAs and the dual paths through our redundant fiber switches.
My question is... did I do this the right way? When I do a 'vgdisplay' now I get:
[root@localhost ~]# vgdisplay
Found duplicate PV Up921bgzWK4I7xtw7JiOB3VrdbVZhH2Z: using /dev/disk/by-path/pci-0000:13:00.0-fc-0x50001fe1500ca568:0x0001000000000000-part1 not /dev/sda1
Found duplicate PV Up921bgzWK4I7xtw7JiOB3VrdbVZhH2Z: using /dev/disk/by-id/scsi-3600508b400070ac40000900000090000-part1 not /dev/disk/by-path/pci-0000:13:00.0-fc-0x50001fe1500ca568:0x0001000000000000-part1
at the beginning of the expected output. I want to avoid having to test the redundancy by disconnecting the fiber if possible. And in all honesty I don't think this is the right way to set up redundant paths on CentOS (or any other Linux distro for that matter). I think the "alternate paths" functionality is specific to HP-UX's implementation of LVM. But I suspect there is a "right way" to do alternate paths on Linux for redundancy.
Here's what I've come up with from start to finish (OS install through finally setting up multipath using the RAID subsystem) and I will be testing it from the ground up today. I mistakenly installed the 32-bit version of CentOS during my testing phase, so I am starting over with 64-bit today. If I run into any snags, I'll probably post back here. This is the procedure:
Install CentOS 5 (64-bit) with Xen Kernel
Install 'screen' (My personal preference for detached terminal sessions. For those who don't know, think of screen as "VNC" for the command line in terms of coming back to a terminal that you detached from earlier.), 'gcc', 'kernel-xen-devel-2.6.18-8.el5' using 'yum':
Install Emulex 'lpfc' driver to support the fiber channel cards. There is a simple install script in the driver tarball that woks if you have a compiler and kernel sources for your running kernel. The driver that comes with the OS doesn't appear to support the HBAnyware software mentioned in the next item which was the only way I was able to discover the HBA world wide names. I couldn't find that info in /proc or /sys, where most docs claimed they would be. What I did find in /sys seemed to indicate that it's set by software. HBAnyware seemed to be the easiest route to take.
Install Emulex HBAnyware support software (installs to /usr/sbin/hbanyware by default) using the install script in the tar distribution (it's huge at 200+ megs...)
(NOTE: This is not really necessary since RAID level "multipath" takes care of my issue nicely. I didn't know this when I posted so I'm making this note to let others know that step five can be skipped altogether if you don't need multipath at the LVM level.) Install multipath support software using 'yum' and then configure:
yum install device-mapper-multipath
Edit the /etc/multipath.conf file to contain these changes:
# Blacklist all devices by default. Remove this to enable multipathing
# on the default devices.
#blacklist {
# devnode "*"
#}
defaults {
user_friendly_names yes
}
defaults {
udev_dir /dev
polling_interval 10
selector "round-robin 0"
path_grouping_policy failover
getuid_callout "/sbin/scsi_id -g -u -s /block/%n"
prio_callout /bin/true
path_checker readsector0
rr_min_io 100
rr_weight priorities
failback immediate
no_path_retry fail
user_friendly_name yes
}
'modprobe dm-multipath' (load the dm-multipath module)
'modprobe dm-round-robin' (load the dm-round-robin module)
'service multipathd start' (start the multipathd service)
'multipath -v3 -ll' (check the current multipath device info)
'chkconfig multipathd on' (set multipathd to start on boot)
Partition /dev/sda through /dev/sdd to be partition type 'fd' (Linux RAID autodetect):
'fdisk /dev/sda' (Create partition 1 and set it to type 'fd' then save and exit)
'sfdisk -d /dev/sda | sfdisk /dev/sdb'
'sfdisk -d /dev/sda | sfdisk /dev/sdc'
'sfdisk -d /dev/sda | sfdisk /dev/sdd'
Set up multipath RAID for redundancy. The RAID "level" called multipath isn't really RAID. It's just using the RAID subsystem to present the four paths (represented by /dev/sda through /dev/sdd in my case) as one device: /dev/md0. Make sure you have the md module loaded as well or 'mdadm' will complain. Also, make sure that you HAVEN'T used the 'dmsetup' command to assign the disk device nodes to dmmp (dm multipath). That interferes with RAID being able to access them. Just in case you did... you will need to do this first (an example for my case. 'mpath0' might not be in your configuration):
dmsetup -C remove mpath0
Here is the set up for multipath RAID:
'mdadm -C /dev/md0 --level=multipath --raid-devices=4 /dev/sda1 /dev/sdb1 /dev/sdc1 /dev/sdd1'
'mdadm --detail --scan > /etc/mdadm.conf' (saves the above configuration to /etc/mdadm.conf so the RAID will start automatically on boot)
It sure did! Since CentOS uses the 'yum' system some searching on the web and within the yum respositories led me to this: