Sorry guys but let me start off by saying I am an AIX noob! I am running AIX 6.1 and recently purchased two iSCSI TOE cards. The plan is to connect these two cards to a Dell EqualLogic SAN and that has two volumes configured for logs and data.
The point of the two cards is to do load balancing and fail over if one card does, obviously I am interested in making this reliable and always available.
What I have done so far is:
Configured the ics0 and ics1 to have
IP addresses on the iSCSI network with the SAN
discovery policy of file
Edited the /etc/iscsi/targetshw# to include the paths to both volumes that I have configured
Now, when I do lspv, I see four new hdisks, because each card has connected to each volume. So obviously I have duplicate connections.
What I want to do is have these two cards load balance the connection and failover if a card dies and that's where I am stuck. I have read a little on MPIO but not having much luck with iSCSI TOE.
Can anyone add any configuration advice, it would be much appreciated!!
Thanks, I skimmed the article and it kind of assured me of something I read in an article Dell published. I will copy and paste it below, the article was from 2009 so I was hoping that maybe they had resolved this by now. I went ahead and opened a case with Dell to see what they will suggest or if they will confirm it is not supported.
[LEFT]MPIO is not supported between the IBM HBA cards and EqualLogic PS Series storage
arrays. If you have multiple HBA ports, you should split your volumes among your ports,[/LEFT]
but be careful to only connect once to any volume.
My hdisks to the EqualLogic show up as "Other iSCSI Disk Drive" so I assume really my issue is because it is truely not supported. I did read this in one of the articles you linked from IBM.
Devices not supported by the AIX disk PCMs or tape PCMs require the device vendor to provide attributes predefined in the ODM, a PCM, and any other supporting code necessary to recognize the device as MPIO-capable.
When I run lsattr -El hdisk# for the disk of the SAN it does not show the PCM or algorithm settings.
Still waiting for the response from Dell, I did not see any downloads on their site for ODM. I will look at the possible workaround you posted.
Maybe you need a newer AIX? "Beginning with AIX 5L� Version 5.2 with the 5200-04 Recommended Maintenance package, iSCSI disk devices are supported as MPIO devices."
My understanding is it's because I am using HBA cards. Otherwise if I was using the iSCSI software initiator and NIC ports it would work.
Just spoke to Dell and they linked me the same MPIO article on IBM's site. We came to the same conclusion that it seems like the AIX is missing some sort of software or driver to make it properly recognize the EQL.
There is actually a switch between the two, I failed to mention that before. So each controller has it's own cable. An ethernet cable goes from each HBA to two different switches, and then the SAN has four connections total between the two switches. So I want to be able to lose a card or a switch and still be functional.
---------- Post updated at 11:18 AM ---------- Previous update was at 11:13 AM ----------
An MPIO other iSCSI disk is a subset of other iSCSI disks. A device is supported as an MPIO other iSCSI disk only if there is no vendor-provided ODM predefined attributes and the device has been certified to operate with the AIX PCM. Certification does not guarantee that all device capabilities are supported when the device is configured as an MPIO other iSCSI disk.
To determine which devices are supported as MPIO other iSCSI disks, run the following script: odmget -quniquetype=disk/iscsi/mpioosdisk PdAt | grep deflt | cut -d'"' -f2
The script output displays a list of inquiry data that contains the device vendor and model.
When I ran this command I got an error
0518-507 odmget: Could not retrieve object for PdAt, ODM error number 5904
I was wondering if there is an ethernet load balancing and failover solution for this, since it is partially dual/quad ethernet. Are there actually two scsi cables or other controller paths in the SAN that need to be balanced ?
I do have the EqualLogic login and looked but found nothing useful there for AIX.
The SAN is completely using iSCSI, has two separate controllers completely that are for redundancy in case of hardware failure. That does not have to be balanced because it's only used if the other controller isn't active. It has two ports on each controller that it will load balance or failover as well.
Are the two controllers on different cables?
If so, are the disks dual-port scsi, not mirrors?
Could it load balance the controllers remotely? (I guess if the primary controller failed, it could not pass traffic to the other.) In other words, would network load balancing and failover be sufficient? (Not sure how good AIX is at network load balancing!)
It seems Veritas can take over this function, which they call DMP. They still need the ODM set up right, but turn off MPIO!
Another nice article. I think the key is the ODM setup.
All "Other iscsi ..." means is that there are no ematching entries for these disks in the predefined database.
You can still ask Dell about ODM entries for AIX, or art least recommendations on max transfer size/count, qdepth, etc..
The two Dell tech's I've talked to are pretty clueless when it comes to AIX, they have linked me articles regarding configuring the iSCSI adapters in general like assigning IP addresses. There is a Dell article from 2009 that talks about the qdepth, but doesn't really help me for load balancing of failover.
My two iSCSI adapters if ics0 and ics1 are both set to file and I have edited the /etc/iscsi/targetshw0 and targetshw1 to connect to both volumes. According to the Dell article you linked before for Equal Logic configuration, MPIO is not supported and I should just use one card for each volume which does not give me any fail over or load balancing on the AIX server.
---------- Post updated at 02:45 PM ---------- Previous update was at 02:43 PM ----------
Also to add, I have spent hours today googling for someone else who has posted on EqualLogic and odm and have not found really anyone. I know that the EMC SAN's do have an odm and have seen that talked about multiple times.
I really do not want to use a different program to do MPIO because there will also be an attached SCSI SAN that will hold two other volumes the AIX server will access as well and it is currently using MPIO for that and I do not want to make any changes to it.
Can I ask a general question and maybe someone can chime in just to clarify?
We were sold two HBA iSCSI TOE PCI-X cards and both cards were installed with the idea that we would have load balancing and failover connected to the Dell EqualLogic.
Because of the type of cards I assume that I should be configuring the ics0 and ics1 interfaces, can someone just confirm that for me? Because there's also the iscsi0 showing up and my understanding that is a software initator that I would use if I didn't have these HBA cards.
The reason I ask is I am reviewing the only Dell document I could find regarding this and it talkes about Etherchannel and using interfaces called ent0 and ent1, which shows up in my system, but I can't assign IP addresses to those interfaces. Only to my main interface en0 and I do not want that to change since my iSCSI network is separate.
When I do a
lsdev -Cs pci I do see the ics0 and ics1 iSCSI Adapters show in the list. I also see the ent0 and ent1 and they are PCI-X adapters and that kind of confuses me.
To be honest i don't know your cards and will probably not be of much help to you. Just a few things here i can clear up:
"Etherchannel" is IBMs name for bundling several IP interfaces together to one. You can do that either for failover purposes or for load balancing. The process is: you first create the physical interfaces (entX, entX+1, ...), don't give them any IP address or configure them otherwise, then create an "Etherchannel" device from them. You probably want to follow the SMIT panels for this, they are pretty self-explanatory.
Issue a
lsdev -Cc adapters
which should show you your cards as well as by which device name they go by.