DUAL VIOS & Client LPAR hangs at 25b3

I have a DUAL VIO ( IBM Virtual I/O ) setup on p 570.
Two Vio server ( VIOS ) and many LPAR clients.
VIO ( latest version + service pack + applied the fix ) and AIX 6.1 ML2

When both VIOs are running, and if I turn on a Client LPAR, the LPAR hangs at LED 25b3 for more than 1 hour then it boots. But if I switch off one VIO then boot the Client LPAR it boots fine (very fast within 5 minutes)

My client lpar has SAN luns ( going via both VIOs ) and rootvg is mirrored (one virtual disk coming from VIO1 and the other coming from VIO2 )

Storage is IBM DS4200... Installed MPIO drivers...

anyone faced similar problem...searched the internet, found few users who have same problem but there was no resolution posted on the internet
see here AIX v5.3 Dual VIO - LPAR install slow. AIX for POWER Systems pSeries Tech Talk Forums
http://www.ibm.com/developerworks/forums/thread.jspa?messageID=14172543

http://www.ibm.com/developerworks/forums/thread.jspa?threadID=303411&tstart=0\#14385108

http://boardreader.com/thread/Boot\_failure\_with\_LED\_517_28d5bX6i43.html
http://www.ibm.com/developerworks/forums/thread.jspa?threadID=303411&tstart=0\#14385143

AIX SAN LUN connection

someone posted

contacted IBM support and they say check LUN ACCESS MODE ? and there is no such option in DS4200

if they mean access mode on the AIX / VIO side then that is no_reserve ( but the problem still exists )

============================
IBM AIX VIOs LPAR LED 25b3 516 517

I had a similar problem and below is how I was able to finally get it fixed. We had two vio servers as well.

Shutdown vio2
Boot up the LPAR with only vio1 connection
Remove the non-rootvg disks from the LPAR that is unable to boot
Copy vio2's HMC profile to "vio2_no_fiber_cards"
Remove the fiber cards from vio2's "vio2_no_fiber_cards" profile
Boot up vio2 from the "vio2_no_fiber_cards" profile so that it boots with no fiber cards.
On vio2, remove the non-rootvg VTDs for the LPAR. They will be in the defined state.
Shutdown vio2
Boot up vio2 from the "vio2_normal" profile so the fibercards are back in the game
Verify that the disks are all back with the correct hdisk numbers (compare against vio1)
Verify that all the other VTDs (virtual target devices) are still there and now "Available" (lsmap �all)
Re-create the LPAR's VTDs (the one that was getting the 25b3 error)
Run cfgmgr on the LPAR and verify it can see the paths (lspath) through vio2.
Try and reboot the LPAR and see if it can come back up.
After a successful reboot, call vioc2 good

To summarize... some how the vio2 server's mapping got messed up. By booting the vio2 server with no fibercards, I could remove the then "defined" disk mappings and the disks without messing anything up. I then rebooted vio2 with fibercards and in the process of doing so, the vio2 server "refound" the disks. I re-created the VTD mappings on the vio2 server, then ran cfgmgr on the LPAR and then the problem went away. I was able to see both paths again with "lspath" and was able to reboot the LPAR with no hangups.

actually we figured out what was the problem. The problem was with SAN Zoning. In case of DUAL VIO Setup SAN Zoning has to be different.
Each switch should see both Servers

When using only one Fibre Channel per Virtual I/O Server, you need an
additional switch to have a highly available configuration. In this case, it is
important that the SAN zoning be configured such that the single Fibre Channel
in each Virtual I/O Server LPAR is zoned to see both storage controllers in the
FAStT. If a second Fibre Channel is used for additional redundancy, the storage
administrator must ensure that each Fibre Channel is zoned to only one of the
DS4200 controllers. Rules for attachment of FAStT storage units to AIX can be
found in the Storage Manager documentation of the Storage Manager products.

SEE IBM REDBOOKS PowerVM Virtualization on IBM System p: Introduction and Configuration Fourth Edition, SG24-7940-03
Redbooks, published 6 May 2008, last updated 11 January 2010,

IBM System p Advanced POWER Virtualization (PowerVM) Best Practices, REDP-4194-00
Redpapers, published 9 October 2006, last updated 22 May 2009, Rating: (based on 10 reviews)
83% 3. Advanced POWER Virtualization on IBM System p Virtual I/O Server Deployment Examples, REDP-4224-00
Redpapers, published 13 February 2007, last updated 19 June 2007, Rating: (based on 3 reviews)
& RED PAPER 4194