Disk controller reassignment

kerrygold · June 7, 2013, 6:09am

Hi

The scenario in the oracle t4-1 hardware with solaris 10 is as follows:

The initial install on the server was performed with an incorrect disk layout.
New disks were ordered and slot into the server.
The disks are allocated a controller and a disk number based on the disk or replacement disk WWN.
This allocation is dependent on the disk not on the slot.
The install on the server requires the slots and the controller and the disk allocation to be different than the original install
The disk allocation is at a lower level than the operating system and despite several installs of the operating system and attempts to manual configure/unconfigure the disks this has not been possible

I need to reassign the controller numbers for my disks.

Currently, they are like this:

# format
Searching for disks...done


AVAILABLE DISK SELECTIONS:
       0. c0t5000CCA03C3B0B70d0 <HITACHI-H106030SDSUN300G-A2B0 cyl 46873 alt 2 hd 20 sec 625>  solaris
          /scsi_vhci/disk@g5000cca03c3b0b70
       1. c0t5000CCA03C3F98F8d0 <SUN600G cyl 64986 alt 2 hd 27 sec 668>
          /scsi_vhci/disk@g5000cca03c3f98f8
       2. c0t5000CCA03C34EB5Cd0 <SUN300G cyl 46873 alt 2 hd 20 sec 625>
          /scsi_vhci/disk@g5000cca03c34eb5c
       3. c0t5000CCA03C407F48d0 <SUN600G cyl 64986 alt 2 hd 27 sec 668>
          /scsi_vhci/disk@g5000cca03c407f48
       4. c0t5000CCA03C374828d0 <SUN600G cyl 64986 alt 2 hd 27 sec 668>
          /scsi_vhci/disk@g5000cca03c374828
       5. c0t5001517BB2856A03d0 <ATA-INTELSSDSA2BZ30-0362 cyl 35769 alt 2 hd 128 sec 128>
          /scsi_vhci/disk@g5001517bb2856a03
       6. c0t5001517BB285698Bd0 <ATA-INTELSSDSA2BZ30-0362 cyl 35769 alt 2 hd 128 sec 128>
          /scsi_vhci/disk@g5001517bb285698b
       7. c0t5001517BB2856946d0 <ATA-INTELSSDSA2BZ30-0362 cyl 35769 alt 2 hd 128 sec 128>
          /scsi_vhci/disk@g5001517bb2856946
Specify disk (enter its number): ^C

but I would like them to be like this:

# format
Searching for disks...done


AVAILABLE DISK SELECTIONS:
       0. c0t5000CCA03C3B0B70d0 <HITACHI-H106030SDSUN300G-A2B0 cyl 46873 alt 2 hd 20 sec 625>  solaris
          /scsi_vhci/disk@g5000cca03c3b0b70
       1. c0t5000CCA03C34EB5Cd0 <SUN300G cyl 46873 alt 2 hd 20 sec 625>
          /scsi_vhci/disk@g5000cca03c34eb5c
       2. c0t5000CCA03C3F98F8d0 <SUN600G cyl 64986 alt 2 hd 27 sec 668>
          /scsi_vhci/disk@g5000cca03c3f98f8
       3. c0t5000CCA03C407F48d0 <SUN600G cyl 64986 alt 2 hd 27 sec 668>
          /scsi_vhci/disk@g5000cca03c407f48
       4. c0t5000CCA03C374828d0 <SUN600G cyl 64986 alt 2 hd 27 sec 668>
          /scsi_vhci/disk@g5000cca03c374828
       5. c0t5001517BB2856A03d0 <ATA-INTELSSDSA2BZ30-0362 cyl 35769 alt 2 hd 128 sec 128>
          /scsi_vhci/disk@g5001517bb2856a03
       6. c0t5001517BB285698Bd0 <ATA-INTELSSDSA2BZ30-0362 cyl 35769 alt 2 hd 128 sec 128>
          /scsi_vhci/disk@g5001517bb285698b
       7. c0t5001517BB2856946d0 <ATA-INTELSSDSA2BZ30-0362 cyl 35769 alt 2 hd 128 sec 128>
          /scsi_vhci/disk@g5001517bb2856946
Specify disk (enter its number): ^C

i.e. i want to swap around the position of disk 1 and disk 2.

Does anyone know how to do this?

hicksd8 · June 7, 2013, 6:28am

I'm not at all sure that I understand what you are trying to do having re-read your post twenty times.

Why do you care what order the drives are?

Is this the scenario? You installed a server and then decided that the disk drives needed changing for some reason? You put a completely new set of drives in the server and restored the original installation from backup but it won't run???? Is that it?

Also, what are the filesystems? ufs, zfs or what?

kerrygold · June 7, 2013, 6:38am

Hi hicksd8,

Thanks for the reply.
Its a zfs server and the factory install was done using an incorrect disk configuration (two 600gb disks and two 300gb disks). By the time the server got to me, I needed to reinstall solaris with the correct number of disks, so i added three 300gbSSD disks (for a sybase db) and another 600gb disk for the hotspare pool.

However the solaris install puts the root disk on disk0 (300gb) but at the end of the solaris sys-unconfig, the svm defines disk1 (600gb) as the root mirror.
Ideally, i need my 300gb root disk mirrored to another 300gb disk.

I've moved around the disks physically and tried cfgadm, but no luck.
Do you have any ideas?

hicksd8 · June 7, 2013, 6:48am

So are you restoring data from the original installation or is this a completely new build with the new disks?

NOTE TO MODERATORS: Can you please move this thread to mainstream Solaris forum. The OP is in hardware but it's actually a configuration issue. We need members with the right skills to see this.

kerrygold · June 7, 2013, 7:02am

Hi,

Its a completely new build...

GP81 · June 7, 2013, 9:28am

@hicksd8

You are not alone

@kerrygold
Is it zfs mirror?
Why don't you just detach submirror, and attach other 300GB disk?
Please post output from

# zpool status rpool

hicksd8 · June 8, 2013, 6:43am

Hey, I just had another thought on this.

You say that the platform is a T4-1 and this box usually has a built-in hardware raid controller. You seem to be saying that the 'factory' install mirrored the first two slots, which is normal for factory setup.

Does your box have a built-in hardware raid controller?

If so, could that be the reason the first two slots are mirrored??

If so, you can break that mirror with the raid tools and create another mirror of your choice.

I may be completely wrong on this but you need to check it out.
(Google "T4-1 RAID")

bartus11 · June 8, 2013, 6:50am

Speaking from the experience with Netra T4-2: when you configure hardware RAID, the resulting disk in "format" contains "LSI" in its name, which is not the case here. Also, when two disks are hardware-mirrored, they are visible as single disk in Solaris, so there would be 7 disks visible here and not 8.

hicksd8 · June 8, 2013, 6:54am

@bartus11....I agree with you but what possible explanation is there for Solaris to decide to mirror a 300GB drive to a 600GB drive, all on its own on a new build???

How does that possibly work???

The only way that I can think of is a pre-existing hardware RAID configuration.

bartus11 · June 8, 2013, 7:07am

To understand the situation, we should first get the output of

zpool status
metastat -a

GP81 · June 8, 2013, 7:16pm

Is raidctl works on that platform?
If yes, then it could be helpful too.

hicksd8 · June 9, 2013, 6:00am

@GP81.....

I don't think 'raidctl' will work on a T4-1. Raidctl is for the older built-in LSI hardware raid controllers used on hardware like the T2000.

If it was a LSI like that, then as Bartus11 says, once a raid mirror is configured the mirror drive is 'hidden' from view.

This is a T4-1 which uses a different raid controller and different command set.......

Create a Hardware RAID Volume - SPARC T4-1 Server HTML Document Collection

I've already made the point that typical factory configurations have the first two slots mirrored when supplied to the customer and this appears to be what we've got here.

We've got to the point where several members are trying to help and speculating over the resolution of the problem but what we need is Kerrygold to post more output.

I agree with Bartus11 that we all wait for Kerrygold to get back online otherwise we are all just guessing.

My own experience with T4 series is pretty thin anyway.

kerrygold · June 10, 2013, 5:28am

Hi,

Thanks for the help so far.
The problem is seen just after the solaris is installed (it tries to create the root mirror on the 600gb disk instead of the 300gb)

the output of metastat is:

# metastat d0
d0: Mirror
    Submirror 0: d10
      State: Okay         
    Submirror 1: d20
      State: Resyncing    
    Resync in progress: 0 % done
    Pass: 1
    Read option: roundrobin (default)
    Write option: parallel (default)
    Size: 542925000 blocks (258 GB)

d10: Submirror of d0
    State: Okay         
    Hot spare pool: hsp001
    Size: 542925000 blocks (258 GB)
    Stripe 0:
        Device                             Start Block  Dbase        State Reloc Hot Spare
        /dev/dsk/c0t5000CCA03C3B0B70d0s0          0     No            Okay   Yes 


d20: Submirror of d0
    State: Resyncing    
    Hot spare pool: hsp001
    Size: 1048576968 blocks (500 GB)
    Stripe 0:
        Device                             Start Block  Dbase        State Reloc Hot Spare
        /dev/dsk/c0t5000CCA03C3F98F8d0s0          0     No            Okay   Yes 


Device Relocation Information:
Device                           Reloc  Device ID
/dev/dsk/c0t5000CCA03C3B0B70d0   Yes    id1,sd@n5000cca03c3b0b70
/dev/dsk/c0t5000CCA03C3F98F8d0   Yes    id1,sd@n5000cca03c3f98f8
#

I will try to break mirrors manually with raid tools and see what i can do...

hicksd8 · June 10, 2013, 1:01pm

Personally I've never seen a Solaris install routine set up a mirror all on its own. Okay, I guess you learn something new every day!!!

However, the metastat output confirms that is what it's done so, as you say, you can break the mirror and remake the mirror with a different drive using the tools. Give it a go.