Lun remove, stuck in /dev/dsk and /dev/rdsk

DustinT · April 30, 2013, 7:47am

So, we removed a LUN from the SAN and the system is refusing to remove the references to it in the /dev folder. I've done the following:

devfsadm -Cv
powermt -q
luxadm -e offline <drive path>
luxadm probe

All those commands failed to remove the path. The drive stills shows up as <drive type unknown> using the format command. Any ideas?

---------- Post updated at 07:47 AM ---------- Previous update was at 07:21 AM ----------

Oh, I forgot to add I've reboot --r too. Here's some ouput that show more info on the error and steps I've taken.

bash-3.2# cfgadm -a -o show_SCSI_LUN
Ap_Id                          Type         Receptacle   Occupant     Condition
c1                             fc           connected    unconfigured unknown
c2                             fc-fabric    connected    configured   unknown
c2::500601623ce01998,0         disk         connected    configured   unknown
c2::500601623ce01998,1         disk         connected    configured   unknown
c2::500601623ce01998,2         disk         connected    configured   unknown
c2::5006016a3ce01998,0         disk         connected    configured   unknown
c2::5006016a3ce01998,1         disk         connected    configured   unknown
c2::5006016a3ce01998,2         disk         connected    configured   unknown
bash-3.2# echo|format
Searching for disks...done


AVAILABLE DISK SELECTIONS:
       0. c0t0d0 <SUN80G cyl 65533 alt 2 hd 16 sec 149>
          /pci@7c0/pci@0/pci@8/scsi@2/sd@0,0
       1. c2t5006016A3CE01998d0 <drive type unknown>
          /pci@780/SUNW,qlc@0,1/fp@0,0/ssd@w5006016a3ce01998,0
       2. c2t5006016A3CE01998d1 <DGC-RAID5-0429 cyl 40958 alt 2 hd 256 sec 10>
          /pci@780/SUNW,qlc@0,1/fp@0,0/ssd@w5006016a3ce01998,1
       3. c2t5006016A3CE01998d2 <DGC-RAID5-0429 cyl 51198 alt 2 hd 256 sec 16>  new_lun
          /pci@780/SUNW,qlc@0,1/fp@0,0/ssd@w5006016a3ce01998,2
       4. c2t500601623CE01998d0 <drive type unknown>
          /pci@780/SUNW,qlc@0,1/fp@0,0/ssd@w500601623ce01998,0
       5. c2t500601623CE01998d1 <DGC-RAID5-0429 cyl 40958 alt 2 hd 256 sec 10>
          /pci@780/SUNW,qlc@0,1/fp@0,0/ssd@w500601623ce01998,1
       6. c2t500601623CE01998d2 <DGC-RAID5-0429 cyl 51198 alt 2 hd 256 sec 16>  new_lun
          /pci@780/SUNW,qlc@0,1/fp@0,0/ssd@w500601623ce01998,2
       7. emcpower0e <DGC-RAID5-0429 cyl 40958 alt 2 hd 256 sec 10>
          /pseudo/emcp@0
       8. emcpower2d <DGC-RAID5-0429 cyl 51198 alt 2 hd 256 sec 16>  new_lun
          /pseudo/emcp@2
Specify disk (enter its number): Specify disk (enter its number):
bash-3.2# cfgadm -c unconfigure -o unusable_SCSI_LUN c2::5006016A3CE01998d0
bash-3.2# cfgadm -c unconfigure -o unusable_SCSI_LUN c2::500601623CE01998d0
bash-3.2# luxadm probe
No Network Array enclosures found in /dev/es

Found Fibre Channel device(s):
  Node WWN:50060160bce01998  Device Type:Disk device
    Logical Path:/dev/rdsk/c2t5006016A3CE01998d0s2
    Logical Path:/dev/rdsk/c2t500601623CE01998d0s2
  Node WWN:50060160bce01998  Device Type:Disk device
    Logical Path:/dev/rdsk/c2t5006016A3CE01998d1s2
    Logical Path:/dev/rdsk/c2t500601623CE01998d1s2
  Node WWN:50060160bce01998  Device Type:Disk device
    Logical Path:/dev/rdsk/c2t5006016A3CE01998d2s2
    Logical Path:/dev/rdsk/c2t500601623CE01998d2s2
bash-3.2# devfsadm -Cv

hicksd8 · April 30, 2013, 7:50am

This may be a silly question but since you know which 'device' it is, why don't you manually remove the device node yourself from /dev/rdsk, /dev/dsk and /dev/cfg?

If it won't let you do that with the system up, take it down, boot from CD, mount the HD root filesystem under /a, and get in there and do it.

Or am I missing something here?

Why do you necessarily need to get rid of it anyway?

DustinT · April 30, 2013, 8:09am

No, it's probably not a silly question. But, mucking about inside /dev isn't something I'm willing to do with my current skills. It needs to be removed because we had to make some changes on our storage and that LUN is no longer available. Now I just need to convince the server to just let it go.

---------- Post updated at 08:08 AM ---------- Previous update was at 07:58 AM ----------

Well.... I decided to delete the /dev/dsk and /dev/rdsk references. While booted into single user mode, it worked with no problem. But, when I ran a devfsadm -Cv it recreated all the references.

bash-3.2# devfsadm -Cv
devfsadm[1365]: verbose: symlink /dev/dsk/c2t500601623CE01998d0s0 -> ../../devices/pci@780/SUNW,qlc@0,1/fp@0,0/ssd@w500601623ce01998,0:                      a
devfsadm[1365]: verbose: symlink /dev/dsk/c2t500601623CE01998d0s1 -> ../../devices/pci@780/SUNW,qlc@0,1/fp@0,0/ssd@w500601623ce01998,0:                      b
devfsadm[1365]: verbose: symlink /dev/dsk/c2t500601623CE01998d0s2 -> ../../devices/pci@780/SUNW,qlc@0,1/fp@0,0/ssd@w500601623ce01998,0:                      c
devfsadm[1365]: verbose: symlink /dev/dsk/c2t500601623CE01998d0s3 -> ../../devices/pci@780/SUNW,qlc@0,1/fp@0,0/ssd@w500601623ce01998,0:                      d
devfsadm[1365]: verbose: symlink /dev/dsk/c2t500601623CE01998d0s4 -> ../../devices/pci@780/SUNW,qlc@0,1/fp@0,0/ssd@w500601623ce01998,0:                      e
devfsadm[1365]: verbose: symlink /dev/dsk/c2t500601623CE01998d0s5 -> ../../devices/pci@780/SUNW,qlc@0,1/fp@0,0/ssd@w500601623ce01998,0:                      f
devfsadm[1365]: verbose: symlink /dev/dsk/c2t500601623CE01998d0s6 -> ../../devices/pci@780/SUNW,qlc@0,1/fp@0,0/ssd@w500601623ce01998,0:                      g
devfsadm[1365]: verbose: symlink /dev/dsk/c2t500601623CE01998d0s7 -> ../../devices/pci@780/SUNW,qlc@0,1/fp@0,0/ssd@w500601623ce01998,0:                      h
devfsadm[1365]: verbose: symlink /dev/rdsk/c2t500601623CE01998d0s0 -> ../../devices/pci@780/SUNW,qlc@0,1/fp@0,0/ssd@w500601623ce01998,0                      :a,raw
devfsadm[1365]: verbose: symlink /dev/rdsk/c2t500601623CE01998d0s1 -> ../../devices/pci@780/SUNW,qlc@0,1/fp@0,0/ssd@w500601623ce01998,0                      :b,raw
devfsadm[1365]: verbose: symlink /dev/rdsk/c2t500601623CE01998d0s2 -> ../../devices/pci@780/SUNW,qlc@0,1/fp@0,0/ssd@w500601623ce01998,0                      :c,raw
devfsadm[1365]: verbose: symlink /dev/rdsk/c2t500601623CE01998d0s3 -> ../../devices/pci@780/SUNW,qlc@0,1/fp@0,0/ssd@w500601623ce01998,0                      :d,raw
devfsadm[1365]: verbose: symlink /dev/rdsk/c2t500601623CE01998d0s4 -> ../../devices/pci@780/SUNW,qlc@0,1/fp@0,0/ssd@w500601623ce01998,0                      :e,raw
devfsadm[1365]: verbose: symlink /dev/rdsk/c2t500601623CE01998d0s5 -> ../../devices/pci@780/SUNW,qlc@0,1/fp@0,0/ssd@w500601623ce01998,0                      :f,raw
devfsadm[1365]: verbose: symlink /dev/rdsk/c2t500601623CE01998d0s6 -> ../../devices/pci@780/SUNW,qlc@0,1/fp@0,0/ssd@w500601623ce01998,0                      :g,raw
devfsadm[1365]: verbose: symlink /dev/rdsk/c2t500601623CE01998d0s7 -> ../../devices/pci@780/SUNW,qlc@0,1/fp@0,0/ssd@w500601623ce01998,0                      :h,raw
devfsadm[1365]: verbose: symlink /dev/dsk/c2t5006016A3CE01998d0s0 -> ../../devices/pci@780/SUNW,qlc@0,1/fp@0,0/ssd@w5006016a3ce01998,0:                      a
devfsadm[1365]: verbose: symlink /dev/dsk/c2t5006016A3CE01998d0s1 -> ../../devices/pci@780/SUNW,qlc@0,1/fp@0,0/ssd@w5006016a3ce01998,0:                      b
devfsadm[1365]: verbose: symlink /dev/dsk/c2t5006016A3CE01998d0s2 -> ../../devices/pci@780/SUNW,qlc@0,1/fp@0,0/ssd@w5006016a3ce01998,0:                      c
devfsadm[1365]: verbose: symlink /dev/dsk/c2t5006016A3CE01998d0s3 -> ../../devices/pci@780/SUNW,qlc@0,1/fp@0,0/ssd@w5006016a3ce01998,0:                      d
devfsadm[1365]: verbose: symlink /dev/dsk/c2t5006016A3CE01998d0s4 -> ../../devices/pci@780/SUNW,qlc@0,1/fp@0,0/ssd@w5006016a3ce01998,0:                      e
devfsadm[1365]: verbose: symlink /dev/dsk/c2t5006016A3CE01998d0s5 -> ../../devices/pci@780/SUNW,qlc@0,1/fp@0,0/ssd@w5006016a3ce01998,0:                      f
devfsadm[1365]: verbose: symlink /dev/dsk/c2t5006016A3CE01998d0s6 -> ../../devices/pci@780/SUNW,qlc@0,1/fp@0,0/ssd@w5006016a3ce01998,0:                      g
devfsadm[1365]: verbose: symlink /dev/dsk/c2t5006016A3CE01998d0s7 -> ../../devices/pci@780/SUNW,qlc@0,1/fp@0,0/ssd@w5006016a3ce01998,0:                      h
devfsadm[1365]: verbose: symlink /dev/rdsk/c2t5006016A3CE01998d0s0 -> ../../devices/pci@780/SUNW,qlc@0,1/fp@0,0/ssd@w5006016a3ce01998,0                      :a,raw
devfsadm[1365]: verbose: symlink /dev/rdsk/c2t5006016A3CE01998d0s1 -> ../../devices/pci@780/SUNW,qlc@0,1/fp@0,0/ssd@w5006016a3ce01998,0                      :b,raw
devfsadm[1365]: verbose: symlink /dev/rdsk/c2t5006016A3CE01998d0s2 -> ../../devices/pci@780/SUNW,qlc@0,1/fp@0,0/ssd@w5006016a3ce01998,0                      :c,raw
devfsadm[1365]: verbose: symlink /dev/rdsk/c2t5006016A3CE01998d0s3 -> ../../devices/pci@780/SUNW,qlc@0,1/fp@0,0/ssd@w5006016a3ce01998,0                      :d,raw
devfsadm[1365]: verbose: symlink /dev/rdsk/c2t5006016A3CE01998d0s4 -> ../../devices/pci@780/SUNW,qlc@0,1/fp@0,0/ssd@w5006016a3ce01998,0                      :e,raw
devfsadm[1365]: verbose: symlink /dev/rdsk/c2t5006016A3CE01998d0s5 -> ../../devices/pci@780/SUNW,qlc@0,1/fp@0,0/ssd@w5006016a3ce01998,0                      :f,raw
devfsadm[1365]: verbose: symlink /dev/rdsk/c2t5006016A3CE01998d0s6 -> ../../devices/pci@780/SUNW,qlc@0,1/fp@0,0/ssd@w5006016a3ce01998,0                      :g,raw
devfsadm[1365]: verbose: symlink /dev/rdsk/c2t5006016A3CE01998d0s7 -> ../../devices/pci@780/SUNW,qlc@0,1/fp@0,0/ssd@w5006016a3ce01998,0                      :h,raw

---------- Post updated at 08:09 AM ---------- Previous update was at 08:08 AM ----------

So, the driver that providing the 'ssd' support appears to be stuck in a loop where it's refusing to let go of the old references??

hicksd8 · April 30, 2013, 8:10am

As per my post, you need to remove from /dev/cfg too.

You need to remove the device node from all three directories.

You may need to use 'rm -f' if it won't go.

DustinT · April 30, 2013, 8:13am

under /dev/cfg I have the following

bash-3.2# ls
c0  c1  c2
bash-3.2# cat c2
cat: input error on c2: No such device or address
bash-3.2# cd c2
bash: cd: c2: Not a directory

I have other LUNS off of c2 that I don't want to lose. How do I go about removing that reference since c2 isn't a file I can edit??

---------- Post updated at 08:13 AM ---------- Previous update was at 08:12 AM ----------

bash-3.2# ls -l /dev/cfg
total 3
lrwxrwxrwx   1 root     root          45 Nov 30  2010 c0 -> ../../devices/pci@7c0/pci@0/pci@8/scsi@2:scsi
lrwxrwxrwx   1 root     root          42 Apr  5  2011 c1 -> ../../devices/pci@780/SUNW,qlc@0/fp@0,0:fc
lrwxrwxrwx   1 root     root          44 Apr  5  2011 c2 -> ../../devices/pci@780/SUNW,qlc@0,1/fp@0,0:fc

hicksd8 · April 30, 2013, 8:21am

If you have other LUN's on c2 then you'll need to (in single user)

rm -r /a/dev/cfg/c2

then

devfsadm -r /a -p /a/etc/path_to_inst

to write the devices you still need back.

Remember to check that /a/etc/vfstab is correct and not trying to mount something which isn't there any more.

DustinT · April 30, 2013, 8:25am

I'm trying a modified version of this.

rm /dev/dsk/<path>
rm /dev/rdsk/<path>
rm /dev/cfg/c2
devfsadm -r /a -p /etc/path_to_inst
reboot -- -r

hicksd8 · April 30, 2013, 8:28am

Yep, that looks okay if the system will let you do that whilst it's multi-user.

Otherwise resort to single user.

I think that ought to fix it.

DustinT · April 30, 2013, 8:30am

It re-added the disks.....

---------- Post updated at 08:30 AM ---------- Previous update was at 08:30 AM ----------

Do you know of anyone hiring Windows admins?

hicksd8 · April 30, 2013, 8:43am

Strange! What version of Solaris is this exactly?

Is this old enough to still have a boot archive?

Scratching head!

---------- Post updated at 01:43 PM ---------- Previous update was at 01:34 PM ----------

Another thought is have you power cycled the box since removing the LUN?

Shouldn't do, but perhaps the HBA still has the LUN discovered and just thinks that it's gone offlline and is coming back.

https://sites.google.com/site/brenosobral/solaris/remover-devices-san-attached-storage

DustinT · April 30, 2013, 9:39am

bash-3.2# cat /etc/release
                      Solaris 10 10/09 s10s_u8wos_08a SPARC
           Copyright 2009 Sun Microsystems, Inc.  All Rights Reserved.
                        Use is subject to license terms.
                           Assembled 16 September 2009

I'll go unplug it now.

---------- Post updated at 09:39 AM ---------- Previous update was at 08:50 AM ----------

Yup, it didn't help at all.

hicksd8 · April 30, 2013, 9:50am

Did you see the link I posted in #10?

Any good?

DustinT · April 30, 2013, 6:00pm

I finally got it fixed. I had to delete all the references in /dev/dsk, /dev/cfg/(controller) and /dev/rdsk, pwercf -q, powermt check (then remove all), powercf -q and finally a powermt save...

What a sequence!

Hicks, thanks for your help. That's two beers I owe you.

abhi_8029 · May 16, 2013, 3:09am

Hi Dustin,

Glad to hear that your problem is solved. If you know which devices / LUN to remove, the below powerpath commands can be used to remove them.

powermt remove force dev=cXtXdX ( replace with correct ctd number)
powermt display paths
powermt check ( will prompt you to remove dead paths)
powermt config
powermt save

This is better and safe than fiddling with solaris devices under /dev/dsk.