Long story short, we built an alternate boot env. back in July and applied the latest CPU to about 15 servers. Of the 15 servers, 7 servers failed to start the zones after the luactivate / reboot. The zones failed to rename from <zone>-<boot_env> back to <zone>. This is fixed in patch 121430-95.
Anyway, we got that mess fixed up (or so we thought) but now I'm trying to apply the Oct CPU and running into an issue. I can build a new boot env. I can even apply-prereq but I cannot patch the BE.
It appears there is zone/lu mount that is causing me this problem.
# ./installpatchset -B Oct2017CPU --s10patchset
ERROR: Failed to determine zone configuration for target boot environment.
Please ensure all LU boot environments are unmounted, and altroots other
than the target altroot are unmounted prior to executing this script.
# df -h | grep -i swap | grep lu
swap 309G 0K 309G 0% /zoneroot/spahwfm1/lu
# umount /zoneroot/spahwfm1/lu
umount: /zoneroot/spahwfm1/lu busy
# fuser -cu /zoneroot/spahwfm1/lu
/zoneroot/spahwfm1/lu:
# umount -f /zoneroot/spahwfm1/lu
umount: Operation not supported
umount: cannot unmount /zoneroot/spahwfm1/lu
We rebooted one of the servers and the lu mount did not come back and than we can complete the installcluster / installpatchset.
Is there any way for me to force an unmount of this zone lu SWAP?
I don't think I'll be able to request that we reboot 6 production servers just to patch them and reboot them again.
HELP!
EDIT: Two things I want to mention, no, the /etc/zones/index does not list the global zone as configured instead of install (at global or in BE) and no I am not missing zone packages. I know this is related to the zone lu SWAP.
On Solaris 10 local zone swap space is inherited from the global zone. So the swap you are trying to manipulate should show up there.
This swap space is obviously still in use so you probably need to tell the system to stop using it (provided the system has enough swap space elsewhere to keep running).
This is a Solaris 10 server with a ZFS root file system.
The swap is at the global zone. That is where I am seeing it. I checked the (single) zone running on the box and it doesn't appear to be using that swap volume.
[root@spahwfm1gz:/zoneroot/spahwfm1] # ls -lh
total 46
drwxr-xr-x 14 root sys 55 Oct 25 18:30 dev
drwxr-xr-x 2 root root 117 Aug 24 14:12 lu
drwxr-xr-x 50 root root 59 Oct 26 09:18 root
[root@spahwfm1gz:/zoneroot/spahwfm1] # cd lu
[root@spahwfm1gz:/zoneroot/spahwfm1/lu] # ls -la
total 19
drwxr-xr-x 2 root root 117 Aug 24 14:12 .
drwx------ 5 root root 5 May 5 20:30 ..
[root@spahwfm1gz:/zoneroot/spahwfm1/lu] # du -hs
8K .
[root@spahwfm1gz:/zoneroot/spahwfm1/lu] #
I also cannot remove it, was hoping to than be able to brute force an umount
[root@spahwfm1gz:/zoneroot/spahwfm1/lu] # cd ..
[root@spahwfm1gz:/zoneroot/spahwfm1] # rmdir lu
rmdir: directory "lu": Directory is a mount point or in use
[root@spahwfm1gz:/zoneroot/spahwfm1] # rm -Rf lu/
rm: Unable to remove directory lu/: Device busy
Can you go into that directory /zoneroot/spahwfm1/lu again and use some other 'ls' switches like '-b' (unprintable character display) and '-i' (print inode numbers) to check again whether there's a hidden file in there.
It's not in vfstab and I see nothing in the folder. What is frustrating is I know this 'lu' swap directory will be gone on reboot and than the installpatchset will work properly. This is a left over remnant of a bad Live Upgrade.
[root@spahwfm1gz:/] # cd /zoneroot/spahwfm1/lu
[root@spahwfm1gz:/zoneroot/spahwfm1/lu] # ls -b
[root@spahwfm1gz:/zoneroot/spahwfm1/lu] # ls -i
[root@spahwfm1gz:/zoneroot/spahwfm1/lu] # cat /etc/vfstab | grep -i swap
/dev/zvol/dsk/rpool/swap - - swap - no -
swap - /tmp tmpfs - yes -
[root@spahwfm1gz:/zoneroot/spahwfm1/lu] #
I've been messing about with this here on my own system and trying to work this out as we go along!
It appears that swapfiles created (eg, using 'mkfile') are not supported on ZFS filesystem. That I guess means that this swap area is a disk pool. Can you list what pools you have and see if you can find it. Just for good measure, try listing the pools when within the local zone too (although that shouldn't work).
[root@spahwfm1gz:/] # zlogin spahwfm1
[Connected to zone 'spahwfm1' pts/1]
spahwfm1 # zfs list
no datasets available
Here is mount -v for swap from global
[root@spahwfm1gz:/] # mount -v | grep -i swap
swap on /etc/svc/volatile type tmpfs read/write/setuid/devices/rstchown/xattr/dev=5cc0001 on Tue Jul 25 05:40:41 2017
swap on /tmp type tmpfs read/write/setuid/devices/rstchown/xattr/dev=5cc0002 on Tue Jul 25 05:41:11 2017
swap on /var/run type tmpfs read/write/setuid/devices/rstchown/xattr/dev=5cc0003 on Tue Jul 25 05:41:11 2017
swap on /zoneroot/spahwfm1/lu type tmpfs read/write/setuid/devices/rstchown/xattr/dev=5cc000a on Tue Jul 25 05:42:07 2017
swap on /zoneroot/spahwfm1/root/etc/svc/volatile type tmpfs read/write/setuid/nodevices/rstchown/xattr/zone=spahwfm1/dev=5cc000b on Tue Jul 25 06:46:53 2017
swap on /zoneroot/spahwfm1/root/tmp type tmpfs read/write/setuid/nodevices/rstchown/xattr/zone=spahwfm1/dev=5cc000c on Tue Jul 25 06:46:56 2017
swap on /zoneroot/spahwfm1/root/var/run type tmpfs read/write/setuid/nodevices/rstchown/xattr/zone=spahwfm1/dev=5cc000d on Tue Jul 25 06:46:56 2017
[root@spahwfm1gz:/] # swap -d /zoneroot/spahwfm1/root/var/run
/zoneroot/spahwfm1/root/var/run: Is a directory
[root@spahwfm1gz:/] # swap -d /zoneroot/spahwfm1/root/tmp
/zoneroot/spahwfm1/root/tmp: Is a directory
[root@spahwfm1gz:/] # swap -d /zoneroot/spahwfm1/root/etc/svc/volatile
/zoneroot/spahwfm1/root/etc/svc/volatile: Is a directory
[root@spahwfm1gz:/] # swap -d /zoneroot/spahwfm1/lu
/zoneroot/spahwfm1/lu: Is a directory
It seems like each tmpfs is identified by a unique dev=
For that lu mount it is dev=5cc000a. I'm wondering if there is a way to use that to umount the device?
We know that during Live Update the BE and the new BE share the same swap so, in a crash situation, it could be either BE holding the swap open and preventing deletion. It seems that you have to 'back out' the crashed procedure but that's not simple.
I don't think there is any going back unfortunately. When the problem occurred we had to rename ZFS filesystems along with change their mountpoint and canmount options. Finally we had to promote the zoneroot file system. So, we ended up removing /etc/lutab, /etc/lu/ICF., /etc/lu/INODE., and /etc/lu/vtoc.*
Basically the only BE (until I recently created one for the Oct patches) on the system is the currently running one.
I've inventoried the servers today, out of 7 that had the Live Upgrade "problem", only 4 have this problem. I've asked the customer to reboot the servers which will remove that mount point and allow me to proceed.
I just feel like there must be a way without rebooting to accomplish this...
I just wanted to post a follow up. The customer rebooted the 4 servers today and the zone 'lu' swap is now gone. I can now apply the patch set. I still feel like there must have been a way to accomplish this without a reboot though. Thanks!