How to create a Solaris local zone with an exclusive NIC?

Michele31416 · November 22, 2019, 5:12pm

I'm trying to migrate a Solaris 10 flar archive from a Sun M3000 to a T4-1. When I first created the zone I followed the Oracle instructions here: Transitioning an Oracle Solaris 10 Instance to an Oracle Solaris 11 System - Transitioning From Oracle Solaris 10 to Oracle Solaris 11 That worked OK, in the sense that I was able to boot the zone and access it via ssh and vncviewer. However, I did not realize that without an exclusive NIC, my zone would not be able to do DHCP which means the Sun Ray server would not work, and I have to have that.

Now the T4-1 has four gigabit NICS, labeled on the back net0 - net3. I had one cable connected to net0. So I added a second cable from a free port on my switch to net1 and tried to reconfigure my zone to use it instead of net0. That's when all hell broke loose. I can still talk to the global zone over my LAN but my local zone will no longer even boot.

Here's a few possibly relevant pieces:

root@hemlock:/# dladm
LINK                CLASS     MTU    STATE    OVER
ldoms-vsw0.vport0   vnic      1500   up       net0
ldoms-vsw0.vport1   vnic      1500   up       net0
net0                phys      1500   up       --
net1                phys      1500   up       --
net2                phys      1500   unknown  --
net3                phys      1500   unknown  --
net4                phys      1500   up       --

sp-phys0            phys      1500   up       --

Attempting to turn on net1 and boot the zone, I get this:

root@hemlock:/# ifconfig net1 plumb
ifconfig: cannot plumb net1: Interface already exists
root@hemlock:/# ifconfig net1 dhcp start
root@hemlock:/# zoneadm -z s10zone install -u -a /rpool/s10archive/s10.flar
zoneadm: zone s10zone: WARNING: Skipping network interface: interface 'net1' is used in the global zone.
ERROR: 
Zone rpool data set rpool/zones/s10zone/rpool already exists. Aborting the operation
zoneadm: zone 's10zone': ERROR: installation failed: zone switching to configured state
root@hemlock:/#

It doesn't like the global zone having net1? OK, so we try

root@hemlock:/# ifconfig net1 unplumb
root@hemlock:/# zoneadm -z s10zone install -u -a /rpool/s10archive/s10.flar
ERROR: 
Zone rpool data set rpool/zones/s10zone/rpool already exists. Aborting the operation
zoneadm: zone 's10zone': ERROR: installation failed: zone switching to configured state
root@hemlock:/#

My zone is already configured? So where is it?

root@hemlock:/# zoneadm list
global
root@hemlock:/#

Rather than continuing thrashing about in the dark, I figure it's time to ask for help. I freely admit to having no clue what I'm doing.

Peasant · November 23, 2019, 2:34am

Looks like you are mixing ldoms and zones on the same box ?
This is doable, but ill advised.
Create a LDOM first, then zone inside or restore flar archive directly into a LDOM.

As for exclusive / shared IP stack in zone, the basic difference is that exclusive stack operates on L2 (MAC layer), while shared stack on L3 (IP layer).

That means in real life that if you have, say, net0 interface....

You create exclusive zone, it will create additional interface with new MAC address, called VNIC, this is default dladm show-vnic
This interface to outside world is a fully functional L2 interface, and network configuration is done in non-global zone.
You create shared zone, it will share net0 with global zone, creating virtual IP over a net0 and give it to the zone.
This interfaces shares MAC address with global zone interface, and network configuration is done via zonecfg command from global zone.

Global zone can be ran on bare metal or inside ldoms, but do not mix zones with oracle vm server for sparc on same box.
Two options :

[METAL <--> GLOBAL ZONE] [NON-GLOBAL ZONE (shared or exclusive)]
[METAL <--> ORACLE VM] [LDOM/GLOBAL ZONE] [NON-GLOBAL ZONE (shared or exclusive)]

While configuring ldom you need to add (or set) your vnet in this fasion :

ldm add-vnet alt-mac-addrs=auto,auto vnet0 primary-vsw0 ldom

Which will allow zones running in ldom to have up to 2 additional MAC address interfaces or vnics on vnet0 interface.

Hope that helps
Regards
Peasant.

Michele31416 · November 23, 2019, 12:01pm

Unfortunately I did not know that when I started. Yes, I have Solaris 11.4 installed on the T4-1 bare metal. Under the 11.4 global zone I have an LDOM running Linux for SPARC. That's working properly. Then I created a local zone under the global zone to run my Solaris 10 image. That zone is not running in the LDOM, it's side-by-side with it. I'd rather keep it this way than having to create another LDOM which would eat up more host memory and more CPU's, unless it just won't work at all. In any case, I finally got the zone to boot by saying this:

root@hemlock:/rpool# zoneadm -z s10zone attach -F
root@hemlock:/rpool# zoneadm -z s10zone boot
root@hemlock:/rpool# dladm
LINK                CLASS     MTU    STATE    OVER
ldoms-vsw0.vport0   vnic      1500   up       net0
ldoms-vsw0.vport1   vnic      1500   up       net0
net0                phys      1500   up       --
net2                phys      1500   unknown  --
net3                phys      1500   unknown  --
net4                phys      1500   up       --
sp-phys0            phys      1500   up       --
s10zone/net0        vnic      1500   up       net0
net1                phys      1500   unknown  --
s10zone/net1        phys      1500   unknown  --
root@hemlock:/rpool#

Notice that now how the listing has changed. net1 is listed last and there's two entires for s10zone. However, when the zone boots, it tries to connect to bge0 which was the NIC on the M3000 where this image was created. Of course it can't find it so it goes into maintenance mode immediately.

[NOTICE: Zone booting up]

SunOS Release 5.10 Version Generic_Virtual 64-bit
Copyright (c) 1983, 2013, Oracle and/or its affiliates. All rights reserved.
Failed to plumb IPv4 interface(s): bge0
Failed to plumb IPv6 interface(s): bge0
Failed to configure IPv4 DHCP interface(s): bge0
Moving addresses from missing IPv4 interface(s): bge0 (not moved -- not in an IPMP group).
Moving addresses from missing IPv6 interface(s): bge0 (not moved -- not in an IPMP group).
Nov 22 17:53:54 svc.startd[1221]: svc:/network/physical:default: Method "/lib/svc/method/net-physical" failed with exit status 96.
Nov 22 17:53:54 svc.startd[1221]: network/physical:default misconfigured: transitioned to maintenance (see 'svcs -xv' for details)
/usr/sbin/pmconfig: di_init init/access error
Hostname: s10zone
/usr/sbin/pmconfig: cannot open "/dev/pm": No such file or directory
Nov 22 17:53:55 svc.startd[1221]: svc:/platform/sun4u/dcs:default: Method "/lib/svc/method/svc-dcs" failed with exit status 96.
Nov 22 17:53:55 svc.startd[1221]: platform/sun4u/dcs:default misconfigured: transitioned to maintenance (see 'svcs -xv' for details)
Nov 22 17:53:56 svc.startd[1221]: svc:/system/sysevent:default: Method "/lib/svc/method/svc-syseventd start" failed with exit status 95.
Nov 22 17:53:56 svc.startd[1221]: system/sysevent:default failed fatally: transitioned to maintenance (see 'svcs -xv' for details)
Requesting System Maintenance Mode
(See /lib/svc/share/README for more information.)
Console login service(s) cannot run

Root password for system maintenance (control-d to bypass): 
single-user privilege assigned to /dev/console.
Entering System Maintenance Mode

Nov 22 17:55:10 su: 'su root' succeeded for root on /dev/console
Oracle Corporation	SunOS 5.10	Generic Patch	January 2005
# more /etc/hosts
#
# Internet host table
#
127.0.0.1	localhost	loghost
::1	localhost	loghost

And there's a device inside the zone for igb1 now:

# ls /dev/ig*
/dev/igb1

But when I try to plumb it I get
# ifconfig igb1 plumb
ifconfig: cannot plumb igb1: Datalink does not exist
#
This seems to be the missing link (sorry). If I could get the s10zone to plumb igb1, all might be well. Is there perhaps some magic word I need to give in zonecfg:net?

Peasant · November 23, 2019, 12:25pm

Solaris 10 in LDOM should work fine from flar archive.
ldmp2v -
Oracle(R) VM Server for SPARC 3.5 Reference Manual

If something does not work you can always poweroff and delete ldom to reclaim resources.

Do not mix both on same box ....

Regards
Peasant.

Michele31416 · November 23, 2019, 1:12pm

OK, hold everything. I decided to check the zonecfg for s10zone:

root@hemlock:/rpool# zonecfg -z s10zone
zonecfg:s10zone> info
zonename: s10zone
zonepath: /zones/s10zone
brand: solaris10
hostid: 80995cda
net 0:
	physical: net1
anet 0:
	linkname: net0
	configure-allowed-address: true
	auto-mac-address: 2:8:20:a5:23:b3
zonecfg:s10zone>

The clue is the line that says net0 is physical:net1. So I went back to the zone and instead of trying to plumb igb1 I said

# ifconfig net0 plumb
# ifconfig net0
net0: flags=100001000842<BROADCAST,RUNNING,MULTICAST,IPv4,PHYSRUNNING> mtu 1500 index 2
	inet 0.0.0.0 netmask 0 
	ether 2:8:20:a5:23:b3 
# ifconfig net0 192.168.0.78 netmask 255.255.255.0
# ifconfig net0 up && ifconfig net0
net0: flags=100001000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4,PHYSRUNNING> mtu 1500 index 2
	inet 192.168.0.78 netmask ffffff00 broadcast 192.168.0.255
	ether 2:8:20:a5:23:b3 
# ifconfig -a
lo0: flags=2001000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv4,VIRTUAL> mtu 8232 index 1
	inet 127.0.0.1 netmask ff000000 
net0: flags=100001000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4,PHYSRUNNING> mtu 1500 index 2
	inet 192.168.0.78 netmask ffffff00 broadcast 192.168.0.255
	ether 2:8:20:a5:23:b3 
lo0: flags=2002000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv6,VIRTUAL> mtu 8252 index 1
	inet6 ::1/128 
# ping 192.168.0.1
192.168.0.1 is alive
#

And this time it worked! Unfortunately, it did not survive a reboot. So I edited /etc/hosts, /etc/netmasks, and /etc/hostname.net0. Now I have networking when I reboot but it's still not working right:

[NOTICE: Zone rebooting]


SunOS Release 5.10 Version Generic_Virtual 64-bit
Copyright (c) 1983, 2013, Oracle and/or its affiliates. All rights reserved.
Failed to configure IPv4 DHCP interface(s): igb1
/usr/sbin/pmconfig: di_init init/access error
/usr/sbin/pmconfig: cannot open "/dev/pm": No such file or directory
Hostname: s10zone
Nov 23 10:08:07 svc.startd[17615]: svc:/network/ipfilter:default: Method "/lib/svc/method/ipfilter start" failed with exit status 96.
Nov 23 10:08:07 svc.startd[17615]: network/ipfilter:default misconfigured: transitioned to maintenance (see 'svcs -xv' for details)
Nov 23 10:08:08 svc.startd[17615]: svc:/system/sysevent:default: Method "/lib/svc/method/svc-syseventd start" failed with exit status 95.
Nov 23 10:08:08 svc.startd[17615]: system/sysevent:default failed fatally: transitioned to maintenance (see 'svcs -xv' for details)
Requesting System Maintenance Mode
(See /lib/svc/share/README for more information.)
Console login service(s) cannot run

Root password for system maintenance (control-d to bypass): Nov 23 10:08:08 svc.startd[17615]: svc:/platform/sun4u/dcs:default: Method "/lib/svc/method/svc-dcs" failed with exit status 96.
Nov 23 10:08:08 svc.startd[17615]: platform/sun4u/dcs:default misconfigured: transitioned to maintenance (see 'svcs -xv' for details)
Nov 23 10:08:08 svc.startd[17615]: svc:/network/iscsi/initiator:default: Method "/lib/svc/method/iscsid start" failed with exit status 96.
Nov 23 10:08:08 svc.startd[17615]: network/iscsi/initiator:default misconfigured: transitioned to maintenance (see 'svcs -xv' for details)

single-user privilege assigned to /dev/console.
Entering System Maintenance Mode

Nov 23 10:08:12 su: 'su root' succeeded for root on /dev/console
Oracle Corporation	SunOS 5.10	Generic Patch	January 2005
# ping 192.168.0.1
192.168.0.1 is alive
#

Michele31416 · November 24, 2019, 11:29am

Yikes. That looks even more complicated than what I'm trying to do now. I'm also not sure how to do ldmp2v with an existing flar. Nor did I see anything about how to use my net1 interface instead of net0.

Michele31416 · November 25, 2019, 5:33pm

It doesn't look like ldmp2v is going to do it. From the docs:

My source (M3000) is running Solaris 10 with a zfs root. It does nto even have the ldmp2v command. And the target (T4-1) is running Solaris 11.4.