LDOM Solaris 11 add Network vsw (Virtual switch)

Wittman · December 11, 2017, 12:00pm

hello, I have a problem when configuring the network at the LDOM level, on a SPARC-Enterprise-T5120 machine

The network goes through VLAN 503 (level, network tagget)
And I need to give network to the guest domain: madarrwebsol10
I hope someone can bring some light ....
Excuse my English ...

****

SunOS madarrlab01 5.11 11.3 sun4v sparc SUNW,SPARC-Enterprise-T5120

root@madarrlab01:~# ipadm
NAME              CLASS/TYPE STATE        UNDER      ADDR
lo0               loopback   ok           --         --
   lo0/v4         static     ok           --         127.0.0.1/8
   lo0/v6         static     ok           --         ::1/128
vlan503           ip         ok           --         --
   vlan503/v4     static     ok           --         172.22.206.154/23

VLAN is taggint for VLAN503 on net0

root@madarrlab01:~# dladm show-phys
LINK              MEDIA                STATE      SPEED  DUPLEX    DEVICE
net1              Ethernet             unknown    0      unknown   e1000g1
net2              Ethernet             unknown    0      unknown   e1000g2
net0              Ethernet             up         1000   full      e1000g0
net3              Ethernet             unknown    0      unknown   e1000g3

**************
root@madarrlab01:~# ldm list
NAME             STATE      FLAGS   CONS    VCPU  MEMORY   UTIL  UPTIME
primary          active     -n-cv-  SP      6     9504M    0.5%  12m
madarrwebsol10   inactive   ------          6     6644M

root@madarrlab01:~# dladm show-link
LINK                CLASS     MTU    STATE    OVER
net1                phys      1500   unknown  --
net2                phys      1500   unknown  --
net0                phys      1500   up       --
net3                phys      1500   unknown  --
vlan503             vlan      1500   up       net0

root@madarrlab01:~# ldm add-vsw  net-dev=net0 primary-vsw0 primary
root@madarrlab01:~# ldm add-vnet vnet1 primary-vsw0 madarrwebsol10

Now, agregate virtual vswitch

root@madarrlab01:~# dladm show-link
LINK                CLASS     MTU    STATE    OVER
net1                phys      1500   unknown  --
net2                phys      1500   unknown  --
net0                phys      1500   up       --
net3                phys      1500   unknown  --
net4                phys      1500   up       --
vlan503             vlan      1500   up       net0

root@madarrlab01:~# dladm show-phys
LINK              MEDIA                STATE      SPEED  DUPLEX    DEVICE
net1              Ethernet             unknown    0      unknown   e1000g1
net2              Ethernet             unknown    0      unknown   e1000g2
net0              Ethernet             up         1000   full      e1000g0
net3              Ethernet             unknown    0      unknown   e1000g3
net4              Ethernet             up         1000   full      vsw0

********************
ldm start-reconf primary
shutdown -y -g0 -i6

reboot and boot Guest Domain-.... with Solaris 10

View configuration LDOM:

ldm list-bindings

VSW
    NAME             MAC               NET-DEV   ID   DEVICE     LINKPROP   DEFAULT-VLAN-ID PVID VID                  MTU   MODE   INTER-VNET-LINK
    primary-vsw0     00:14:4f:f8:de:90 net0      0    switch@0              1               1                         1500         on   
        PEER                        MAC               PVID VID                  MTU   MAXBW      LINKPROP   INTERVNETLINK
        vnet1@madarrwebsol10        00:14:4f:f9:60:de 1                         1500

Guest Domain:

madarrwebsol10
NETWORK
    NAME             SERVICE                     ID   DEVICE     MAC               MODE   PVID VID                  MTU   MAXBW      LINKPROP
    vnet1            primary-vsw0@primary        0    network@0  00:14:4f:f9:60:de        1                         1500                
        PEER                        MAC               MODE   PVID VID                  MTU   MAXBW      LINKPROP
        primary-vsw0@primary        00:14:4f:f8:de:90        1                         1500

Now start domain....

root@madarrlab01:~# ldm bind madarrwebsol10
root@madarrlab01:~# ldm start-domain madarrwebsol10

root@madarrlab01:~# ldm list
NAME             STATE      FLAGS   CONS    VCPU  MEMORY   UTIL  UPTIME
primary          active     -n-cv-  SP      6     9504M    3.4%  9m
madarrwebsol10   active     -t----  5000    6     6644M     17%  11s

telnet 0 5000

login with root ..

and:

bash-3.2# dladm show-link
vnet0           tipo: no vlan   mtu: 1500       dispositivo: vnet0


bash-3.2# dladm show-dev
vnet0           v�-'nculo: up     velocidad: 0     Mbps   d�-'plex: unknown

bash-3.2# ifconfig -a
lo0: flags=2001000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv4,VIRTUAL> mtu 8232 index 1
        inet 127.0.0.1 netmask ff000000
vnet0: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 2
        inet 172.22.206.110 netmask fffffe00 broadcast 172.22.207.255
        ether 0:14:4f:f9:60:de

no ping 172.22.206.110

Any ideas???

Thanks!

Peasant · December 11, 2017, 11:30pm

Do not create vlan interface (dladm create-vlan ..).
Delete that interface.

If using VLAN tagging add the virtual switch in this manner :

ldm add-vsw net-dev=net0 pvid=1, vid=503 <human_name>-vsw0 primary

Then add tagged interface into guest domain :

ldm add-vnet pvid=503 vnet0 <human_name>-vsw0 madarrwebsol10

This way, tagging will be done by hypervisor vnet/vsw, not inside guest domain.
Guest domain will be in VLAN 503 exclusively.

Good practice would be to aggregate all four network ports into one aggr0 then use that for as net-dev.
You will get redundancy in case of failure, as well as combined bandwidth of 4 network ports.

Also, on that aggregated interface specify as many VLAN ids that are used in your environment when creating VSW.
The switch network ports should be configured as trunk.

Hope that helps
Regards
Peasant.

Wittman · December 12, 2017, 5:09am

So...

I can create vlan with aggr

dladm create-aggr -l net0 -l net1 aggr0
dladm modify-aggr -L active -T long aggr0
dladm create-vlan -l aggr0 -v 503 vlan503 
ipadm create-ip vlan503 
ipadm create-addr -T static -a 172.22.206.154/23

root@madarrlab01:~# dladm show-link
LINK                CLASS     MTU    STATE    OVER
net1                phys      1500   up       --
net2                phys      1500   unknown  --
net0                phys      1500   up       --
net3                phys      1500   unknown  --
net4                phys      1500   up       --
aggr0               aggr      1500   up       net0 net1
vlan503             vlan      1500   up       aggr0

root@madarrlab01:~# dladm show-phys
LINK              MEDIA                STATE      SPEED  DUPLEX    DEVICE
net1              Ethernet             up         1000   full      e1000g1
net2              Ethernet             unknown    0      unknown   e1000g2
net0              Ethernet             up         1000   full      e1000g0
net3              Ethernet             unknown    0      unknown   e1000g3
net4              Ethernet             up         1000   full      vsw0



root@madarrlab01:~# ldm add-vsw net-dev=aggr0 pvid=1 vid=503 madarrwebsol10-vsw0 primary

root@madarrlab01:~# ldm add-vnet pvid=503 vnet0 madarrwebsol10-vsw0 madarrwebsol10

reboot system.

and

root@madarrlab01:~# ldm list-config
factory-default
config2 [next poweron]
configuraci\0)+\//+n_1

root@madarrlab01:~# ldm list
NAME             STATE      FLAGS   CONS    VCPU  MEMORY   UTIL  UPTIME
primary          active     -n-cv-  SP      6     9504M     15%  4m
madarrwebsol10   inactive   ------          6     6644M

root@madarrlab01:~# ldm bind madarrwebsol10
root@madarrlab01:~# ldm start-domain madarrwebsol10
LDom madarrwebsol10 started

telnet 0 5000

bash-3.2# hostname
madarrwebsol10
bash-3.2#  dladm show-dev
vnet0           vnculo: up     velocidad: 0     Mbps   dplex: unknown

ifconfig vnet0 plumb up
ifconfig vnet0 172.22.206.110 netmask + 255.255.254.0

 ping 172.22.206.110

Haciendo ping a 172.22.206.110 con 32 bytes de datos:
Responding from 172.22.206.110: bytes=32 tiempo=48ms TTL=250
Responding from 172.22.206.110: bytes=32 tiempo=4ms TTL=250
Responding from 172.22.206.110: bytes=32 tiempo=5ms TTL=250
Responding from 172.22.206.110: bytes=32 tiempo=7ms TTL=250

IT WORKS!!!

Thanks guys!!

Peasant · December 12, 2017, 9:53am

Glad you got it working.

A couple of things you should notice.

You only need VLAN (dladm create-vlan) interface for the hypervisor IP address.
I would recommend using seperate VLAN id for hypervisor ip address (security wise).

Creating VSW with name of virtual machine is not proper naming.

Human is for instance : lab-vsw0, prod0-vsw or similar descriptive name.
You will use that virtual switch for other machines as well, so naming it after one ldom is not so straightforward.

Be careful about naming in Oracle VM server hypervisor, it will allow you to name anything you like as you like it.
When you have a lot of machines, naming policy will save you extra investigation what is mapped where (disks, vsw to ldom etc.)

Hope that helps
Regards
Peasant.

Wittman · December 12, 2017, 10:15am

I understand, I will change it.
Thank you again for the help.
Regards