Solaris 11.4 setting static ports for NFSv3

Hi Folks,

I've been trying to configure a Zone running Solaris 11.4 on a T8-1 to run the NFSv3 client on static ports, in the good old days you would have setup the /etc/defaults/rpcbind and the /etc/defaults/nfs files up with the required values and bob would have been your uncle!

However it seems that this now has to be done using the svccfg command where the properties are set using the "setprop" option in the "nfs/client", "nfs/lockmgr" and the "nfs/status" services.

I have tried the svccfg command with;

"setprop client_config/nfs_port = 2049"

This returns

"No such property group client_config"

• are there any precursor steps that I'm missing?

Regards

Dave

Certainly there is a svcprop or similar command that dumps the current keys and values...
Then simply use these key names.
And use svcs to dump the active service names.

Hi Dave,

We haven't seen you on here for quite a while. Welcome back.

I've read your post a few times but I'm not getting a handle on this. It's also been quite a while since I did this myself.

So the T8-1 is running 11.4 as Global?

The Global has it's network interface configured and working?

The intended Zone will share that interface or will have a dedicated interface configured?

The Zone network interface is configured and working? Can ping, etc?

Please elaborate to give some clarity as to where you're at. Thanks.

Hi,

Yes, been away from the keyboard for a while but have been asked to help with a security issue by an old friend. Don't really want to write a tomb of a post, so here is a heads up on the config of the test server.

Hardware is a T8-1, running Solaris 11.4 (January Patches) and latest FW.

There is a Control domain which is also the I/O domain and five guest domains, these guest domains have 2 or three zones each with an exclusive IP Address for Network and an exclusive IP Address for the NFS Disk.

Each Zone has an iSCSI disk of 60Gb for the OS install and an NFS disk of varying size for the application workload. The iSCSI disks are presented through the Control domain and on to the guest domains in the normal manner, but the NFS disks are mounted into the zones directly - not from the control domain. With all disk being presented from a NetApp appliance, whch seems to be where the issue with the Illumio product arrises.

As NFSv3 is in use for all the client mounts and is stateless and Solaris 11.4 uses packet filtering we have an issue which revolves around the port numbers changing when the software is activated. This is why I was trying to work out how to configure the "nfs/client", "nfs/nlockmgr" and the "nfs/status" services to use a static port when communicating with the NetApp.

The old fashioned way of doing this would have been to configure the ports in "/etc/default/nfs" and "/etc/default/rpcbind" and it would have looked something like;

root@test-dba:/etc/default# cat nfs

#

# Copyright (c) 2001, 2016, Oracle and/or its affiliates. All rights reserved.

#

 

#

# Moved to SMF. Use sharectl(8) to manage NFS properties.

#

root@test-dba:/etc/default# cat nfs_new

#

# Copyright (c) 2001, 2016, Oracle and/or its affiliates. All rights reserved.

#

 

#

# Moved to SMF. Use sharectl(8) to manage NFS properties.

#

# Added as a just incase for Illumio-VEN - as NFSv3 is problematic and needs to be locked to certain ports.

#

NFS_PORT=2049

MOUNTD_PORT=635

STATD_PORT=4046

LOCKD_PORT=4045

root@test-dba:/etc/default# cat rpcbind

# This file created by Dave Munro to allow the use of static ports on NFSv3

RPCBIND_STATIC_PORTS=YES

root@test-dba:/etc/default#

Now it seems that you have to configure the individual services, which is where I've become a bit unstuck - it's not intuitive for me - but then I'm grumpy and old

So what I've tried now is;

root@test-dba:~# svccfg -s svc:/network/nfs/client addpg client application

root@test-dba:~# svccfg -s svc:/network/nfs/client setprop client/mount_options = astring: "-o port=2049"

root@test-dba:~# svccfg -s svc:/network/nfs/client refresh

root@test-dba:~# svcprop -p client/mount_options svc:/network/nfs/client

-o\ port=2049

root@test-dba:~# svcadm restart nfs/client

root@test-dba:~# svcs -xv

root@test-dba:~# rpcinfo -p

   program vers proto   port  service

    100000    4   tcp    111  rpcbind

    100000    3   tcp    111  rpcbind

    100000    2   tcp    111  rpcbind

    100000    4   udp    111  rpcbind

    100000    3   udp    111  rpcbind

    100000    2   udp    111  rpcbind

    100024    1   udp  49246  status

    100024    1   tcp  48248  status

    100133    1   udp  49246

    100133    1   tcp  48248

    100021    1   udp   4045  nlockmgr

    100021    2   udp   4045  nlockmgr

    100021    3   udp   4045  nlockmgr

    100021    4   udp   4045  nlockmgr

    100021    1   tcp   4045  nlockmgr

    100021    2   tcp   4045  nlockmgr

    100021    3   tcp   4045  nlockmgr

    100021    4   tcp   4045  nlockmgr

1073741824    1   tcp  38690

root@test-dba:~# nfsstat -c | grep Version

Version 2: (0 calls)

Version 3: (58 calls)

Version 4.0: (0 calls)

Version 4.0: (0 operations)

Version 4.1: (0 calls)

Version 4.1: (0 operations)

Version 2: (0 calls)

Version 3: (0 calls)

root@test-dba:~# df -k /app

Filesystem           1024-blocks        Used   Available Capacity  Mounted on

172.17.48.6:/dba_test_area2

                       209715204      195120   209520084     1%    /app

root@test-dba:~# umount /app

root@test-dba:~# mountall

root@test-dba:~# df -k /app

Filesystem           1024-blocks        Used   Available Capacity  Mounted on

172.17.48.6:/dba_test_area2

                       209715204      195120   209520084     1%    /app

root@test-dba:~# rpcinfo -p

   program vers proto   port  service

    100000    4   tcp    111  rpcbind

    100000    3   tcp    111  rpcbind

    100000    2   tcp    111  rpcbind

    100000    4   udp    111  rpcbind

    100000    3   udp    111  rpcbind

    100000    2   udp    111  rpcbind

    100024    1   udp  49246  status

    100024    1   tcp  48248  status

    100133    1   udp  49246

    100133    1   tcp  48248

    100021    1   udp   4045  nlockmgr

    100021    2   udp   4045  nlockmgr

    100021    3   udp   4045  nlockmgr

    100021    4   udp   4045  nlockmgr

    100021    1   tcp   4045  nlockmgr

    100021    2   tcp   4045  nlockmgr

    100021    3   tcp   4045  nlockmgr

    100021    4   tcp   4045  nlockmgr

1073741824    1   tcp  38690

root@test-dba:~#

But what I need is some assistance to ensure that I get all the required services modified, why I hear some of you saying. Well this is where it gets a bit scarey from a Solaris perspective, a failure of the configuration takes down the whole server - communication to everything except the primary LDOM is lost.

You cannot connect to any Guest LDOM's even using telnet to the console doesn't work, you cant ssh to any zones, cant zlogin to any zones - you have a simple option reboot the lot.

I have never seen anything inside a zone kill it's host LDOM other than resource usage, this actually kills all zones whether in the same LDOM or not and all LDOM's except the control LDOM - so I'd like to be as ready as I can for the testing and have a reasonable degree of confidence in not killing the test box (also used by dev people which is where the problem lies) again.

Regards

Dave

Yea, I can appreciate why you may well be having trouble with this one! Couldn't you pick something easier?

I always see red when such configurations rely on EMC or NetApp SAN storage. I always find that both these manufacturers require such specific and idiosyncratic configurations that they publish "Connectivity Guides" pdf's that take about a fortnight just to read (albeit extremely well documented instruction).

I guess that you've search for "NetApp Solaris connectivity guide" and the like and read it all. Specifically it seems that NetApp supply host based tools for Solaris:

e.g.

I assume that the NetApp OS is ONTAP?

Let me think some more and I'll post again.

PS. Certainly press the "pdfs" button on that page for more info.

Hi Dennis,

Thanks very much for that link, it will I'm sure be very handy if this goes much further. The disk isn't where this issue lies, it is definately the network and the Illumio-VEN software which is the problem area.

So when the Illumio-VEN is set to "Enforcement Mode" it applies a set of packet filter rules in Solaris 11.4, these are collected while the product is in "Visibility Mode" and are used to create a policy then applied to the packet filter.

All the OS disks with the exception of the primary domain (Internal Disk) are iSCSI and are on a specific VLAN (VLAN300), at the moment it looks like there are no issues here.

As a proof of concept I built a standard zone, on the LDOM system disk - just vanilla solaris installed the Illumio-VEN and moved it through "Visibility Mode" to "Enforcement Mode" without issue - proving that the Illumio-VEN software did what it said on the tin.

The process was then repeated, this time allocating an iSCSI disk for the zone - again without any issue. At this point I introduced the NFSv3 disk, here is where things got interesting and scary at the same time.

After moving the Illumio-VEN client into "Enforcement Mode" the system hung for want of a better word, in the morning I logged onto the primary LDOM - just standard ssh through a putty session. I could run "ldm ls" and all looked OK, no excessive load nothing in any of the logs that indicated a problem. But you couldn't login to any of the other LDOM's or Zones, there were no error messages - "telnet 0 5001" or any other console just hung, all zones just timed out when you tried to ssh to them.

As the T8-1 is also a development environment, is some respects it is also "quasi-production" - getting it back into service took precedence.

So what did I learn about this setup, well the culprit is almost certainly the statelessness of NFSv3 - but the effect is where it got scary. I've worked with zones in LDOM's for probably 20 years and have never seen something running in a zone get out of it. In this case it seemed to impact the whole IP stack, with the exception of the primary LDOM.

So to progress this, I need to get NFSv3 to run as statefull i.e. always on the same port with the same function - which you used to do with the /etc/default/nfs file and the /etc/default/rpcbind file. Then edit the /etc/services file to define the ports and reboot, now it seems that the configuration needs to be in a service manifest and under control of the SMF. I'll dig a bit further and test a bit more, then post the findings back here.

Best Regards

Dave

Hi Dave,

Thanks for the additional info. I'm reading the whole thing from the top again and trying to picture the whole configuration.

So the NetApp is publishing both the iSCSI targets (for the control domain) and is the NFSv3 server (for the individual zones)?

So starting a NFS client within a zone is where the trouble starts? Could it be that the NFS client interferes somehow with the block access to the iSCSI storage disk containing the zone's O/S? That's what is sounds like. What do you think?

A stupid question perhaps, (sorry, I know that you are well experienced) but have you checked that the NFS client and NFS server versions (2,3 or 4) are the same?

To crash the zones dead surely we have to assume that either Illumio or NFS is interfering with iSCSI traffic thereby the zone O/S is stopped.

Dennis

Hmmm........could the problem be the NetApp configuration?

This talks about iSCSI timeouts.

I'm still thinking!

Yea, my suspicion would be that, in order for the zone to crash, i.e. lose communication with it's iSCSI O/S disk, something is interfering with iSCSI traffic. Such an iSCSI connected disk can be regarded for all intents and purposes, to be a local drive (as if it was within the local cabinet) albeit on a long SCSI cable between the mobo and the drive (i.e. over the network). So there's no way NFS (or any app) should be able to interfere with that communication. So, at the moment, that's my thinking.

Unfortunatly i no longer have access to Solaris 11.x at all and docs seem to be getting worst and worst

But from reading a bit docs looks like you will need to use (in non-global zone) for NFSv3 over UDP
sharectl set -p client_versmin=3 nfs # and max
sharectl set -p resvport=on nfs
Modify /etc/netconfig to include reserved ports option - from what i'm seeing in docs (0-1023 baked in?)
Mount NFS using resvport option.

In any case, NFS should not break your infra in such ways, so i doubt that IDS software is doing something highly invasive, such as blocking complete network access - hence nothing works network wise.
You should be able to confirm such cutoff in somekind of logs or dashboard on server side of Illumio-VEN or even stop / change to monitor mode of such agent remotely when incident occurs ?

If you leave that software in monitor mode for longer time, does the 'policy' file to be applied changes over time?

Regards
Peasant.

Hi Folks,

Just an update on this to let you all know that I'm still looking at it. A couple of things have come to light during my investigations, but the current situation is this.

@Peasant Thanks for the pointer, I'd setup the versions max and min previously - that was all fine but was unaware of the resvport option which I tried only to find out that the changes in 11.4 where the port range is baked into the kernel also deprecated the "resvport" option.

@hicksd8 Thanks for the link I'll follow up on that a bit more, when I can access the NetApp - also the NetApp is quite happy with NFSv3 or v4.1 and seems to function OK. The reason I haven't just moved it all to NFSv4.1 is that the Oracle DBA's are telling me it would create too much work at their end - although I was unaware that the Oracle RDBMS was protocol aware - but seemingly it is.

I've been capturing traffic data using snoop and tshark to build up a picture, which I'm hoping to use to assist in the writing of the next Illumio Policy. The reason that I haven't been proactive with updates is that there is a Change Freeze at the bank until Monday, so once I've resumed the testing proper I'll post another update.

Regards

Dave

UDP NFSv3 will be faster then TCP based NFSv4, just due to nature of TCP itself and different implementation of NFS

This is why probably oracle db does not like it or recommends v3 and requires additional tune and/or options for V4.

Probably oracle is aware in case it checks mountpoint on which database files live.

Regards
Peasant.

NFSv3 was optimized for tcp, so tcp is not slower.
In Solaris the NFSv3 transport defaults to tcp, only the mounting defaults to udp.

Mea aculpa for wrong information about NFSv3 UDP usage.

But i still stand on claim that UDP in non congested networks would be faster then TCP for NFSv3 traffic.
Probably negligible but still present, i wonder how much would it scale with bandwidth.

I can give this a shot using VM NFS server <-> VM NFS client on my PC over openvswitch and multiple vlans, if there is interest from other subjects in this discussion.

Regards
Peasant.