I have identical M5000 machines that are needing to transfer very large amounts of data between them. These are fully loaded machines, and I've already checked IO, memory usage, etc... I get poor network performance even when the machines are idle or copying via loopback. The 10 GB NICs are setup in aggregates, so we should be seeing some serious performance out of these. However, I'm only seeing about 30 MB/sec. These are brand new Solaris 11.1 installs, and fully patched.
I'm getting 30 MB/sec no matter which connection I use, whether it's 1 GB and 10 GB connections, physical, VLAN, etc...
richardc@SERVERX:/kernel/drv$ dladm show-phys
LINK MEDIA STATE SPEED DUPLEX DEVICE
net1 Ethernet unknown 0 unknown bge1
net6 Ethernet up 10000 full ixgbe0
net0 Ethernet unknown 0 unknown bge0
net4 Ethernet up 1000 full nxge2
net2 Ethernet unknown 0 unknown nxge0
net5 Ethernet unknown 0 unknown nxge3
net7 Ethernet up 10000 full ixgbe1
net3 Ethernet up 1000 full nxge1
richardc@SERVERX:/kernel/drv$ dladm
LINK CLASS MTU STATE OVER
net1 phys 1500 unknown --
net6 phys 1500 up --
net0 phys 1500 unknown --
net4 phys 1500 up --
net2 phys 1500 unknown --
net5 phys 1500 unknown --
net7 phys 1500 up --
net3 phys 1500 up --
prim1 aggr 1500 up net3 net4
bkup1 aggr 1500 up net6 net7
rep1 vlan 1500 up bkup1
richardc@SERVERX:/kernel/drv$ ipadm
NAME CLASS/TYPE STATE UNDER ADDR
bkup1 ip ok -- --
bkup1/v4 static ok -- XXX.XX.235.4/22
lo0 loopback ok -- --
lo0/v4 static ok -- 127.0.0.1/8
prim1 ip ok -- --
prim1/v4 static ok -- XXX.XX.225.244/22
rep1 ip ok -- --
rep1/v4 static ok -- XXX.XX.198.176/25
sppp0 ip ok -- --
richardc@SERVERX:/kernel/drv$ ifconfig -a
lo0: flags=2001000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv4,VIRTUAL> mtu 8232 index 1
inet 127.0.0.1 netmask ff000000
prim1: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 2
inet XXX.XX.225.244 netmask fffffc00 broadcast XXX.XX.227.255
bkup1: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 3
inet XXX.XX.235.4 netmask fffffc00 broadcast XXX.XX.235.255
rep1: flags=201000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4,CoS> mtu 1500 index 4
inet XXX.XX.198.176 netmask ffffff80 broadcast XXX.XX.198.255
sppp0: flags=10010008d1<UP,POINTOPOINT,RUNNING,NOARP,MULTICAST,IPv4,FIXEDMTU> mtu 1500 index 5
inet 10.1.1.3 --> 10.1.1.1 netmask ff000000
lo0: flags=2002000848<LOOPBACK,RUNNING,MULTICAST,IPv6,VIRTUAL> mtu 8252 index 1
inet6 ::/128
prim1: flags=20002000840<RUNNING,MULTICAST,IPv6> mtu 1500 index 2
inet6 ::/0
bkup1: flags=20002000840<RUNNING,MULTICAST,IPv6> mtu 1500 index 3
inet6 ::/0
rep1: flags=20202000840<RUNNING,MULTICAST,IPv6,CoS> mtu 1500 index 4
inet6 ::/0
richardc@SERVERX:/kernel/drv$
For example, when doing an SCP of a 200 MB file it takes 8 seconds and runs at 30 MB/sec:
root@SERVERX:/var/tmp# dlstat -rt -i 1 | grep rep
rep1 545.27K 314.76M 177.00K 32.18M
rep1 0 0 0 0
rep1 0 0 0 0
rep1 0 0 0 0
rep1 6 376 0 0
rep1 1.45K 110.42K 919 15.09M
rep1 2.84K 215.39K 1.80K 29.69M
rep1 3.03K 229.36K 1.92K 31.66M
rep1 3.03K 229.50K 1.69K 31.45M
rep1 2.92K 220.44K 1.50K 30.17M
rep1 2.97K 225.20K 1.74K 30.96M
rep1 2.93K 221.15K 1.34K 30.25M
rep1 2.89K 218.22K 1.35K 29.67M
rep1 789 59.82K 306 8.01M
rep1 0 0 0 0
rep1 0 0 0 0
rep1 0 0 0 0
I figured maybe it was something to do with the network, but I see this slow speed no matter which interface I use -- even localhost. I tried doing a local SCP back to the same machine, and it's taking 8 seconds there too even though it isn't actually hitting the network cards. If I do it as a regular CP it copies the file in less than a second.
richardc@SERVERX:/var/tmp$ scp esf40_20111108.iso 127.0.0.1:/var/tmp/test
Password:
esf40_20111108.iso 100% |************************************************************************************| 224 MB 00:08
richardc@SERVERX:/var/tmp$
Does anyone have any ideas on how to improve the performance? I figure I can't be the only one who has run into this, but my Google searches haven't turned up anything.
