Network Perforamnce - It's just slow!

Hello everyone,

I've been a life long Unix/Linux user but I'll be the first to admit I have little specific AIX knowledge at this point and I've inherited these systems for better or worse so please forgive if I ask something in the wrong context. And yes, I've searched google for 3 days now

I have a bunch of Power755 machines running AIX 6.1.0.0. I have multiple FC adapters to storage and 10GbE ethernet.

On the very same network, switch and storage I also have a bunch of x86 machines running RHEL75, multiple FC adapters to storage and 10GbE ethernet.

On the Linux boxes I can copy files (scp, ftp, nfs, rsync) at about 150MB/s.

On the AIX boxes those same copies are running around 40MB/s.

And, I can't figure out why???

I've been though several network tuning walkthru's found on google and nothing seems to make any difference.

Obviously the Power755's are 8 year old hardware where there x86 machines are circa 2017/2018 hardware and the AIX boxes only have 32GB of RAM where my x86 stuff has 256GB and 512GB. But, does that really account for the difference in network performance? I just have a hard time believing that IBM would put out a piece of iron that doesn't perform of up the published specs...

Anyway, if someone can help point me in the direction to track this down I'd much appreciate it.

The FC cards are all same speed, 8Gbps, there's plenty of flash and spindles in the NetApp that I know there's no bottle neck there. We've swapped ethernet adapters, cables, swtichports, etc. and I'm left with little choice but to draw the conclusion that somehow these AIX boxes are configured incorrectly. Oh, and yes, the ethernet ports �show' they are up at 10GB.

Thanks in advance,

-brian

What make/model are the AIX box FC storage adapters? Have you investigated the BIOS configuration of those?

The on-board configuration of FC storage adapters is usually critical for good transfer performance (but you probably know that).

Have you tested the storage subsystem performance?

Thanks for hte reply, I have not probed the FC BIOS or configuration but I have done a quick test to eliminate it from the equation and taken everything to local i/o, off the external storage - just for sanity sake.

linux_nsd_server# scp /large_file_on_fast_storage.bin linux_10gb_network_client:/dev/null == 154MB/s
linux_nsd_server# scp /large_file_on_fast_storage.bin aix_10gb_network_client:/dev/null == 39MB/s

So taking all the client network storage i/o out of the picture and this is still repeatable with very similar numbers across all my Linux and AIX boxes.

-b

So you seem to be convinced that this is an ethernet problem. What have you checked?

What about maximum payload configuration? I've seen major issues caused by payload being set to 1502 instead of 1504, for example.
Are some boxes configured to use jumbo packets and others not?
I assume a single chattering ethernet adapter cannot be the issue because the problem is widespread across multiple boxes.
Have you tried putting a wireshark on the traffic to see what's going on?
Do your switches have LED's that tell you what speed has been negotiated?
Have you checked the spec of the ethernet adapters to see what they are capable of?
Stupid question: I assume there is no MS-Windows influence in this at all as they always have network issues with very large files?
Do you have one network device that doesn't handle jumbo packets?

Sorry, more questions than answers I know. Please give more information if you can.