Our org uses the MOVEIT application to upload files from an AIX v7.3.2.1 server to the MOVEIT Windows server. One particular job was taking over 18 hours to upload 51 files at approximately 9.1GB in size from the AIX server to the MOVEIT server. Using WireShark, it was determined that the AIX server was sending segments exceeding the 9000 jumbo frames standard setting. We verified that jumbo frames are allowed across our network on all routers.We turned off mtu_bypass and this cut the job to just under 3 hours.
We then used a smaller 187MB file to run the following tests:
With mtu bypass on and mtu at 1500 this upload took 27 mins.
With mtu bypass off and mtu at 1500, this upload then took 8 mins.
With mtu bypass off and mtu at 9000, this upload took 2.5 mins.
For upload 2 & 3, the WireShark capture files show no relative increase in the size of packets being sent. They seemed to stay under the 1500 limit. My peer and I are infants in using WireShark and analyzing the results. We are hoping someone can help us understand the following:
Why the AIX server wasn't sending larger files with its MTU higher?
Why it was faster even when it appears to be applying the same MTU size limit?
FWIW, and yes, I'm biased, IMHO, you cannot do any performance measurement when you have Windows in the equation! (You're on a Unix/Linux forum here).
I would very much be inclined to install Moveit onto another Linux box and test the Linux to Linux transfer speed(s), especially on very long multi-file-sized, multi-file transfers.
My experience is that on such big transfers Windows will start off fast and then throttle back more and more as time goes on and can get very slow.
Also, certainly it's no good comparing a small, single file transfer of 187MB with a multi-file 9.1GB transfer because Windows won't throttle a small file transfer.
Alternatively, can you use a tool that displays real-time transfer rate (MB/sec or the like) so you can monitor whether it starts off fast and then throttles?
I repeat, can you do a Linux to Linux test?
Of course, in all cases, the MTU packet size should be set to the same size at both ends of the transfer (in addition to all nodes/switches in between accommodating that size).
First of all, networking relies on protocols, and not all file transfer protocols operate the same way. You mentioned using MOVEit, @Kentlee65, but didn’t specify the protocol. This is important because MOVEit supports several secure file transfer protocols like FTPS, SFTP, and HTTPS, all of which use TCP/IP as their foundation. Assuming TCP/IP is in play, let’s focus on MTU behavior.
1. Why the AIX server wasn't sending larger files with a higher MTU
The Maximum Transmission Unit (MTU) represents the largest packet size that can traverse the network without fragmentation. However, having a high MTU setting does not guarantee it will be used if certain conditions aren’t met:
Path MTU Discovery (PMTUD):
PMTUD is the standard mechanism in TCP/IP networks to determine the effective MTU along the path between two devices.
If any intermediate router or device in the network path does not support jumbo frames (MTU > 1500 bytes), the path’s effective MTU is reduced to the smallest supported MTU.
ICMP Blockage:
PMTUD relies on ICMP "Fragmentation Needed" messages to adjust the MTU dynamically. If these messages are blocked (e.g., by firewalls), the AIX server cannot properly discover the effective MTU and might default to conservative values.
MTU Bypass Issues:
With mtu_bypass enabled, the AIX server may incorrectly assume the network supports larger packets without proper validation, causing inefficiencies or packet loss if the path MTU is smaller.
Segment-Level Limitations:
Even with a high MTU, application-layer protocols or specific software (like MOVEit) might impose their own limits on segment sizes, capping packet sizes below the MTU.
2. Why it was faster with the same MTU size limit
Even when packet sizes remained under the same MTU limit, other factors likely improved performance:
Disabling mtu_bypass:
With mtu_bypass disabled, the AIX server adheres to proper TCP segmentation and PMTUD processes. This prevents inefficiencies like retransmissions caused by oversized packets exceeding the path MTU.
Reduction in Fragmentation:
If the network was fragmenting packets due to an unsupported MTU, disabling mtu_bypass ensures packet sizes align with the actual MTU, avoiding the overhead of fragmentation and reassembly.
Improved Flow Control:
Properly segmented packets allow TCP to manage flow control and congestion avoidance more effectively, leading to faster data transfers.
Reduced Retransmissions:
Oversized packets that exceed the path MTU often result in dropped packets and retransmissions. By adhering to the correct MTU, packet loss is minimized, and transfer speeds increase.
Testing and Validation
You can manually test and determine the effective MTU along the path using the ping command with the DF flag set. This ensures that the largest packet size is transmitted without fragmentation:
ping -s 1472 -M do <destination>
-s: Packet size (1472 bytes here represents a 1500-byte MTU with 28 bytes for IP/TCP headers).
-M do: "Don't Fragment" flag.
If the packet gets through, increase the size until it fragments, then reduce to find the path MTU.
Note: Not all versions of ping support this option.
Critical Network Context Missing
To fully answer your questions, it’s essential to know:
Are the AIX server and MOVEit server on the same LAN segment, or are they geographically separated, crossing multiple hops or routers?
Is ICMP traffic blocked along the path?
Are jumbo frames (MTU > 1500) consistently supported across all routers, switches, and interfaces?
Without this information, we can only provide general advice about MTU behavior and TCP/IP interactions.
Summary
Your issue @Kentlee65 likely stems from misaligned MTU expectations, improper PMTUD behavior, or inefficiencies caused by mtu_bypass. Ensuring proper PMTUD operation, validating MTU support end-to-end, and disabling mtu_bypass to let TCP handle segmentation properly were all steps in the right direction.
If you can provide more details about the network topology, protocols used, and configuration, we can refine the analysis further!