I'm writing an application (using BSD sockets on a Linux host) which communicates over TCP/IP with an embedded device. This embedded device has an old and real slow integrated circuit (Epson S1S6000) which handles all of the TCP/IP communication for it.
Problem is, this circuit (S1S6000) advertises a TCP window size of 536 bytes and while analyzing the traffic with wireshark I can see that my application (or the Linux TCP stack) only transmits packets of half that size. This results in a lot of unnecessary transmission overhead which slows down the communication quite a bit.
Do you know how to change this behavior? Maybe some socket option, or TCP stack setting?
It sounds like you turned off the nyquist algorythm stesockopt TCP level TCP_NODELAY, which is good, so it does not hold data in the buffer for a bit speculating there might be more. Your problem is that you are writing or sending small parts, and need to send the whole message at once, perhaps using a fdopen FILE* and fflush().
Note that by making full packets, you are creating delay. If the first small packet overhead is so great, the packets should be full from that point on. I hope you set the setsockopt SO_SNDBUF big, so it can accumulate enough to keep sending full packets.
Of course, if you could increase the window even just to 2920 ( 2 x ( 1500 - 40 )), that would be grand, allowing you to send two full ethernet packets before blocking.
mcnamara: No, there is unfortunately no way to get around using another chip.
And I should have mentioned that the MSS given from this device/chip is at 536 bytes. And this communication takes place at a local network. And there are no ICMP messages indicating that this MSS is set too high.
From a Windows application, which too communicates with this device, the packet size used is 536 bytes (well, only when needed of course). The chip is terrible slow anyway, but doing all this extra ACK'ing is making it much worse.
I'm passing buffers to send() which, some of them, are quite a lot bigger than 536 bytes and still the biggest TCP packet seen has a data length of 268 bytes. And as said the window size announced is always at 536 bytes from the chip. Still I fail to see how this makes any sense.
DGPickett: I can see that send() is swallowing chunks bigger than 268 bytes. I have tried playing around with TCP options like NODELAY, and changing socket buffer sizes. And have tried to change some of the settings found at /proc/sys/net related to TCP, but still nothing have changed this behavior.
Maybe it has an MTU of less. Obsolete IP option 12 can send this during connect without the ICMP of PATH MTU Discovery. IP, Internet Protocol
The MSS is the total remote buffer, the MTU is the packet max. Disregarding the chance of TCP options, a packet header is 20 IP + 20 TCP, so a packet sends MTU - 40, and the number of packets sent before ack is MSS/(MTU-40) including any partial. Since this is local, the ack round trip should be very short, so packets should not be MSS trimmed unless there is blocking downstream.
Well, I don't find anything in the IP header(s) which looks like IP option 12. Have attached a small wireshark dump, if you would like to have look though.
If it is to any help, the MTU of the interface at my host machine is set to 1500. And regarding the chip, I know it can handle a MTU of 536 + headers, as I can see the Windows application send it packets of this size (from the same machine too).
Verified that the same behavior exists under Debian (with a 2.6.35 kernel). And under Arch Linux (with a 2.6.34 kernel).
And verified that the same application executed under a windows host (well, cygwin) does send TCP packets of 536 bytes when needed.
---------- Post updated at 06:12 PM ---------- Previous update was at 02:34 PM ----------
If anyone is interested...
Found the reason of the behavior after digging around in the TCP code of Linux. Found this function which limits the mss to half the window size (tcp_bound_to_half_wnd()). Well, it actually avoid this limiting if the max window size is less than 512, unfortunately my window is a tiny bit bigger than that so it doesn't apply to this rule.
I guess there is a line of thought that you want to have balanced double buffering, not irregular sized packets, so if the MSS gets low, half the MSS per packet is a balanced double buffer. However, the bigger packet not double buffered is same or faster and has lower latency for the data at the end of the packet.
