Tuning Amazon Linux for TCP Handshake, receiving 200-300KB response, and then closing connection

Cory Kendall

5/2/24, 12:01 AM

I have an application which regularly creates a new TCP connection, makes a request, and then gets a just under ~300KB response before closing down the connection. Assuming that I can't redesign this workload, and that the latency between my side and the other side of the connection is ~5ms, how can I optimize for performance?

From a packet capture, I can see:

SYN-SYNACK-ACK handshake (~15ms)
I send a small request (<1KB), they ACK it (~10ms)
They begin sending the 300KB response
They fill the TCP Window with ~30K bytes (~5 ms for first byte, but then only 0.05ms for me to get all 30K bytes)
They pause while they're waiting for my ACKs to reach them, then fill the new window (now ~50K as I'm advertising in my ACK)
They fill that window and pause...
...

So to solve this situation, I'm considering updating net.ipv4.tcp_rmem from the defaults:

net.ipv4.tcp_rmem = 4096        87380   6291456

to something like:

net.ipv4.tcp_rmem = 307200      307200   6291456

But I'm not really sure of the implications this would have on the system. Assume that I have many Gigs of ram free in steady state. What are the downsides to a change like this? If I understand the memory usage of TCP, this will cause me to advertise a 300KB window on my very first ack, and assuming the other side is ready to send 300KB at once, they will.

1 + 0

tcp

linux-networking

ipv4

Score:1

Server

user1018817

5/2/24, 1:25 AM

I think TCP slow start will still limit your receive window as it starts.

Is it possible to change the app to reuse the TCP connections? That would be the best solution.

Alternatively, you could try adding a local HTTP proxy. That would have 0ms latency locally, and then could reuse TCP connections to your remote server.

+ 1

Cory Kendall

5/2/24, 10:09 PM

Reuse is possible, but I'm making the requests from a fleet of 1000+ hosts, the customer is calling a shared resource earlier that fans out to these 1000+ hosts which would be difficult to rearchitect, and the 1000+ hosts are multitenant, meaning they are individually serving ~300 requests per second which are all making connections to different endpoints. Connection reuse is challenging because the connections to this endpoint are only going to be ~20 requests per second, and each one of those 20 will naturally be made from a unique host in the pool of 1000+.

Elon Musk

I sit in a Tesla and translated this thread with Ai:

EN: Tuning Amazon Linux for TCP Handshake, receiving 200-300KB response, and then closing connection

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.