Score:1

Why NIC ring parameters are not pre-set at their Hardware max capabilities?

us flag

Check the NIC ring buffer:

# ethtool -g eth0
Ring parameters for eth0:
Pre-set maximums:
RX:        4096
RX Mini:      0
RX Jumbo:     0
TX:        4096
Current hardware settings:
RX:         256
RX Mini:      0
RX Jumbo:     0
TX:         256

One can set "RX/TX" up to the limit shown in the "Pre-set maximums" like:

# ethtool -G eth0 rx 4096 rx 4096

Question is: by default;, why are these set so low (in every server I have, all of them are at 256) instead of a higher value, or their Hardware max capabilities? Is there any drawbacks (if yes, wich?) increasing these values?

ewwhite avatar
ng flag
Good question!!
Score:1
cn flag

First, numbers you set aren't in bytes as many people think, they are in descriptors (and the descriptor size is hardware-dependent). So, when you increase ring length you request more memory to be allocated in kernel for these descriptors. In general, you want this kernel memory in L1 cache so that interrupt processing will be as fast as possible. Increasing ring size makes this less probable and in some cases completely impossible.

Next thing is interrupt coalescing - in general, when you increase ring buffer size, the NIC will adjust its low/high marks appropriately and will trigger interrupt when more data is buffered (that is less often). Time required by kernel to process these larger amounts of data during interrupt processing would also increase as the result.

All of the above results in a simple bucket effect - with a larger ring packet drop probability decreases and network latency increases. This may be perfectly fine if you're streaming large files over TCP and may be completely undesirable if you're a low-latency small packet application (i.e. gaming and such).

Default numbers you see are reasonable trade-off between the two.

CrazyRabbit avatar
us flag
Thanks for this explanation. I'm using an application to serve files over TCP and was having fifo erros so based on this https://serverfault.com/questions/650596/meaning-of-rx-queue-csum-err-and-rx-fifo-errors i've increased it. Is there any way, other than trial and error, to increase this to a "good value" between the 256 default and the 4096 max accepted value?
Peter Zhabin avatar
cn flag
@user1773988, for a specific workload for which packet size is known we use `iperf` with appropriate parameters to emulate load and see performance and system CPU load (which is also affected by this setting) to find a sweet spot. We once had to set these on a bunch of Linux firewalls in a role of poor man's BRAS in an ISP environment and that's where trial and error is the only method..
CrazyRabbit avatar
us flag
Thanks for you input. Subsidiary question; can these be decreased bellow 256? Let's say a Redis server, small packets, dedicated NIC, havig rx and tx set to 128?
Peter Zhabin avatar
cn flag
@user1773988 this can be lowered up to 48 AFAIR, but packet drop probabilities will increase esp in case of an unexpected pps bursts. Redis is a TCP application and drops are not going to do any good. I don't have first-hand experience with low latency stuff, just from ISP perspective, so we were almost always rising these values but to a certain point based on 24h monitoring stats.
mangohost

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.