Score:2

Kernel TCP tuning for buffer overrun and packet collapse

cn flag

We have a Nginx+Ruby application where the data between Nginx and the application are interfaced through a socket.

As part of performance improvement, when we analyzed the netstat statistics on the application side we saw these:

    5334 packets pruned from receive queue because of socket buffer overrun
    2299951 packets collapsed in receive queue due to low socket buffer
    227365 delayed acks sent

Assuming the rmem buffer capacity is the issue we find that:

$ sysctl net.ipv4.tcp_rmem
net.ipv4.tcp_rmem = 4096    87380   6291456

However, looking at cat /proc/net/protocols:

protocol  size sockets  memory press maxhdr  slab module     cl co di ac io in de sh ss gs se re sp bi br ha uh gp em
PINGv6    1120      0      -1   NI       0   yes  ipv6        y  y  y  n  n  y  n  n  y  y  y  y  n  y  y  y  y  y  n
RAWv6     1120      0      -1   NI       0   yes  ipv6        y  y  y  n  y  y  y  n  y  y  y  y  n  y  y  y  y  n  n
UDPLITEv6 1280      0       3   NI       0   yes  ipv6        y  y  y  n  y  y  y  n  y  y  y  y  n  n  n  y  y  y  n
UDPv6     1280      0       3   NI       0   yes  ipv6        y  y  y  n  y  y  y  n  y  y  y  y  n  n  n  y  y  y  n
TCPv6     2152      4      52   no     256   yes  ipv6        y  y  y  y  y  y  y  y  y  y  y  y  y  n  y  y  y  y  y
PACKET    1408      0      -1   NI       0   no   kernel      n  n  n  n  n  n  n  n  n  n  n  n  n  n  n  n  n  n  n
UNIX      1024     24      -1   NI       0   yes  kernel      n  n  n  n  n  n  n  n  n  n  n  n  n  n  n  n  n  n  n
UDP-Lite  1088      0       3   NI       0   yes  kernel      y  y  y  n  y  y  y  n  y  y  y  y  y  n  n  y  y  y  n
PING       912      0      -1   NI       0   yes  kernel      y  y  y  n  n  y  n  n  y  y  y  y  n  y  y  y  y  y  n
RAW        920      0      -1   NI       0   yes  kernel      y  y  y  n  y  y  y  n  y  y  y  y  n  y  y  y  y  n  n
UDP       1088      4       3   NI       0   yes  kernel      y  y  y  n  y  y  y  n  y  y  y  y  y  n  n  y  y  y  n
TCP       1992    209      52   no     256   yes  kernel      y  y  y  y  y  y  y  y  y  y  y  y  y  n  y  y  y  y  y
NETLINK   1048      0      -1   NI       0   no   kernel      n  n  n  n  n  n  n  n  n  n  n  n  n  n  n  n  n  n  n

We see that there is no memory pressure.

We would like some advice along the following:

  1. How can we find out if the buffer was actually overrun? Can we find out what the allocated buffer size was during a certain time period? Or what the maximum buffer size was? If the actual maximum buffer size was 6 MiB then we can assume the buffer was overrun.
  2. If the buffer was overrun, why the memory pressure flag is not set? Does the memory pressure flag means that the system was not able to allocate the minimum buffer size?
  3. Do buffer overruns and packet collapses always mean the socket buffer size was not enough?

Thank you.

mangohost

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.