Score:1

ksoftirqd maxing out for network interfaces

ru flag

We have a Debian server with one link to the internal VLAN, and one to the external - both connect directly to the same switch.

On both links, we're intermittently seeing an unusually high amount of bad receive events, as well as high latency.

Kernel  Interface  table
Iface   MTU        Met    RX-OK       RX-ERR  RX-DRP     RX-OVR     TX-OK       TX-ERR  TX-DRP  TX-OVR  Flg
ethA    1500       0      884347583   0       49965509   49965509   1697514631  0       0       0       BMRU
ethB    1500       0      1611102819  0       77615811   77615811   819321274   0       0       0       BMRU

We're also seeing the ksoftirqd processes for both interfaces max out at 90+% most of the time, even when things should ostensibly be quiet.

44 root      20   0       0      0      0 R  98.8  0.0   2557:46 ksoftirqd/3
51 root      20   0       0      0      0 R  85.6  0.0   2722:33 ksoftirqd/5

As I understand it, this means the server is maxing out the assigned CPU to process all the packets coming in on this interface. But even when we're seeing ~50Mbps coming in, (and identical servers are handling >800Mbps) these processes can max out and RX-DRPs skyrocket. irqbalance is running, and /proc/interrupts confirms that these CPUs aren't busy handling really anything else.

Are there any clear potential causes for this?

Would assigning multiple CPUs (via smp_affinity) to handle the interrupts for those interfaces potentially help? I'm unable to find a single example of someone assigning multiple CPUs to a single network interface, so it seems unconventional at best, system-break-worthy at worst.

This has been causing issues in production for a while, so I'd happily accept any potential workaround.

mangohost

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.