Join Log in
Score:3
JonS avatar Server

Tune Linux & Nignx to handle 10k Connections @10Gbps Server

br flag
JonS

I just got a new 10Gbps server with 8 CPU Cores, 64GB RAM and 1TB NVMe

OS Centos 7.9 kernel 3.10.0-1160.36.2.el7.x86_64 also tried kernel-ml 5.13
SELinux is disabled.
firewalld and irqbalance stopped

I've done network test using iperf3, speed is confirmed around 9.5 Gbps.

Then another test using 10 x 1Gbps servers to download a static file from the server, the server was able to push almost the full 10Gbps to the 10 servers easily.

So we put the server to production serving clients downloading static files using Nginx. It is able provide stable performance until reaching ~2,000 connections then performance starts to drop significantly. I see traffic declines when connections increases, so serving more than 4,000 connections give only 2Gbps!

Image 1 Shows Traffic and HTTP

The most confusing is that CPU is almost idle, RAM is free, IO usage is low thanks to NVMe and large RAM, but when server has thousands of connections, speed becomes slow on all services HTTP, FTP, SSH even yum updating takes so long time to respond. It seems like a congestion in network or packets or some throttling in kernel or nic.

htop top nload

I tried most tuning tips

ifconfig eth0 txqueuelen 20000
ifconfig eth0
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        ether 00:16:3e:c2:f5:21  txqueuelen 20000  (Ethernet)
        RX packets 26012067560  bytes 1665662731749 (1.5 TiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 30684216747  bytes 79033055227212 (71.8 TiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

tc -s -d qdisc show dev eth0

qdisc mq 1: root
    Sent 7733649086021 bytes 1012203012 pkt (dropped 0, overlimits 0 requeues 169567)
    backlog 4107556b 2803p requeues 169567
qdisc pfifo_fast 0: parent 1:8 bands 3 priomap  1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1
    Sent 2503685906926 bytes 1714686297 pkt (dropped 0, overlimits 0 requeues 1447)
    backlog 4107556b 2803p requeues 1447
qdisc pfifo_fast 0: parent 1:7 bands 3 priomap  1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1
    Sent 532876060762 bytes 366663805 pkt (dropped 0, overlimits 0 requeues 7790)
    backlog 0b 0p requeues 7790
qdisc pfifo_fast 0: parent 1:6 bands 3 priomap  1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1
    Sent 563510390106 bytes 387948990 pkt (dropped 0, overlimits 0 requeues 9694)
    backlog 0b 0p requeues 9694
qdisc pfifo_fast 0: parent 1:5 bands 3 priomap  1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1
    Sent 563033712946 bytes 387564038 pkt (dropped 0, overlimits 0 requeues 10259)
    backlog 0b 0p requeues 10259
qdisc pfifo_fast 0: parent 1:4 bands 3 priomap  1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1
    Sent 562982455659 bytes 387451904 pkt (dropped 0, overlimits 0 requeues 10706)
    backlog 0b 0p requeues 10706
qdisc pfifo_fast 0: parent 1:3 bands 3 priomap  1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1
    Sent 559557988260 bytes 385263948 pkt (dropped 0, overlimits 0 requeues 9983)
    backlog 0b 0p requeues 9983
qdisc pfifo_fast 0: parent 1:2 bands 3 priomap  1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1
    Sent 528903326344 bytes 364105031 pkt (dropped 0, overlimits 0 requeues 7718)
    backlog 0b 0p requeues 7718
qdisc pfifo_fast 0: parent 1:1 bands 3 priomap  1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1
    Sent 1919099245018 bytes 1313486295 pkt (dropped 0, overlimits 0 requeues 111970)
    backlog 0b 0p requeues 111970

ethtool -k eth0

Features for eth0:
rx-checksumming: on [fixed]
tx-checksumming: on
        tx-checksum-ipv4: off [fixed]
        tx-checksum-ip-generic: on
        tx-checksum-ipv6: off [fixed]
        tx-checksum-fcoe-crc: off [fixed]
        tx-checksum-sctp: off [fixed]
scatter-gather: on
        tx-scatter-gather: on
        tx-scatter-gather-fraglist: off [fixed]
tcp-segmentation-offload: off
        tx-tcp-segmentation: off
        tx-tcp-ecn-segmentation: off
        tx-tcp6-segmentation: off
        tx-tcp-mangleid-segmentation: off
udp-fragmentation-offload: on
generic-segmentation-offload: off
generic-receive-offload: off
large-receive-offload: off [fixed]
rx-vlan-offload: off [fixed]
tx-vlan-offload: off [fixed]
ntuple-filters: off [fixed]
receive-hashing: off [fixed]
highdma: on [fixed]
rx-vlan-filter: on [fixed]
vlan-challenged: off [fixed]
tx-lockless: off [fixed]
netns-local: off [fixed]
tx-gso-robust: off [fixed]
tx-fcoe-segmentation: off [fixed]
tx-gre-segmentation: off [fixed]
tx-ipip-segmentation: off [fixed]
tx-sit-segmentation: off [fixed]
tx-udp_tnl-segmentation: off [fixed]
fcoe-mtu: off [fixed]
tx-nocache-copy: off
loopback: off [fixed]
rx-fcs: off [fixed]
rx-all: off [fixed]
tx-vlan-stag-hw-insert: off [fixed]
rx-vlan-stag-hw-parse: off [fixed]
rx-vlan-stag-filter: off [fixed]
busy-poll: off [fixed]
tx-gre-csum-segmentation: off [fixed]
tx-udp_tnl-csum-segmentation: off [fixed]
tx-gso-partial: off [fixed]
tx-sctp-segmentation: off [fixed]
rx-gro-hw: off [fixed]
l2-fwd-offload: off [fixed]
hw-tc-offload: off [fixed]
rx-udp_tunnel-port-offload: off [fixed]

sysctl -p

vm.max_map_count = 1048575
net.ipv4.tcp_timestamps = 0
net.ipv4.conf.all.rp_filter = 1
net.ipv4.conf.default.rp_filter = 1
net.ipv4.tcp_ecn = 0
net.ipv4.tcp_sack = 1
net.ipv4.tcp_syncookies = 0
net.ipv4.conf.all.log_martians = 1
vm.swappiness = 10
net.core.somaxconn = 65535
net.ipv4.tcp_max_syn_backlog = 65536
net.core.netdev_max_backlog = 250000
fs.file-max = 100000
net.ipv4.ip_local_port_range = 13000 65000
net.ipv4.udp_rmem_min = 8192
net.ipv4.udp_wmem_min = 8192
net.ipv4.conf.all.send_redirects = 0
net.ipv4.conf.all.accept_redirects = 0
net.ipv4.conf.all.accept_source_route = 0
net.ipv4.ip_forward = 0
net.ipv6.conf.all.forwarding = 0
net.ipv4.tcp_slow_start_after_idle = 0
net.core.rmem_max = 2147483647
net.core.rmem_default = 2147483647
net.core.wmem_max = 2147483647
net.core.wmem_default = 2147483647
net.core.optmem_max = 2147483647
net.ipv4.tcp_rmem = 4096 87380 2147483647
net.ipv4.tcp_wmem = 4096 65536 2147483647
net.ipv4.tcp_low_latency = 1
net.ipv4.tcp_adv_win_scale = 1
net.ipv4.tcp_keepalive_time = 60
net.netfilter.nf_conntrack_tcp_timeout_time_wait = 5
net.ipv4.tcp_max_tw_buckets = 2000000
net.ipv4.tcp_fin_timeout = 10
net.ipv4.tcp_tw_reuse = 1
net.ipv4.tcp_keepalive_intvl = 15
net.ipv4.tcp_keepalive_probes = 5
net.netfilter.nf_conntrack_max = 655360
net.netfilter.nf_conntrack_tcp_timeout_established = 10800
net.ipv6.conf.all.disable_ipv6 = 1
net.ipv6.conf.default.disable_ipv6 = 1
net.ipv6.conf.lo.disable_ipv6 = 1

ulimit -a

core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 256680
max locked memory       (kbytes, -l) unlimited
max memory size         (kbytes, -m) unlimited
open files                      (-n) 100000
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 100000
cpu time               (seconds, -t) unlimited
max user processes              (-u) 100000
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

nginx.conf

worker_processes        auto;
worker_rlimit_nofile    100000;

thread_pool default threads=256     max_queue=65536;

events {
    worker_connections  65536;
    worker_aio_requests 65536;
    multi_accept on;
    accept_mutex on;
    use epoll;
}

http {
    server_tokens off;
    server_names_hash_max_size      4096;
    server_names_hash_bucket_size   128;

    tcp_nopush     on;
    tcp_nodelay     on;
    client_body_timeout 12;
    client_header_timeout 12;
    keepalive_timeout 15;
    keepalive_requests 1000;
    send_timeout 10;

    aio                         threads=default;
    sendfile                    on;
    sendfile_max_chunk          512k;
    open_file_cache             max=100000  inactive=10m;
    open_file_cache_valid       10m;
    open_file_cache_min_uses    10;
    open_file_cache_errors      on;

    gzip  off;
}

So the question is: How to serve 10k Connections with 10Gbps traffic downloading static files? Is it an issue of linux or nginx or hardware?

510
2 + 3
Brandon Xavier avatar
us flag
Brandon Xavier
Is there a specific guide you followed for your tuning? In particular I'm curious about: A) the choice not to enable jumbo frames B) IRQ affinity/pinning - I see you have irqbalance disabled but no mention of pinning the NIC's IRQs to a particular CPU C) the various parameters that were changed from the defaults (sorry, but determining this from just your running config isn't really a good use of anybody's time ;-)
1
Reply
JonS avatar
br flag
JonS
I've read many articles online https://www.kernel.org/doc/ols/2009/ols2009-pages-169-184.pdf https://fasterdata.es.net/host-tuning/linux/ https://darksideclouds.wordpress.com/2016/10/10/tuning-10gb-nics-highway-to-hell/ A) Because as I know it must be enabled on all switches, network devices and connecting clients, so it is useful only for transferring data between high speed servers. B) Didn't set affinity because this is virtual machine, can I set it? C) I guess the default parameters are known to anyone who has spent good time in linux and nginx and can be found online.
0
Reply
JonS avatar
br flag
JonS
indeed all NIC interrupts are going only to single CPU https://pastebin.com/9JTbgt5J so it's a bottleneck, how to set irq affinity on VM virtio?
0
Reply
Score:3
avatar Server
br flag
Joe

Already answered by Brandon. Turn on irqbalance. Run numad and tuned. Stop trying to tune unless you have a specific workload that requires it. Where are your wrk test results from testing 2000-10000 requests before you deployed? This problem should never have been seen in production. It clearly would have been identified by testing. Real world use will often uncover uncommon bugs, but many/most configuration and application bugs can be identified and corrected during testing. There are many docs available regarding irq affinity. I doubt your use case can do better than using the tuning tools built in. More than likely, your hand tuning will perform worse.

0 + 2
JonS avatar
br flag
JonS
I've enabled irqbalance, now the interrupts are balanced on the 8 CPU cores, but the problem still exists, it seems a problem handling packets properly. i turned on large-receive-offload and tcp-segmentation-offload but not big difference.
0
Reply
Brandon Xavier avatar
us flag
Brandon Xavier
Hopefully you're working on the host before adjusting the guest.
0
Reply
Score:1
Jose Fernando Lopez Fernandez avatar Server
sz flag
Jose Fernando Lopez Fernandez

The output from top says your kernel is being inundated with soft interrupts from all of the incoming connections. The connections are coming in so fast that the hardware interrupts triggered by the network card are queueing soft interrupts faster than the kernel can deal with them. This is why your CPU, RAM, and IO usage is so low; the system keeps getting interrupted by incoming connections. What you need here is a load-balancer.

0 + 2
JonS avatar
br flag
JonS
We already use load-balancer to send traffic to 2 servers with same config, so I want every server to handle many connections with high bandwidth. Does adding more CPU Cores solves the issue of Soft interrupts?
0
Reply
JonS avatar
br flag
JonS
indeed all NIC interrupts are going only to single CPU https://pastebin.com/9JTbgt5J so it's a bottleneck, how to set irq affinity on VM virtio?
0
Reply
ธงชาติไทย
Elon Musk
I sit in a Tesla and translated this thread with Ai:

mangohost

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.