Score:1

How to configure Nginx to proxy more than 200K MQTT connections?

ne flag

I need to proxy a lot of MQTT connections (over 200K) and distribute them over a number of ports at the backend server. Below is my Nginx configuration.

load_module /usr/lib/nginx/modules/ngx_stream_module.so;

user www-data;
pid /run/nginx.pid;

worker_rlimit_nofile 204800;
worker_cpu_affinity auto;
worker_processes auto;

events {
    accept_mutex on;             # Multiple workers should accept connection
    worker_connections 206200;   # Total connections with upstream and clients
    multi_accept off;            # Accept one connection in each worker
}

stream {
    upstream backend {
          server backendserver.com:1885;
          server backendserver.com:1886;
          server backendserver.com:1887;
          server backendserver.com:1888;
          server backendserver.com:1889;
          server backendserver.com:1890;
          server backendserver.com:1891;
          server backendserver.com:1892;
          server backendserver.com:1893;
          server backendserver.com:1894;
          server backendserver.com:1895;
          server backendserver.com:1896;
          server backendserver.com:1897;
          server backendserver.com:1898;
          server backendserver.com:1899;
          server backendserver.com:1900;
          server backendserver.com:1901;
          server backendserver.com:1902;
          server backendserver.com:1903;
          server backendserver.com:1904;
    }

    server {
        listen 8883;
        proxy_pass backend;
    }
}

I also have set the following system configurations:

sysctl -w fs.file-max=11000000
sysctl -w fs.nr_open=11000000
sysctl -w net.core.somaxconn=65535
sysctl -w net.ipv4.tcp_max_syn_backlog=65535
sysctl -w net.ipv4.ip_local_port_range="1025 65535"
sysctl -w net.ipv4.tcp_mem="100000000 100000000 100000000"
sysctl -w net.ipv4.tcp_rmem='2048 2048 2048'
sysctl -w net.ipv4.tcp_wmem='4096 4096 4096'
sysctl -w net.core.rmem_default=4096
sysctl -w net.core.wmem_default=4608
sysctl -w net.core.rmem_max=4096
sysctl -w net.core.wmem_max=4608

Also I have set ulimit -n 999999.

But I only can create around 128K-130K connections. After that, Nginx logs erros like below:

2023/08/16 06:46:32 [error] 4172305#4172305: *206569 recv() failed (104: Unknown error) while proxying and reading from upstream, client: 72.xxx.xxx.xxx, server: 0.0.0.0:8885, upstream: "139.xxx.xxx.xxx:8889", bytes from/to client:984/5224, bytes from/to upstream:5224/984

I checked directly connecting to all of the backend ports and they work fine.

Furthermore, when I only had one entry in upstream backend block, I could only create around 64K connections. When I add the second entry, I could go upto around 128K-130K. But then adding more entries does not let me go beyond that number of connections.

I have already tried doing below:

  1. Set worker_rlimit_nofile and worker_connections to 999999.
  2. Add more server blocks with different ports.
  3. 10x tcp_rmem, tcp_wmem, rmem_default, wmem_default, rmem_max, wmem_max.
  4. Setting multi_accept on;

How can I make my Nginx server to proxy more than 130K connections (at least 200K)?

I sit in a Tesla and translated this thread with Ai:

mangohost

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.