Join Log in
Score:1
avatar Server

Linux server not allowing more than 2048 concurrent connections

in flag
Dhruv Sehgal

I tried to do load test on MQTT from my MACOS, and was able to successfully achieve more than 12k connections till my bandwidth got exhausted.

I tried to do the same test on GCP machine and it gave me connection timed out exception once the opened ports reached 2048, and 2048 connections with MQTT broker.

When connecting, my ConnectionTimeout is 100s (waiting for conack) and KeepAlive = 300s (once connection is established)

This issue is happening regardless of load test software i.e. mzbench, jmeter and emqtt-bench. So, I think this issue is related to the linux server.

I am not looking to achieve 1 Million open connections, but looking for atleast 30K open connections.

Have already tried to change ulimit and these are my ulimit configurations

core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 63887
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 102400
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 200000
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

cat on proc also gives max file open as 102400

also these are values set in my sysctl

fs.file-max = 200000
net.core.rmem_max = 16777216
net.core.wmem_max = 16777216
net.ipv4.tcp_rmem = 4096 87380 16777216
net.ipv4.tcp_wmem = 4096 65536 16777216
net.ipv4.tcp_syncookies = 1
net.ipv4.tcp_mem = 50576   64768   98152
net.core.netdev_max_backlog = 2500

Edit: Added machine details and test pattern

Machine type: n2-highcpu-16 (16 vCPUs, 16 GB memory)

Result of lscpu

Architecture:          x86_64
CPU op-mode(s):        32-bit, 64-bit
Byte Order:            Little Endian
CPU(s):                16
On-line CPU(s) list:   0-15
Thread(s) per core:    2
Core(s) per socket:    8
Socket(s):             1
NUMA node(s):          1
Vendor ID:             GenuineIntel
CPU family:            6
Model:                 85
Model name:            Intel(R) Xeon(R) CPU
Stepping:              7
CPU MHz:               2800.200
BogoMIPS:              5600.40
Hypervisor vendor:     KVM
Virtualization type:   full
L1d cache:             32K
L1i cache:             32K
L2 cache:              1024K
L3 cache:              33792K
NUMA node0 CPU(s):     0-15
Flags:                 fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc rep_good nopl xtopology nonstop_tsc cpuid tsc_known_freq pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch invpcid_single ssbd ibrs ibpb stibp ibrs_enhanced fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm mpx avx512f avx512dq rdseed adx smap clflushopt clwb avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves arat avx512_vnni md_clear arch_capabilities

Test pattern: Opening 200 connections per second and waiting for conack at a constant rate.

135
1 + 8
tcp
jp flag
Serhii Rohoza
Which machine type did you use? How fast did you establish all of your connections?
0
Reply
in flag
Dhruv Sehgal
@SerhiiRohoza have edited the question and added the details asked
0
Reply
Matthew Ife avatar
jo flag
Matthew Ife
Can you provide additional information as to the broker you are using on Linux?
0
Reply
in flag
Dhruv Sehgal
@MatthewIfe Yes, the broker I am using is also on linux. However, I don't think that it should be the root cause since I am able to open 12k connections from my local but only 2k connections from my test server.
0
Reply
Matthew Ife avatar
jo flag
Matthew Ife
I'm interested in the software implementation you're using as different mechanisms of concurrency have different limits. Also your ulimits in your shell aren't necessarily the ulimits of the software but since you've not offered the implementation of the broker you're using nor the means by which you deployed it theres not enough information to work with.
0
Reply
in flag
Dhruv Sehgal
Brokers used are emqx brokers deployed on similar gcloud machine. Ulimits provided above are the client's ulimit which is trying to create connection on the broker.
0
Reply
John Hanley avatar
cn flag
John Hanley
I might be wrong, but I think your GCP VM is subject to simultaneous open ports by the VPC firewall/gateway. Connections in Google Cloud are stateful and are tracked so that return traffic is accepted.
0
Reply
John Hanley avatar
cn flag
John Hanley
Machine types with 1 - 8 vCPUs are limited to 500 connections per vCPU per 5 seconds with a max of 4000 for larger machine types. https://cloud.google.com/vpc/docs/quota#per_instance This does not match your ceiling of 2000 ...
0
Reply
Score:1
avatar Server
in flag
Dhruv Sehgal

I was able to solve this problem, thanks to the comments above.

I was hitting public LoadBalancer IP from my test VM. However, GCP has limit of max 2048 connections from a VM to public IP. Once I changed this to private IP, I was able to achieve close to 65k connections.

0 + 0
ธงชาติไทย
Elon Musk
I sit in a Tesla and translated this thread with Ai:

mangohost

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.