Join Log in
Score:1
avatar Server

Round robin usage of multiple IPs for outgoing connections on single interface

us flag
Racer

my issue: I have a process creating more than 10k TCP-Connection to the same destination IP&Port per second. After a short period of time, no new connections can be created any more, as there are no more source ports available.

I already tried to mitigate this by setting net.ipv4.ip_local_port_range and net.ipv4.tcp_fin_timeout but it did not solve the issue.

Now my thought was: If I could tell the kernel to round robin source IP-Addresses, then I could easily add more IPs to my one interface and therefore have a higher limit of outgoing connections.

I tried adding multiple routes to the same destination via the same interface but specify different source IPs:

ip route add default via 10.1.1.1 dev eth0 src 10.1.1.10
ip route add default via 10.1.1.1 dev eth0 src 10.1.1.11

But i get RTNETLINK answers: File exists.

Am I on the right track here? Is adding another source IP "they way to go" in those cases?

Is there a way to round robin / load balance the usage of source IPs through the kernel somehow?

My hope was, that in case I manage to place two routes with the same weight but different src-addresses towards the same network, the kernel would round robin that.

(Background: I am running HAproxy on that machine and have to load balance more than 10k connections towards a single backend server)

85
0 + 0
Score:3
avatar Server
iq flag
MikeyB

Not easily via the kernel, no.

I am running HAproxy on that machine and have to load balance more than 10k connections towards a single backend server

You can tell haproxy to use a specific source IP address for outgoing connections to a server, for example:

  server app1_s1 10.0.1.1:80 source 10.1.1.10
  server app1_s2 10.0.1.1:80 source 10.1.1.11
  server app2_s1 10.0.1.2:80 source 10.1.1.10
  server app2_s2 10.0.1.2:80 source 10.1.1.11
0 + 0
us flag
Racer
That would at lease prevent my issue, so thanks for that so far :) My hope was to let the kernel handle that, but if that is not possible, then I will go with that. When you said "Not easily via the kernel, no.", does it mean, there is no easy way, or does it mean, there is actually no way at all? I don't really like the fact, that then I have to define every backend server twice, but I will address this towards HAProxy and place a feature request to add more than one source IP to one server line / to one backend config block.
0
Reply
iq flag
MikeyB
that is something I miss and is available on other products - the ability to specify multiple addresses or a range of addresses to use here instead of a single one.
0
Reply
Score:2
drookie avatar Server
za flag
drookie

You still don't. net.ipv4.ip_local_port_range should be enough even with default settings. So I bet either you have way more than 10K connections, or you diagnose it wrong.

See, the "ipsrc:port ipdst:port" quadruplet is unique throughout the machine IP stack, and since the total number of ports in a network stack is 65536, this gives you the possibility to have 65536 connections in a theory. Okay, we minus the WKS block (-1024), but this still gived you 55536 connections with a tampered net.ipv4.ip_local_port_range. You need to go with the additional IPs/interfaces only if you need more than that.

And usually this issue is solved with adding more client IPs and/or server IPs if the destination is also inside your stack, or with using multiple routing tables if it's not.

0 + 0
us flag
Racer
I have 10k new connections per second, and they are quite short living. But: Once the connections are closed (as far as I understand) they go into the TIME_WAIT state. So they are not useable for 60 Seconds. And that's why I also seem to exhaust the tweaked `net.ipv4.ip_local_port_range` :/ The server is not really in my stack, so doing adjustments there is not an option for me. That's why I wanted to go with the option of using multiple IPv4 addresses on the same interface and letting the kernel load balance between those
0
Reply
drookie avatar
za flag
drookie
The total amount of connections matters, not only ESTABLISHED ones. Connection is a connection regardless of it's state.
0
Reply
us flag
Racer
Yes, I am aware of that. And as the amount of connection exceed the available ports, even after adjusting `net.ipv4.ip_local_port_range` I am looking for a way to ideally get the kernel balance between multiple source IPs on one interface. (As I can't edit anything on the server side) Only if the kernel is not able to do that, I would go for the other option of having HAProxy itself handle multiple source IPs.
0
Reply
ธงชาติไทย
Elon Musk
I sit in a Tesla and translated this thread with Ai:

mangohost

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.