Join Log in
Score:2
avatar Server

Drop ALL the TCP connections (ESTABLISHED,RELATED) in Ubuntu

us flag
Dr. Gianluigi Zane Zanettini

I'm facing a few hosts sending a flood of requests to my webserver (NGINX). I'm trying to block them via iptables, with ipset and a good old DROP rule.

The rule is effective against NEW connections, but as soon as the kiddies can come in and set up an ESTABLISHED or RELATED connection, my DROP rule fails because my firewall also has a iptables -A INPUT -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT rule.

The relevant section of my firewall config is:

# iptables -nL
Chain INPUT (policy ACCEPT)
target     prot opt source               destination         
ACCEPT     all  --  0.0.0.0/0            0.0.0.0/0            ctstate RELATED,ESTABLISHED
DROP       all  --  0.0.0.0/0            0.0.0.0/0            match-set Blacklist src
ACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0            multiport dports 80,443
DROP       all  --  0.0.0.0/0            0.0.0.0/0            

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination         

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination

The Blacklist ipset includes the offending IP:

# ipset list Blacklist | grep <attacking-server-ip>
<attacking-server-ip>

The question is: how can I KILL ALL the TCP connections, even the ESTABLISHED or RELATED ones?

Things I've tried:

Stop-wait-30-secs-start the webserver (service nginx stop && sleep 30 && service nginx start) - it was worth a shot, but since the connection is already ESTABLISHED, it persists.

conntrack --flush and conntrack -F. No effect

tcpkill: as long as I keep the process running it kinda works. But as soon as I close the process, the connection come back. I can't really explain it

conntrack -D --orig-src <attacking-server-ip>: the connection got deleted, but then appear back immeditaly.

To check the connection I use:

 netstat -putan | grep '<attacking-server-ip>'
tcp        0      0 <my-server-ip>:443       <attacking-server-ip>:60763     ESTABLISHED 2372191/nginx: work 
tcp        0      0 <my-server-ip>:443       <attacking-server-ip>:60807     ESTABLISHED 2372191/nginx: work 
tcp        0      0 <my-server-ip>:443       <attacking-server-ip>:60545     ESTABLISHED 2372190/nginx: work 
tcp        0      0 <my-server-ip>:443       <attacking-server-ip>:59785     TIME_WAIT   -                   
tcp        0      0 <my-server-ip>:443       <attacking-server-ip>:59730     TIME_WAIT   -                   
tcp        0      0 <my-server-ip>:443       <attacking-server-ip>:59841     TIME_WAIT   -                   
tcp        0      0 <my-server-ip>:443       <attacking-server-ip>:60578     ESTABLISHED 2372191/nginx: work 
tcp        0      0 <my-server-ip>:443       <attacking-server-ip>:60941     ESTABLISHED 2372191/nginx: work 
tcp        0      0 <my-server-ip>:443       <attacking-server-ip>:59390     TIME_WAIT   -                   
tcp        0      0 <my-server-ip>:443       <attacking-server-ip>:59849     TIME_WAIT   -                   
tcp        0      0 <my-server-ip>:443       <attacking-server-ip>:60744     ESTABLISHED 2372191/nginx: work 
tcp        0      0 <my-server-ip>:443       <attacking-server-ip>:59766     TIME_WAIT   -                   
tcp        0      0 <my-server-ip>:443       <attacking-server-ip>:59819     TIME_WAIT   -                   
tcp        0      0 <my-server-ip>:443       <attacking-server-ip>:60679     ESTABLISHED 2372191/nginx: work 
tcp        0      0 <my-server-ip>:443       <attacking-server-ip>:59603     TIME_WAIT   -                   
tcp        0      0 <my-server-ip>:443       <attacking-server-ip>:60134     ESTABLISHED 2372191/nginx: work 
tcp        0      0 <my-server-ip>:443       <attacking-server-ip>:60907     ESTABLISHED 2372191/nginx: work 
tcp        0      0 <my-server-ip>:443       <attacking-server-ip>:59732     TIME_WAIT   -                   
tcp        0      0 <my-server-ip>:443       <attacking-server-ip>:60128     ESTABLISHED 2372191/nginx: work 
tcp        0      0 <my-server-ip>:443       <attacking-server-ip>:60437     ESTABLISHED 2372191/nginx: work 
tcp        0      0 <my-server-ip>:443       <attacking-server-ip>:59623     TIME_WAIT   -                   
tcp        0      0 <my-server-ip>:443       <attacking-server-ip>:59356     TIME_WAIT   -                   
tcp        0      0 <my-server-ip>:443       <attacking-server-ip>:60502     ESTABLISHED 2372191/nginx: work 
tcp        0      0 <my-server-ip>:443       <attacking-server-ip>:59414     TIME_WAIT   -                   
tcp        0      0 <my-server-ip>:443       <attacking-server-ip>:60592     ESTABLISHED 2372191/nginx: work 
tcp        0      0 <my-server-ip>:443       <attacking-server-ip>:59370     TIME_WAIT   -                   
tcp        0      0 <my-server-ip>:443       <attacking-server-ip>:60861     ESTABLISHED 2372191/nginx: work 
tcp        0      0 <my-server-ip>:443       <attacking-server-ip>:59783     TIME_WAIT   -                   
tcp        0      0 <my-server-ip>:443       <attacking-server-ip>:59858     TIME_WAIT   -                   
tcp        0      0 <my-server-ip>:443       <attacking-server-ip>:59769     TIME_WAIT   -                   
tcp        0      0 <my-server-ip>:443       <attacking-server-ip>:60817     ESTABLISHED 2372191/nginx: work 
tcp        0      0 <my-server-ip>:443       <attacking-server-ip>:59393     TIME_WAIT   -                   
tcp        0      0 <my-server-ip>:443       <attacking-server-ip>:60479     ESTABLISHED 2372191/nginx: work 
tcp        0      0 <my-server-ip>:443       <attacking-server-ip>:60450     ESTABLISHED 2372191/nginx: work 
tcp        0      0 <my-server-ip>:443       <attacking-server-ip>:59401     TIME_WAIT   -                   
tcp        0      0 <my-server-ip>:443       <attacking-server-ip>:60838     ESTABLISHED 2372191/nginx: work 
tcp        0      0 <my-server-ip>:443       <attacking-server-ip>:60123     ESTABLISHED 2372191/nginx: work 
tcp        0      0 <my-server-ip>:443       <attacking-server-ip>:60854     ESTABLISHED 2372191/nginx: work 
tcp        0      0 <my-server-ip>:443       <attacking-server-ip>:59445     TIME_WAIT   -                   
tcp        0      0 <my-server-ip>:443       <attacking-server-ip>:59419     TIME_WAIT   -                   
tcp        0      0 <my-server-ip>:443       <attacking-server-ip>:60111     ESTABLISHED 2372191/nginx: work 
tcp        0      0 <my-server-ip>:443       <attacking-server-ip>:60934     ESTABLISHED 2372191/nginx: work 
tcp        0      0 <my-server-ip>:443       <attacking-server-ip>:60510     ESTABLISHED 2372191/nginx: work 
tcp        0      0 <my-server-ip>:443       <attacking-server-ip>:60832     ESTABLISHED 2372191/nginx: work 
tcp        0      0 <my-server-ip>:443       <attacking-server-ip>:60922     ESTABLISHED 2372191/nginx: work 
tcp        0      0 <my-server-ip>:443       <attacking-server-ip>:60447     ESTABLISHED 2372191/nginx: work 
tcp        0      0 <my-server-ip>:443       <attacking-server-ip>:60171     ESTABLISHED 2372191/nginx: work 
tcp        0      0 <my-server-ip>:443       <attacking-server-ip>:60536     ESTABLISHED 2372191/nginx: work 
tcp        0      0 <my-server-ip>:443       <attacking-server-ip>:59344     TIME_WAIT   -                   
tcp        0      0 <my-server-ip>:443       <attacking-server-ip>:59394     TIME_WAIT   -                   
tcp        0      0 <my-server-ip>:443       <attacking-server-ip>:59369     TIME_WAIT   -                   
tcp        0      0 <my-server-ip>:443       <attacking-server-ip>:60601     ESTABLISHED 2372191/nginx: work 
tcp        0      0 <my-server-ip>:443       <attacking-server-ip>:59754     TIME_WAIT   -                   
tcp        0   5534 <my-server-ip>:443       <attacking-server-ip>:60954     ESTABLISHED 2372191/nginx: work 
tcp        0      0 <my-server-ip>:443       <attacking-server-ip>:60895     ESTABLISHED 2372191/nginx: work 
tcp        0      0 <my-server-ip>:443       <attacking-server-ip>:60236     ESTABLISHED 2372191/nginx: work 
tcp        0      0 <my-server-ip>:443       <attacking-server-ip>:60099     ESTABLISHED 2372191/nginx: work 
tcp        0      0 <my-server-ip>:443       <attacking-server-ip>:59460     TIME_WAIT   -                   
tcp        0      0 <my-server-ip>:443       <attacking-server-ip>:60570     ESTABLISHED 2372191/nginx: work 
tcp        0      0 <my-server-ip>:443       <attacking-server-ip>:60555     ESTABLISHED 2372191/nginx: work

Also:

# conntrack -L --orig-src <attacking-server-ip>
tcp      6 431977 ESTABLISHED src=<attacking-server-ip> dst=<my-server-ip> sport=62907 dport=443 src=<my-server-ip> dst=<attacking-server-ip> sport=443 dport=62907 [ASSURED] mark=0 use=1
tcp      6 431989 ESTABLISHED src=<attacking-server-ip> dst=<my-server-ip> sport=63062 dport=443 src=<my-server-ip> dst=<attacking-server-ip> sport=443 dport=63062 [ASSURED] mark=0 use=1
tcp      6 431976 ESTABLISHED src=<attacking-server-ip> dst=<my-server-ip> sport=62882 dport=443 src=<my-server-ip> dst=<attacking-server-ip> sport=443 dport=62882 [ASSURED] mark=0 use=1
tcp      6 299 ESTABLISHED src=<attacking-server-ip> dst=<my-server-ip> sport=63215 dport=443 src=<my-server-ip> dst=<attacking-server-ip> sport=443 dport=63215 [ASSURED] mark=0 use=1
tcp      6 431975 ESTABLISHED src=<attacking-server-ip> dst=<my-server-ip> sport=62869 dport=443 src=<my-server-ip> dst=<attacking-server-ip> sport=443 dport=62869 [ASSURED] mark=0 use=1
tcp      6 271 ESTABLISHED src=<attacking-server-ip> dst=<my-server-ip> sport=62800 dport=443 [UNREPLIED] src=<my-server-ip> dst=<attacking-server-ip> sport=443 dport=62800 mark=0 use=1
tcp      6 299 ESTABLISHED src=<attacking-server-ip> dst=<my-server-ip> sport=63211 dport=443 src=<my-server-ip> dst=<attacking-server-ip> sport=443 dport=63211 [ASSURED] mark=0 use=1
tcp      6 431977 ESTABLISHED src=<attacking-server-ip> dst=<my-server-ip> sport=62902 dport=443 src=<my-server-ip> dst=<attacking-server-ip> sport=443 dport=62902 [ASSURED] mark=0 use=1
tcp      6 431987 ESTABLISHED src=<attacking-server-ip> dst=<my-server-ip> sport=63038 dport=443 src=<my-server-ip> dst=<attacking-server-ip> sport=443 dport=63038 [ASSURED] mark=0 use=1
tcp      6 431999 ESTABLISHED src=<attacking-server-ip> dst=<my-server-ip> sport=63195 dport=443 src=<my-server-ip> dst=<attacking-server-ip> sport=443 dport=63195 [ASSURED] mark=0 use=1
tcp      6 431976 ESTABLISHED src=<attacking-server-ip> dst=<my-server-ip> sport=62887 dport=443 src=<my-server-ip> dst=<attacking-server-ip> sport=443 dport=62887 [ASSURED] mark=0 use=1
tcp      6 431988 ESTABLISHED src=<attacking-server-ip> dst=<my-server-ip> sport=63050 dport=443 src=<my-server-ip> dst=<attacking-server-ip> sport=443 dport=63050 [ASSURED] mark=0 use=1
tcp      6 431999 ESTABLISHED src=<attacking-server-ip> dst=<my-server-ip> sport=63201 dport=443 src=<my-server-ip> dst=<attacking-server-ip> sport=443 dport=63201 [ASSURED] mark=0 use=1
tcp      6 431998 ESTABLISHED src=<attacking-server-ip> dst=<my-server-ip> sport=63181 dport=443 src=<my-server-ip> dst=<attacking-server-ip> sport=443 dport=63181 [ASSURED] mark=0 use=1
conntrack v1.4.5 (conntrack-tools): 14 flow entries have been shown
1299
1 + 5
tcp
Steffen Ullrich avatar
se flag
Steffen Ullrich
It is unknown what exactly your "webserver" is, but in case of Apache see [Restart Apache and Kill All Sub Processes](https://serverfault.com/questions/169060/restart-apache-and-kill-all-sub-processes) to make sure that established connections get closed too.
0
Reply
us flag
Dr. Gianluigi Zane Zanettini
Thanks @SteffenUllrich ! My webserver is NGINX. Actually, I already STOPPED the service (I updated for clarity), but made no difference.
0
Reply
Steffen Ullrich avatar
se flag
Steffen Ullrich
How exactly did you stop the web server? How did you check that the connections are still open? The question so far is only giving your interpretation of what you observed, but does not show what exactly you tried and what exactly and how you observed and came to your conclusions. And if the connections come back after tcpkill (assuming your interpretation of what you see is correct) it means that they successfully got closed and got reestablished, i.e. that your blocking of new connections does not work contrary to what you say.
0
Reply
us flag
Dr. Gianluigi Zane Zanettini
@SteffenUllrich Thanks again for your help! I added even more details.
0
Reply
Steffen Ullrich avatar
se flag
Steffen Ullrich
The connections you see in netstat as ESTABLISHED clearly have a running nginx associated with it - which means that this output was not generated while nginx was fully stopped.
0
Reply
Score:3
A.B avatar Server
cl flag
A.B

You have to test for a blacklist before the stateful accept rule, or your test is never reached because it's short-circuited.

Instead of:

iptables -A INPUT -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
iptables -A INPUT -m set --match-set Blacklist src -j DROP

Invert their order:

iptables -A INPUT -m set --match-set Blacklist src -j DROP
iptables -A INPUT -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT

You can do this now by inserting the rule at the first position:

iptables -I INPUT -m set --match-set Blacklist src -j DROP

You can also do this instead of above:

  • remove a conntrack entry using the conntrack command with its -D/--delete command

    Once the entry is removed, the connection will come back through the NEW state (because Netfilter has TCP set to recover mode by default) and should be blocked by the Blacklist still in 2nd position, unless the server replies first which will reestablish the connection. So one of these has to be done:

  • kill the socket (without killing the process, but getting the process to get an error on this socket, apparently ECONNABORTED)

    Recent enough Linux kernels accept ss --kill to forcefully kill a socket, using its built-in filter logic. Example (the source and destinations are always seen from the local system, so inverted. Also mind the spaces between parenthesis):

    ss --kill -tn 'dst == 192.0.2.2 and ( sport == 80 or sport == 443 )'

    (or just the destination, it's about an attacker anyway).

    Unlike tcpkill this doesn't involve sending custom packets anywhere, it's directly done to the socket using a kernel API.

Then for example if the Blacklist is a list of single IP addresses, one could do this to remove all attacker IP addresses from the conntrack state and socket state.

ipset save Blacklist | awk '/^add/ { print $3 }' | 
    while read ip; do
        conntrack -D -s $ip
        ss -K -n dst $ip
        conntrack -D -s $ip # again: in case of race between two previous
    done

If the script adding an entry to the Blacklist (typically something like fail2ban and the actionban = ipset ... in /etc/fail2ban/action.d/iptables-ipset-proto4.conf) is changed to also run each time the conntrack and ss commands, then you don't have to invert the INPUT rules and will spare some CPU and memory (a bit in firewall and more in nginx).

Without killing the socket, the server will sooner or later emit a packet in conntrack state NEW (despite not having a TCP SYN flag) which will re-establish the conntrack entry in ESTABLISHED state once the attacker replies (still in an already established TCP connection). This is conditioned by this toggle which is loose by default:

nf_conntrack_tcp_loose - BOOLEAN

  • 0 - disabled
  • not 0 - enabled (default)

If it is set to zero, we disable picking up already established connections.

That's why connections were "restored" in OP's case. To avoid this to happen (and get an INVALID state instead):

sysctl -w net.netfilter.nf_conntrack_tcp_loose=0

which would prevent connections to recover after a conntrack -F (including a remote SSH!, so better use -D as above). Having this feature enabled is probably most useful only for a firewalling router that reboots.

+ 0
ธงชาติไทย
Elon Musk
I sit in a Tesla and translated this thread with Ai:

mangohost

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.