Join Log in
Score:2
avatar Ubuntu

UFW is rate-limiting connections without any limit rules

ca flag
kleinric

I have a server running hosting restful services via HTTPS. I have UFW set up as below:

user@server:~$ sudo ufw status verbose

Status: active
Logging: on (medium)
Default: deny (incoming), allow (outgoing), disabled (routed)
New profiles: skip

To                         Action      From
--                         ------      ----
22/tcp                     ALLOW IN    Anywhere                  
80/tcp                     ALLOW IN    Anywhere                  
443/tcp                    ALLOW IN    Anywhere                  
22/tcp (v6)                ALLOW IN    Anywhere (v6)             
80/tcp (v6)                ALLOW IN    Anywhere (v6)             
443/tcp (v6)               ALLOW IN    Anywhere (v6)

So it should be letting everything in for HTTPS. However, every now and then when a client makes a number of quick successive connections UFW blocks them.

user@server:~$ cat /var/log/ufw.log | grep BLOCK | tail

Nov 29 15:02:11 server kernel: [ 2695.510376] [UFW BLOCK] IN=ens4 OUT= MAC=xx:xx:xx:... SRC=CLIENT.IP DST=SERVER.IP LEN=40 TOS=0x00 PREC=0x00 TTL=53 ID=60467 DF PROTO=TCP SPT=32123 DPT=443 WINDOW=0 RES=0x00 RST URGP=0 
Nov 29 15:02:11 server kernel: [ 2695.538273] [UFW BLOCK] IN=ens4 OUT= MAC=xx:xx:xx:... SRC=CLIENT.IP DST=SERVER.IP LEN=40 TOS=0x00 PREC=0x00 TTL=53 ID=60469 DF PROTO=TCP SPT=32126 DPT=443 WINDOW=0 RES=0x00 RST URGP=0 
Nov 29 15:02:11 server kernel: [ 2695.545677] [UFW BLOCK] IN=ens4 OUT= MAC=xx:xx:xx:... SRC=CLIENT.IP DST=SERVER.IP LEN=40 TOS=0x00 PREC=0x00 TTL=53 ID=60471 DF PROTO=TCP SPT=32124 DPT=443 WINDOW=0 RES=0x00 RST URGP=0 
Nov 29 15:02:11 server kernel: [ 2695.546473] [UFW BLOCK] IN=ens4 OUT= MAC=xx:xx:xx:... SRC=CLIENT.IP DST=SERVER.IP LEN=40 TOS=0x00 PREC=0x00 TTL=53 ID=60473 DF PROTO=TCP SPT=32125 DPT=443 WINDOW=0 RES=0x00 RST URGP=0 
Nov 29 15:02:11 server kernel: [ 2695.546900] [UFW BLOCK] IN=ens4 OUT= MAC=xx:xx:xx:... SRC=CLIENT.IP DST=SERVER.IP LEN=40 TOS=0x00 PREC=0x00 TTL=53 ID=60475 DF PROTO=TCP SPT=32127 DPT=443 WINDOW=0 RES=0x00 RST URGP=0 
Nov 29 15:02:11 server kernel: [ 2695.782249] [UFW BLOCK] IN=ens4 OUT= MAC=xx:xx:xx:... SRC=CLIENT.IP DST=SERVER.IP LEN=83 TOS=0x00 PREC=0x00 TTL=121 ID=27041 DF PROTO=TCP SPT=32123 DPT=443 WINDOW=378 RES=0x00 ACK PSH FIN URGP=0 
Nov 29 15:02:25 server kernel: [ 2709.850336] [UFW BLOCK] IN=ens4 OUT= MAC=xx:xx:xx:... SRC=CLIENT.IP DST=SERVER.IP LEN=83 TOS=0x00 PREC=0x00 TTL=121 ID=59805 DF PROTO=TCP SPT=32126 DPT=443 WINDOW=378 RES=0x00 ACK PSH FIN URGP=0 
Nov 29 15:02:43 server kernel: [ 2727.821278] [UFW BLOCK] IN=ens4 OUT= MAC=xx:xx:xx:... SRC=CLIENT.IP DST=SERVER.IP LEN=83 TOS=0x00 PREC=0x00 TTL=53 ID=48661 DF PROTO=TCP SPT=32131 DPT=443 WINDOW=398 RES=0x00 ACK PSH URGP=0

This only happens when there is this burst of traffic:

xx.xx.xx.xx - - [29/Nov/2021:15:01:42 +0000] "POST /MyAPI/doWork/ HTTP/1.1" 200 4903 "-" "okhttp/3.14.9"
xx.xx.xx.xx - - [29/Nov/2021:15:01:42 +0000] "POST /MyAPI/doWork/ HTTP/1.1" 200 5182 "-" "okhttp/3.14.9"
xx.xx.xx.xx - - [29/Nov/2021:15:01:42 +0000] "POST /MyAPI/doWork/ HTTP/1.1" 200 4903 "-" "okhttp/3.14.9"
xx.xx.xx.xx - - [29/Nov/2021:15:01:42 +0000] "POST /MyAPI/doWork/ HTTP/1.1" 200 4903 "-" "okhttp/3.14.9"
xx.xx.xx.xx - - [29/Nov/2021:15:01:42 +0000] "POST /MyAPI/doWork/ HTTP/1.1" 200 5182 "-" "okhttp/3.14.9"
xx.xx.xx.xx - - [29/Nov/2021:15:02:11 +0000] "POST /MyAPI/doWork/ HTTP/1.1" 200 4903 "-" "okhttp/3.14.9"
xx.xx.xx.xx - - [29/Nov/2021:15:02:11 +0000] "POST /MyAPI/doWork/ HTTP/1.1" 200 4903 "-" "okhttp/3.14.9"
xx.xx.xx.xx - - [29/Nov/2021:15:02:11 +0000] "POST /MyAPI/doWork/ HTTP/1.1" 200 4903 "-" "okhttp/3.14.9"
xx.xx.xx.xx - - [29/Nov/2021:15:02:11 +0000] "POST /MyAPI/doWork/ HTTP/1.1" 200 4903 "-" "okhttp/3.14.9"
xx.xx.xx.xx - - [29/Nov/2021:15:02:11 +0000] "POST /MyAPI/doWork/ HTTP/1.1" 200 4903 "-" "okhttp/3.14.9"
xx.xx.xx.xx - - [29/Nov/2021:15:02:44 +0000] "POST /MyAPI/doWork/ HTTP/1.1" 200 4903 "-" "okhttp/3.14.9"
xx.xx.xx.xx - - [29/Nov/2021:15:02:44 +0000] "POST /MyAPI/doWork/ HTTP/1.1" 200 4903 "-" "okhttp/3.14.9"
xx.xx.xx.xx - - [29/Nov/2021:15:02:44 +0000] "POST /MyAPI/doWork/ HTTP/1.1" 200 4903 "-" "okhttp/3.14.9"
xx.xx.xx.xx - - [29/Nov/2021:15:02:44 +0000] "POST /MyAPI/doWork/ HTTP/1.1" 200 4903 "-" "okhttp/3.14.9"
xx.xx.xx.xx - - [29/Nov/2021:15:02:45 +0000] "POST /MyAPI/doWork/ HTTP/1.1" 200 374 "-" "okhttp/3.14.9"
xx.xx.xx.xx - - [29/Nov/2021:15:02:42 +0000] "POST /MyAPI/doWork/ HTTP/1.1" 200 4972 "-" "okhttp/3.14.9"
xx.xx.xx.xx - - [29/Nov/2021:15:02:47 +0000] "POST /MyAPI/doWork/ HTTP/1.1" 200 374 "-" "okhttp/3.14.9"
xx.xx.xx.xx - - [29/Nov/2021:15:02:47 +0000] "POST /MyAPI/doWork/ HTTP/1.1" 200 374 "-" "okhttp/3.14.9"
xx.xx.xx.xx - - [29/Nov/2021:15:02:47 +0000] "POST /MyAPI/doWork/ HTTP/1.1" 200 374 "-" "okhttp/3.14.9"
xx.xx.xx.xx - - [29/Nov/2021:15:02:47 +0000] "POST /MyAPI/doWork/ HTTP/1.1" 200 374 "-" "okhttp/3.14.9"
xx.xx.xx.xx - - [29/Nov/2021:15:02:47 +0000] "POST /MyAPI/doWork/ HTTP/1.1" 200 374 "-" "okhttp/3.14.9"
xx.xx.xx.xx - - [29/Nov/2021:15:02:48 +0000] "POST /MyAPI/doWork/ HTTP/1.1" 200 374 "-" "okhttp/3.14.9"
xx.xx.xx.xx - - [29/Nov/2021:15:02:48 +0000] "POST /MyAPI/doWork/ HTTP/1.1" 200 374 "-" "okhttp/3.14.9"
xx.xx.xx.xx - - [29/Nov/2021:15:02:48 +0000] "POST /MyAPI/doWork/ HTTP/1.1" 200 374 "-" "okhttp/3.14.9"
xx.xx.xx.xx - - [29/Nov/2021:15:02:48 +0000] "POST /MyAPI/doWork/ HTTP/1.1" 200 374 "-" "okhttp/3.14.9"
xx.xx.xx.xx - - [29/Nov/2021:15:05:45 +0000] "POST /MyAPI/doWork/ HTTP/1.1" 200 4903 "-" "okhttp/3.14.9"
xx.xx.xx.xx - - [29/Nov/2021:15:05:45 +0000] "POST /MyAPI/doWork/ HTTP/1.1" 200 4903 "-" "okhttp/3.14.9"
xx.xx.xx.xx - - [29/Nov/2021:15:05:45 +0000] "POST /MyAPI/doWork/ HTTP/1.1" 200 5182 "-" "okhttp/3.14.9"
xx.xx.xx.xx - - [29/Nov/2021:15:05:45 +0000] "POST /MyAPI/doWork/ HTTP/1.1" 200 4903 "-" "okhttp/3.14.9"
xx.xx.xx.xx - - [29/Nov/2021:15:05:45 +0000] "POST /MyAPI/doWork/ HTTP/1.1" 200 5182 "-" "okhttp/3.14.9"

Is it possible to stop UFW from doing this?

I'm not an iptables expert, but there seem to be some rules with limits in them. I'm not sure if that's causing the issue, but they're all definitely coming directly from UFW. Here is the output of iptables -v -n -x -L:

Chain INPUT (policy DROP 1 packets, 40 bytes)
    pkts      bytes target     prot opt in     out     source               destination         
  415392 733209937 ufw-before-logging-input  all  --  *      *       0.0.0.0/0            0.0.0.0/0           
  415392 733209937 ufw-before-input  all  --  *      *       0.0.0.0/0            0.0.0.0/0           
   19847 35666218 ufw-after-input  all  --  *      *       0.0.0.0/0            0.0.0.0/0           
   19847 35666218 ufw-after-logging-input  all  --  *      *       0.0.0.0/0            0.0.0.0/0           
   19847 35666218 ufw-reject-input  all  --  *      *       0.0.0.0/0            0.0.0.0/0           
   19847 35666218 ufw-track-input  all  --  *      *       0.0.0.0/0            0.0.0.0/0           

Chain FORWARD (policy DROP 0 packets, 0 bytes)
    pkts      bytes target     prot opt in     out     source               destination         
       0        0 ufw-before-logging-forward  all  --  *      *       0.0.0.0/0            0.0.0.0/0           
       0        0 ufw-before-forward  all  --  *      *       0.0.0.0/0            0.0.0.0/0           
       0        0 ufw-after-forward  all  --  *      *       0.0.0.0/0            0.0.0.0/0           
       0        0 ufw-after-logging-forward  all  --  *      *       0.0.0.0/0            0.0.0.0/0           
       0        0 ufw-reject-forward  all  --  *      *       0.0.0.0/0            0.0.0.0/0           
       0        0 ufw-track-forward  all  --  *      *       0.0.0.0/0            0.0.0.0/0           

Chain OUTPUT (policy ACCEPT 4 packets, 280 bytes)
    pkts      bytes target     prot opt in     out     source               destination         
  312279 36416414 ufw-before-logging-output  all  --  *      *       0.0.0.0/0            0.0.0.0/0           
  312279 36416414 ufw-before-output  all  --  *      *       0.0.0.0/0            0.0.0.0/0           
   16031  1176867 ufw-after-output  all  --  *      *       0.0.0.0/0            0.0.0.0/0           
   16031  1176867 ufw-after-logging-output  all  --  *      *       0.0.0.0/0            0.0.0.0/0           
   16031  1176867 ufw-reject-output  all  --  *      *       0.0.0.0/0            0.0.0.0/0           
   16031  1176867 ufw-track-output  all  --  *      *       0.0.0.0/0            0.0.0.0/0           

Chain ufw-after-forward (1 references)
    pkts      bytes target     prot opt in     out     source               destination         

Chain ufw-after-input (1 references)
    pkts      bytes target     prot opt in     out     source               destination         
       0        0 ufw-skip-to-policy-input  udp  --  *      *       0.0.0.0/0            0.0.0.0/0            udp dpt:137
       0        0 ufw-skip-to-policy-input  udp  --  *      *       0.0.0.0/0            0.0.0.0/0            udp dpt:138
       0        0 ufw-skip-to-policy-input  tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            tcp dpt:139
       0        0 ufw-skip-to-policy-input  tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            tcp dpt:445
       0        0 ufw-skip-to-policy-input  udp  --  *      *       0.0.0.0/0            0.0.0.0/0            udp dpt:67
       0        0 ufw-skip-to-policy-input  udp  --  *      *       0.0.0.0/0            0.0.0.0/0            udp dpt:68
       0        0 ufw-skip-to-policy-input  all  --  *      *       0.0.0.0/0            0.0.0.0/0            ADDRTYPE match dst-type BROADCAST

Chain ufw-after-logging-forward (1 references)
    pkts      bytes target     prot opt in     out     source               destination         
       0        0 LOG        all  --  *      *       0.0.0.0/0            0.0.0.0/0            limit: avg 3/min burst 10 LOG flags 0 level 4 prefix "[UFW BLOCK] "

Chain ufw-after-logging-input (1 references)
    pkts      bytes target     prot opt in     out     source               destination         
       1       40 LOG        all  --  *      *       0.0.0.0/0            0.0.0.0/0            limit: avg 3/min burst 10 LOG flags 0 level 4 prefix "[UFW BLOCK] "

Chain ufw-after-logging-output (1 references)
    pkts      bytes target     prot opt in     out     source               destination         
      82     6813 LOG        all  --  *      *       0.0.0.0/0            0.0.0.0/0            limit: avg 3/min burst 10 LOG flags 0 level 4 prefix "[UFW ALLOW] "

Chain ufw-after-output (1 references)
    pkts      bytes target     prot opt in     out     source               destination         

Chain ufw-before-forward (1 references)
    pkts      bytes target     prot opt in     out     source               destination         
       0        0 ACCEPT     all  --  *      *       0.0.0.0/0            0.0.0.0/0            ctstate RELATED,ESTABLISHED
       0        0 ACCEPT     icmp --  *      *       0.0.0.0/0            0.0.0.0/0            icmptype 3
       0        0 ACCEPT     icmp --  *      *       0.0.0.0/0            0.0.0.0/0            icmptype 11
       0        0 ACCEPT     icmp --  *      *       0.0.0.0/0            0.0.0.0/0            icmptype 12
       0        0 ACCEPT     icmp --  *      *       0.0.0.0/0            0.0.0.0/0            icmptype 8
       0        0 ufw-user-forward  all  --  *      *       0.0.0.0/0            0.0.0.0/0           

Chain ufw-before-input (1 references)
    pkts      bytes target     prot opt in     out     source               destination         
    2452   204587 ACCEPT     all  --  lo     *       0.0.0.0/0            0.0.0.0/0           
  205497 387613737 ACCEPT     all  --  *      *       0.0.0.0/0            0.0.0.0/0            ctstate RELATED,ESTABLISHED
     113     6588 ufw-logging-deny  all  --  *      *       0.0.0.0/0            0.0.0.0/0            ctstate INVALID
     113     6588 DROP       all  --  *      *       0.0.0.0/0            0.0.0.0/0            ctstate INVALID
       0        0 ACCEPT     icmp --  *      *       0.0.0.0/0            0.0.0.0/0            icmptype 3
       0        0 ACCEPT     icmp --  *      *       0.0.0.0/0            0.0.0.0/0            icmptype 11
       0        0 ACCEPT     icmp --  *      *       0.0.0.0/0            0.0.0.0/0            icmptype 12
       0        0 ACCEPT     icmp --  *      *       0.0.0.0/0            0.0.0.0/0            icmptype 8
       0        0 ACCEPT     udp  --  *      *       0.0.0.0/0            0.0.0.0/0            udp spt:67 dpt:68
     926    66054 ufw-not-local  all  --  *      *       0.0.0.0/0            0.0.0.0/0           
       0        0 ACCEPT     udp  --  *      *       0.0.0.0/0            224.0.0.251          udp dpt:5353
       0        0 ACCEPT     udp  --  *      *       0.0.0.0/0            239.255.255.250      udp dpt:1900
     925    64664 ufw-user-input  all  --  *      *       0.0.0.0/0            0.0.0.0/0           

Chain ufw-before-logging-forward (1 references)
    pkts      bytes target     prot opt in     out     source               destination         
       0        0 LOG        all  --  *      *       0.0.0.0/0            0.0.0.0/0            ctstate NEW limit: avg 3/min burst 10 LOG flags 0 level 4 prefix "[UFW AUDIT] "

Chain ufw-before-logging-input (1 references)
    pkts      bytes target     prot opt in     out     source               destination         
      82    10488 LOG        all  --  *      *       0.0.0.0/0            0.0.0.0/0            ctstate NEW limit: avg 3/min burst 10 LOG flags 0 level 4 prefix "[UFW AUDIT] "

Chain ufw-before-logging-output (1 references)
    pkts      bytes target     prot opt in     out     source               destination         
      82     5743 LOG        all  --  *      *       0.0.0.0/0            0.0.0.0/0            ctstate NEW limit: avg 3/min burst 10 LOG flags 0 level 4 prefix "[UFW AUDIT] "

Chain ufw-before-output (1 references)
    pkts      bytes target     prot opt in     out     source               destination         
    2452   204587 ACCEPT     all  --  *      lo      0.0.0.0/0            0.0.0.0/0           
  155111 15748513 ACCEPT     all  --  *      *       0.0.0.0/0            0.0.0.0/0            ctstate RELATED,ESTABLISHED
     229    16406 ufw-user-output  all  --  *      *       0.0.0.0/0            0.0.0.0/0           

Chain ufw-logging-allow (0 references)
    pkts      bytes target     prot opt in     out     source               destination         
       0        0 LOG        all  --  *      *       0.0.0.0/0            0.0.0.0/0            limit: avg 3/min burst 10 LOG flags 0 level 4 prefix "[UFW ALLOW] "

Chain ufw-logging-deny (2 references)
    pkts      bytes target     prot opt in     out     source               destination         
      45     2384 LOG        all  --  *      *       0.0.0.0/0            0.0.0.0/0            ctstate INVALID limit: avg 3/min burst 10 LOG flags 0 level 4 prefix "[UFW AUDIT INVALID] "
      45     2384 LOG        all  --  *      *       0.0.0.0/0            0.0.0.0/0            limit: avg 3/min burst 10 LOG flags 0 level 4 prefix "[UFW BLOCK] "

Chain ufw-not-local (1 references)
    pkts      bytes target     prot opt in     out     source               destination         
     926    66054 RETURN     all  --  *      *       0.0.0.0/0            0.0.0.0/0            ADDRTYPE match dst-type LOCAL
       0        0 RETURN     all  --  *      *       0.0.0.0/0            0.0.0.0/0            ADDRTYPE match dst-type MULTICAST
       0        0 RETURN     all  --  *      *       0.0.0.0/0            0.0.0.0/0            ADDRTYPE match dst-type BROADCAST
       0        0 ufw-logging-deny  all  --  *      *       0.0.0.0/0            0.0.0.0/0            limit: avg 3/min burst 10
       0        0 DROP       all  --  *      *       0.0.0.0/0            0.0.0.0/0           

Chain ufw-reject-forward (1 references)
    pkts      bytes target     prot opt in     out     source               destination         

Chain ufw-reject-input (1 references)
    pkts      bytes target     prot opt in     out     source               destination         

Chain ufw-reject-output (1 references)
    pkts      bytes target     prot opt in     out     source               destination         

Chain ufw-skip-to-policy-forward (0 references)
    pkts      bytes target     prot opt in     out     source               destination         
       0        0 DROP       all  --  *      *       0.0.0.0/0            0.0.0.0/0           

Chain ufw-skip-to-policy-input (7 references)
    pkts      bytes target     prot opt in     out     source               destination         
       0        0 DROP       all  --  *      *       0.0.0.0/0            0.0.0.0/0           

Chain ufw-skip-to-policy-output (0 references)
    pkts      bytes target     prot opt in     out     source               destination         
       0        0 ACCEPT     all  --  *      *       0.0.0.0/0            0.0.0.0/0           

Chain ufw-track-forward (1 references)
    pkts      bytes target     prot opt in     out     source               destination         

Chain ufw-track-input (1 references)
    pkts      bytes target     prot opt in     out     source               destination         

Chain ufw-track-output (1 references)
    pkts      bytes target     prot opt in     out     source               destination         
     188    12756 ACCEPT     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            ctstate NEW
      37     3370 ACCEPT     udp  --  *      *       0.0.0.0/0            0.0.0.0/0            ctstate NEW

Chain ufw-user-forward (1 references)
    pkts      bytes target     prot opt in     out     source               destination         

Chain ufw-user-input (1 references)
    pkts      bytes target     prot opt in     out     source               destination         
       2      108 ACCEPT     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            tcp dpt:22
       1       52 ACCEPT     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            tcp dpt:80
     921    64464 ACCEPT     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            tcp dpt:443

Chain ufw-user-limit (0 references)
    pkts      bytes target     prot opt in     out     source               destination         
       0        0 LOG        all  --  *      *       0.0.0.0/0            0.0.0.0/0            limit: avg 3/min burst 5 LOG flags 0 level 4 prefix "[UFW LIMIT BLOCK] "
       0        0 REJECT     all  --  *      *       0.0.0.0/0            0.0.0.0/0            reject-with icmp-port-unreachable

Chain ufw-user-limit-accept (0 references)
    pkts      bytes target     prot opt in     out     source               destination         
       0        0 ACCEPT     all  --  *      *       0.0.0.0/0            0.0.0.0/0           

Chain ufw-user-logging-forward (0 references)
    pkts      bytes target     prot opt in     out     source               destination         

Chain ufw-user-logging-input (0 references)
    pkts      bytes target     prot opt in     out     source               destination         

Chain ufw-user-logging-output (0 references)
    pkts      bytes target     prot opt in     out     source               destination         

Chain ufw-user-output (1 references)
    pkts      bytes target     prot opt in     out     source               destination

Edit:

Adding full log for a specific IP that was blocked to include the AUDIT lines as well.

cat ufw.log | grep CLIENT.IP

Nov 29 16:00:14 server kernel: [ 6178.504254] [UFW AUDIT INVALID] IN=ens4 OUT= MAC=xx:xx:xx:... SRC=CLIENT.IP DST=SERVER.IP LEN=52 TOS=0x00 PREC=0x00 TTL=121 ID=27580 DF PROTO=TCP SPT=64750 DPT=443 WINDOW=1539 RES=0x00 ACK FIN URGP=0 
Nov 29 16:00:14 server kernel: [ 6178.504263] [UFW BLOCK] IN=ens4 OUT= MAC=xx:xx:xx:... SRC=CLIENT.IP DST=SERVER.IP LEN=52 TOS=0x00 PREC=0x00 TTL=121 ID=27580 DF PROTO=TCP SPT=64750 DPT=443 WINDOW=1539 RES=0x00 ACK FIN URGP=0 
Nov 29 16:00:15 server kernel: [ 6178.733987] [UFW AUDIT INVALID] IN=ens4 OUT= MAC=xx:xx:xx:... SRC=CLIENT.IP DST=SERVER.IP LEN=40 TOS=0x00 PREC=0x00 TTL=53 ID=13970 DF PROTO=TCP SPT=64753 DPT=443 WINDOW=0 RES=0x00 RST URGP=0 
Nov 29 16:00:15 server kernel: [ 6178.734039] [UFW BLOCK] IN=ens4 OUT= MAC=xx:xx:xx:... SRC=CLIENT.IP DST=SERVER.IP LEN=40 TOS=0x00 PREC=0x00 TTL=53 ID=13970 DF PROTO=TCP SPT=64753 DPT=443 WINDOW=0 RES=0x00 RST URGP=0 
Nov 29 16:00:15 server kernel: [ 6178.742917] [UFW AUDIT INVALID] IN=ens4 OUT= MAC=xx:xx:xx:... SRC=CLIENT.IP DST=SERVER.IP LEN=40 TOS=0x00 PREC=0x00 TTL=53 ID=13974 DF PROTO=TCP SPT=64754 DPT=443 WINDOW=0 RES=0x00 RST URGP=0 
Nov 29 16:00:15 server kernel: [ 6178.742943] [UFW BLOCK] IN=ens4 OUT= MAC=xx:xx:xx:... SRC=CLIENT.IP DST=SERVER.IP LEN=40 TOS=0x00 PREC=0x00 TTL=53 ID=13974 DF PROTO=TCP SPT=64754 DPT=443 WINDOW=0 RES=0x00 RST URGP=0 
Nov 29 16:00:15 server kernel: [ 6178.893083] [UFW AUDIT INVALID] IN=ens4 OUT= MAC=xx:xx:xx:... SRC=CLIENT.IP DST=SERVER.IP LEN=75 TOS=0x00 PREC=0x00 TTL=121 ID=27581 DF PROTO=TCP SPT=64750 DPT=443 WINDOW=1539 RES=0x00 ACK PSH FIN URGP=0 
Nov 29 16:00:15 server kernel: [ 6178.893115] [UFW BLOCK] IN=ens4 OUT= MAC=xx:xx:xx:... SRC=CLIENT.IP DST=SERVER.IP LEN=75 TOS=0x00 PREC=0x00 TTL=121 ID=27581 DF PROTO=TCP SPT=64750 DPT=443 WINDOW=1539 RES=0x00 ACK PSH FIN URGP=0 
Nov 29 16:00:15 server kernel: [ 6178.915281] [UFW AUDIT INVALID] IN=ens4 OUT= MAC=xx:xx:xx:... SRC=CLIENT.IP DST=SERVER.IP LEN=75 TOS=0x00 PREC=0x00 TTL=121 ID=27199 DF PROTO=TCP SPT=64752 DPT=443 WINDOW=360 RES=0x00 ACK PSH FIN URGP=0 
Nov 29 16:00:15 server kernel: [ 6178.915314] [UFW BLOCK] IN=ens4 OUT= MAC=xx:xx:xx:... SRC=CLIENT.IP DST=SERVER.IP LEN=75 TOS=0x00 PREC=0x00 TTL=121 ID=27199 DF PROTO=TCP SPT=64752 DPT=443 WINDOW=360 RES=0x00 ACK PSH FIN URGP=0 
Nov 29 16:00:15 server kernel: [ 6178.947972] [UFW AUDIT INVALID] IN=ens4 OUT= MAC=xx:xx:xx:... SRC=CLIENT.IP DST=SERVER.IP LEN=75 TOS=0x00 PREC=0x00 TTL=121 ID=41365 DF PROTO=TCP SPT=64753 DPT=443 WINDOW=1003 RES=0x00 ACK PSH FIN URGP=0 
Nov 29 16:00:15 server kernel: [ 6178.948004] [UFW BLOCK] IN=ens4 OUT= MAC=xx:xx:xx:... SRC=CLIENT.IP DST=SERVER.IP LEN=75 TOS=0x00 PREC=0x00 TTL=121 ID=41365 DF PROTO=TCP SPT=64753 DPT=443 WINDOW=1003 RES=0x00 ACK PSH FIN URGP=0 
Nov 29 16:00:15 server kernel: [ 6178.951350] [UFW AUDIT INVALID] IN=ens4 OUT= MAC=xx:xx:xx:... SRC=CLIENT.IP DST=SERVER.IP LEN=75 TOS=0x00 PREC=0x00 TTL=121 ID=17364 DF PROTO=TCP SPT=64754 DPT=443 WINDOW=401 RES=0x00 ACK PSH FIN URGP=0 
Nov 29 16:00:15 server kernel: [ 6178.951379] [UFW BLOCK] IN=ens4 OUT= MAC=xx:xx:xx:... SRC=CLIENT.IP DST=SERVER.IP LEN=75 TOS=0x00 PREC=0x00 TTL=121 ID=17364 DF PROTO=TCP SPT=64754 DPT=443 WINDOW=401 RES=0x00 ACK PSH FIN URGP=0 
Nov 29 16:00:16 server kernel: [ 6179.679941] [UFW AUDIT INVALID] IN=ens4 OUT= MAC=xx:xx:xx:... SRC=CLIENT.IP DST=SERVER.IP LEN=75 TOS=0x00 PREC=0x00 TTL=121 ID=27582 DF PROTO=TCP SPT=64750 DPT=443 WINDOW=1539 RES=0x00 ACK PSH FIN URGP=0 
Nov 29 16:00:16 server kernel: [ 6179.679968] [UFW BLOCK] IN=ens4 OUT= MAC=xx:xx:xx:... SRC=CLIENT.IP DST=SERVER.IP LEN=75 TOS=0x00 PREC=0x00 TTL=121 ID=27582 DF PROTO=TCP SPT=64750 DPT=443 WINDOW=1539 RES=0x00 ACK PSH FIN URGP=0 
Nov 29 16:00:20 server kernel: [ 6184.407553] [UFW AUDIT INVALID] IN=ens4 OUT= MAC=xx:xx:xx:... SRC=CLIENT.IP DST=SERVER.IP LEN=75 TOS=0x00 PREC=0x00 TTL=121 ID=27584 DF PROTO=TCP SPT=64750 DPT=443 WINDOW=1539 RES=0x00 ACK PSH FIN URGP=0 
Nov 29 16:00:20 server kernel: [ 6184.407579] [UFW BLOCK] IN=ens4 OUT= MAC=xx:xx:xx:... SRC=CLIENT.IP DST=SERVER.IP LEN=75 TOS=0x00 PREC=0x00 TTL=121 ID=27584 DF PROTO=TCP SPT=64750 DPT=443 WINDOW=1539 RES=0x00 ACK PSH FIN URGP=0 
Nov 29 16:00:53 server kernel: [ 6217.594359] [UFW AUDIT INVALID] IN=ens4 OUT= MAC=xx:xx:xx:... SRC=CLIENT.IP DST=SERVER.IP LEN=40 TOS=0x00 PREC=0x00 TTL=53 ID=15646 DF PROTO=TCP SPT=64762 DPT=443 WINDOW=0 RES=0x00 RST URGP=0 
Nov 29 16:00:53 server kernel: [ 6217.594397] [UFW BLOCK] IN=ens4 OUT= MAC=xx:xx:xx:... SRC=CLIENT.IP DST=SERVER.IP LEN=40 TOS=0x00 PREC=0x00 TTL=53 ID=15646 DF PROTO=TCP SPT=64762 DPT=443 WINDOW=0 RES=0x00 RST URGP=0 
Nov 29 16:01:00 server kernel: [ 6224.389153] [UFW AUDIT INVALID] IN=ens4 OUT= MAC=xx:xx:xx:... SRC=CLIENT.IP DST=SERVER.IP LEN=75 TOS=0x00 PREC=0x00 TTL=121 ID=40585 DF PROTO=TCP SPT=64762 DPT=443 WINDOW=345 RES=0x00 ACK PSH FIN URGP=0 
Nov 29 16:01:00 server kernel: [ 6224.389187] [UFW BLOCK] IN=ens4 OUT= MAC=xx:xx:xx:... SRC=CLIENT.IP DST=SERVER.IP LEN=75 TOS=0x00 PREC=0x00 TTL=121 ID=40585 DF PROTO=TCP SPT=64762 DPT=443 WINDOW=345 RES=0x00 ACK PSH FIN URGP=0 
Nov 29 16:01:23 server kernel: [ 6247.359229] [UFW AUDIT INVALID] IN=ens4 OUT= MAC=xx:xx:xx:... SRC=CLIENT.IP DST=SERVER.IP LEN=40 TOS=0x00 PREC=0x00 TTL=53 ID=21101 DF PROTO=TCP SPT=64771 DPT=443 WINDOW=0 RES=0x00 RST URGP=0 
Nov 29 16:01:23 server kernel: [ 6247.359287] [UFW BLOCK] IN=ens4 OUT= MAC=xx:xx:xx:... SRC=CLIENT.IP DST=SERVER.IP LEN=40 TOS=0x00 PREC=0x00 TTL=53 ID=21101 DF PROTO=TCP SPT=64771 DPT=443 WINDOW=0 RES=0x00 RST URGP=0 
Nov 29 16:01:42 server kernel: [ 6266.470287] [UFW AUDIT INVALID] IN=ens4 OUT= MAC=xx:xx:xx:... SRC=CLIENT.IP DST=SERVER.IP LEN=52 TOS=0x00 PREC=0x00 TTL=52 ID=1563 DF PROTO=TCP SPT=64772 DPT=443 WINDOW=352 RES=0x00 ACK FIN URGP=0 
Nov 29 16:01:42 server kernel: [ 6266.470328] [UFW BLOCK] IN=ens4 OUT= MAC=xx:xx:xx:... SRC=CLIENT.IP DST=SERVER.IP LEN=52 TOS=0x00 PREC=0x00 TTL=52 ID=1563 DF PROTO=TCP SPT=64772 DPT=443 WINDOW=352 RES=0x00 ACK FIN URGP=0 
Nov 29 16:02:02 server kernel: [ 6285.692541] [UFW AUDIT INVALID] IN=ens4 OUT= MAC=xx:xx:xx:... SRC=CLIENT.IP DST=SERVER.IP LEN=52 TOS=0x00 PREC=0x00 TTL=52 ID=1564 DF PROTO=TCP SPT=64772 DPT=443 WINDOW=352 RES=0x00 ACK FIN URGP=0 
Nov 29 16:02:02 server kernel: [ 6285.692584] [UFW BLOCK] IN=ens4 OUT= MAC=xx:xx:xx:... SRC=CLIENT.IP DST=SERVER.IP LEN=52 TOS=0x00 PREC=0x00 TTL=52 ID=1564 DF PROTO=TCP SPT=64772 DPT=443 WINDOW=352 RES=0x00 ACK FIN URGP=0 
Nov 29 16:09:45 server kernel: [ 6749.404864] [UFW AUDIT] IN=ens4 OUT= MAC=xx:xx:xx:... SRC=CLIENT.IP DST=SERVER.IP LEN=60 TOS=0x00 PREC=0x00 TTL=52 ID=58119 DF PROTO=TCP SPT=64788 DPT=443 WINDOW=65535 RES=0x00 SYN URGP=0
178
0 + 0
ufw
Doug Smythies avatar
gn flag
Doug Smythies
For each of your `BLOCK` entries, please look in the file for a corresponding `UFW AUDIT INVALID` entry. Except for the last one, they are all session termination related, which happens often with linux because it uses half-duplex handshake and not full duplex handshake. I am saying there might not be an issue at all. Note that UFW rules are hard to follow and it uses the same log prefix in multiple locations, very annoying.
2
Reply
Doug Smythies avatar
gn flag
Doug Smythies
see [here](https://askubuntu.com/questions/1187993/ufw-occasionally-blocking-https-443-tcp-although-configured-to-allow-that-port/1188016#1188016). I do not have a good explanation for the last BLOCK example, and would need to see it in context of a tcpdump capture of the entire session.
0
Reply
ca flag
kleinric
Hi @DougSmythies, thanks! I've added the `UFW AUDIT INVALID` lines as well now. What's the correct command/flags to get the right tcpdump? I suppose I'd just need to run that for a while capturing until a block happens right?
0
Reply
ca flag
kleinric
So on the client-side, when the block happens it does seem to be dropping things - i.e. the app connection fails. If I disable UFW then it doesn't seem to happen.
0
Reply
Doug Smythies avatar
gn flag
Doug Smythies
O.K. so your example, `cat ufw.log | grep CLIENT.IP`, seems fine, and as I mentioned. The last entry `UFW AUDIT` also seems fine. I'll have to come back with a tcpdump suggestion.
0
Reply
Doug Smythies avatar
gn flag
Doug Smythies
I really can not see what might be wrong with your UFW generated iptables rule set. I don't know if it will help, but for tcpdump I can think of two options: By port, `sudo tcpdump -n -tttt -i ens4 port 443` and extract via IP later with grep; By IP, `sudo tcpdump -n -tttt -i ens4 host CLIENT.IP` and extract only port 443 stuff later if needed.
0
Reply
ca flag
kleinric
Thanks! I'll run this tomorrow morning when all the clients are connecting again. Will have a look and report back. Thanks so much for your help!
0
Reply
Score:0
Doug Smythies avatar Ubuntu
gn flag
Doug Smythies

If we take the posted output for cat ufw.log | grep CLIENT.IP and regroup some it into individual sessions, based on client port number we get:

Nov 29 16:00:15 server kernel: [ 6178.733987] [UFW AUDIT INVALID] IN=ens4 OUT= MAC=xx:xx:xx:... SRC=CLIENT.IP DST=SERVER.IP LEN=40 TOS=0x00 PREC=0x00 TTL=53 ID=13970 DF PROTO=TCP SPT=64753 DPT=443 WINDOW=0 RES=0x00 RST URGP=0 
Nov 29 16:00:15 server kernel: [ 6178.734039] [UFW BLOCK] IN=ens4 OUT= MAC=xx:xx:xx:... SRC=CLIENT.IP DST=SERVER.IP LEN=40 TOS=0x00 PREC=0x00 TTL=53 ID=13970 DF PROTO=TCP SPT=64753 DPT=443 WINDOW=0 RES=0x00 RST URGP=0 
Nov 29 16:00:15 server kernel: [ 6178.947972] [UFW AUDIT INVALID] IN=ens4 OUT= MAC=xx:xx:xx:... SRC=CLIENT.IP DST=SERVER.IP LEN=75 TOS=0x00 PREC=0x00 TTL=121 ID=41365 DF PROTO=TCP SPT=64753 DPT=443 WINDOW=1003 RES=0x00 ACK PSH FIN URGP=0 
Nov 29 16:00:15 server kernel: [ 6178.948004] [UFW BLOCK] IN=ens4 OUT= MAC=xx:xx:xx:... SRC=CLIENT.IP DST=SERVER.IP LEN=75 TOS=0x00 PREC=0x00 TTL=121 ID=41365 DF PROTO=TCP SPT=64753 DPT=443 WINDOW=1003 RES=0x00 ACK PSH FIN URGP=0 

Nov 29 16:00:15 server kernel: [ 6178.742917] [UFW AUDIT INVALID] IN=ens4 OUT= MAC=xx:xx:xx:... SRC=CLIENT.IP DST=SERVER.IP LEN=40 TOS=0x00 PREC=0x00 TTL=53 ID=13974 DF PROTO=TCP SPT=64754 DPT=443 WINDOW=0 RES=0x00 RST URGP=0 
Nov 29 16:00:15 server kernel: [ 6178.742943] [UFW BLOCK] IN=ens4 OUT= MAC=xx:xx:xx:... SRC=CLIENT.IP DST=SERVER.IP LEN=40 TOS=0x00 PREC=0x00 TTL=53 ID=13974 DF PROTO=TCP SPT=64754 DPT=443 WINDOW=0 RES=0x00 RST URGP=0 
Nov 29 16:00:15 server kernel: [ 6178.951350] [UFW AUDIT INVALID] IN=ens4 OUT= MAC=xx:xx:xx:... SRC=CLIENT.IP DST=SERVER.IP LEN=75 TOS=0x00 PREC=0x00 TTL=121 ID=17364 DF PROTO=TCP SPT=64754 DPT=443 WINDOW=401 RES=0x00 ACK PSH FIN URGP=0 
Nov 29 16:00:15 server kernel: [ 6178.951379] [UFW BLOCK] IN=ens4 OUT= MAC=xx:xx:xx:... SRC=CLIENT.IP DST=SERVER.IP LEN=75 TOS=0x00 PREC=0x00 TTL=121 ID=17364 DF PROTO=TCP SPT=64754 DPT=443 WINDOW=401 RES=0x00 ACK PSH FIN URGP=0 
Nov 29 16:00:15 server kernel: [ 6178.915281] [UFW AUDIT INVALID] IN=ens4 OUT= MAC=xx:xx:xx:... SRC=CLIENT.IP DST=SERVER.IP LEN=75 TOS=0x00 PREC=0x00 TTL=121 ID=27199 DF PROTO=TCP SPT=64752 DPT=443 WINDOW=360 RES=0x00 ACK PSH FIN URGP=0 
Nov 29 16:00:15 server kernel: [ 6178.915314] [UFW BLOCK] IN=ens4 OUT= MAC=xx:xx:xx:... SRC=CLIENT.IP DST=SERVER.IP LEN=75 TOS=0x00 PREC=0x00 TTL=121 ID=27199 DF PROTO=TCP SPT=64752 DPT=443 WINDOW=360 RES=0x00 ACK PSH FIN URGP=0

Nov 29 16:00:53 server kernel: [ 6217.594359] [UFW AUDIT INVALID] IN=ens4 OUT= MAC=xx:xx:xx:... SRC=CLIENT.IP DST=SERVER.IP LEN=40 TOS=0x00 PREC=0x00 TTL=53 ID=15646 DF PROTO=TCP SPT=64762 DPT=443 WINDOW=0 RES=0x00 RST URGP=0 
Nov 29 16:00:53 server kernel: [ 6217.594397] [UFW BLOCK] IN=ens4 OUT= MAC=xx:xx:xx:... SRC=CLIENT.IP DST=SERVER.IP LEN=40 TOS=0x00 PREC=0x00 TTL=53 ID=15646 DF PROTO=TCP SPT=64762 DPT=443 WINDOW=0 RES=0x00 RST URGP=0 
Nov 29 16:01:00 server kernel: [ 6224.389153] [UFW AUDIT INVALID] IN=ens4 OUT= MAC=xx:xx:xx:... SRC=CLIENT.IP DST=SERVER.IP LEN=75 TOS=0x00 PREC=0x00 TTL=121 ID=40585 DF PROTO=TCP SPT=64762 DPT=443 WINDOW=345 RES=0x00 ACK PSH FIN URGP=0 
Nov 29 16:01:00 server kernel: [ 6224.389187] [UFW BLOCK] IN=ens4 OUT= MAC=xx:xx:xx:... SRC=CLIENT.IP DST=SERVER.IP LEN=75 TOS=0x00 PREC=0x00 TTL=121 ID=40585 DF PROTO=TCP SPT=64762 DPT=443 WINDOW=345 RES=0x00 ACK PSH FIN URGP=0

Now, observe there appears to be some sequence inconsistencies with some ACK FIN packets coming after RST packets and always with a very different TTL (Time To Live). All of these packets come after the TCP session has already been closed and forgotten about.

I don't know anything about okhttp, but if socket sharing is somehow being used, there might be timing issues in closing the socket.

See also another answer here, but I'll copy the relevant part, rewording a little:

For TCP connections, Linux tends to use a "half-duplex" close sequence where either side of the session can initiate connection termination via a single 2 way FIN-ACK handshake (which puts the connection into the CLOSE_WAIT state), instead of a full 4 way FIN-ACK handshake. What very often happens, particularly with a router in between, is one side thinks the session has been closed and the other doesn't. Your server computer has terminated and forgotten about the session, and so considers the packets as invalid new session opening packets and blocks them. Normally no harm done, the actual session worked fine. What is unusual in this case is the RST (Reset) packet before some FIN-ACK packets.

0 + 0
ธงชาติไทย
Elon Musk
I sit in a Tesla and translated this thread with Ai:

mangohost

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.