Join Log in
Score:0
Ivan avatar Server

Proxmox host cannot reach guest: TCP client retransmitting instead of sending ACK after SYN/ACK

in flag
Ivan

Setup: server (HTTP server on 80) on 192.168.1.20, clients on 192.168.1.17, 192.168.1.18

Client 192.168.1.17 can connect to the server fine (Wireshark capture on the client side attached)

1   0.000000    192.168.1.17    192.168.1.20    TCP 78  62275 → 80 [SYN] Seq=0 Win=65535 Len=0 MSS=1460 WS=64 TSval=2874634337 TSecr=0 SACK_PERM
2   0.001393    192.168.1.20    192.168.1.17    TCP 74  80 → 62275 [SYN, ACK] Seq=0 Ack=1 Win=65160 Len=0 MSS=1460 SACK_PERM TSval=3567464873 TSecr=2874634337 WS=128
3   0.001447    192.168.1.17    192.168.1.20    TCP 66  62275 → 80 [ACK] Seq=1 Ack=1 Win=131712 Len=0 TSval=2874634339 TSecr=3567464873
4   0.001510    192.168.1.17    192.168.1.20    HTTP    142 GET / HTTP/1.1 
5   0.002609    192.168.1.20    192.168.1.17    TCP 66  80 → 62275 [ACK] Seq=1 Ack=77 Win=65152 Len=0 TSval=3567464874 TSecr=2874634339
6   0.002609    192.168.1.20    192.168.1.17    HTTP    431 HTTP/1.1 301 Moved Permanently  (text/html)
7   0.002688    192.168.1.17    192.168.1.20    TCP 66  62275 → 80 [ACK] Seq=77 Ack=366 Win=131392 Len=0 TSval=2874634340 TSecr=3567464874
8   0.002859    192.168.1.17    192.168.1.20    TCP 66  62275 → 80 [FIN, ACK] Seq=77 Ack=366 Win=131392 Len=0 TSval=2874634340 TSecr=3567464874
9   0.003468    192.168.1.20    192.168.1.17    TCP 66  80 → 62275 [FIN, ACK] Seq=366 Ack=78 Win=65152 Len=0 TSval=3567464875 TSecr=2874634340
10  0.003551    192.168.1.17    192.168.1.20    TCP 66  62275 → 80 [ACK] Seq=78 Ack=367 Win=131392 Len=0 TSval=2874634340 TSecr=3567464875

While client 192.168.1.18 cannot connect to the server (Wireshark capture on the client side attached). It keeps re-transmitting SYN instead of ACKing the SYN/ACK from the server.

1   0.000000    192.168.1.18    192.168.1.20    TCP 74  40098 → 80 [SYN] Seq=0 Win=64240 Len=0 MSS=1460 SACK_PERM TSval=3825119753 TSecr=0 WS=128
2   0.000414    192.168.1.20    192.168.1.18    TCP 74  80 → 40098 [SYN, ACK] Seq=0 Ack=1 Win=65160 Len=0 MSS=1460 SACK_PERM TSval=1947289254 TSecr=3825119753 WS=128
3   1.009974    192.168.1.18    192.168.1.20    TCP 74  [TCP Retransmission] 40098 → 80 [SYN] Seq=0 Win=64240 Len=0 MSS=1460 SACK_PERM TSval=3825120763 TSecr=0 WS=128
4   1.010796    192.168.1.20    192.168.1.18    TCP 74  [TCP Retransmission] 80 → 40098 [SYN, ACK] Seq=0 Ack=1 Win=65160 Len=0 MSS=1460 SACK_PERM TSval=1947290264 TSecr=3825119753 WS=128
5   2.020735    192.168.1.20    192.168.1.18    TCP 74  [TCP Retransmission] 80 → 40098 [SYN, ACK] Seq=0 Ack=1 Win=65160 Len=0 MSS=1460 SACK_PERM TSval=1947291274 TSecr=3825119753 WS=128
6   3.022183    192.168.1.18    192.168.1.20    TCP 74  [TCP Retransmission] 40098 → 80 [SYN] Seq=0 Win=64240 Len=0 MSS=1460 SACK_PERM TSval=3825122775 TSecr=0 WS=128
7   3.022929    192.168.1.20    192.168.1.18    TCP 74  [TCP Retransmission] 80 → 40098 [SYN, ACK] Seq=0 Ack=1 Win=65160 Len=0 MSS=1460 SACK_PERM TSval=1947292276 TSecr=3825119753 WS=128
8   5.024851    192.168.1.20    192.168.1.18    TCP 74  [TCP Retransmission] 80 → 40098 [SYN, ACK] Seq=0 Ack=1 Win=65160 Len=0 MSS=1460 SACK_PERM TSval=1947294278 TSecr=3825119753 WS=128
9   7.181980    192.168.1.18    192.168.1.20    TCP 74  [TCP Retransmission] 40098 → 80 [SYN] Seq=0 Win=64240 Len=0 MSS=1460 SACK_PERM TSval=3825126935 TSecr=0 WS=128
10  7.182639    192.168.1.20    192.168.1.18    TCP 74  [TCP Retransmission] 80 → 40098 [SYN, ACK] Seq=0 Ack=1 Win=65160 Len=0 MSS=1460 SACK_PERM TSval=1947296436 TSecr=3825119753 WS=128

Both clients are in the same LAN. The caveat is that the server (192.168.1.20) is actually a VM hosted on client (192.168.1.18) and uses bridging for network access.

192.168.1.18 is a Proxmox host, while 192.168.1.20 is a Proxmox guest. The Proxmox host has iptables rules generated by Proxmox firewall.

Chain INPUT (policy ACCEPT 365 packets, 24755 bytes)
 pkts bytes target     prot opt in     out     source               destination
18194 5257K PVEFW-INPUT  all  --  *      *       0.0.0.0/0            0.0.0.0/0

Chain FORWARD (policy ACCEPT 2568 packets, 776K bytes)
 pkts bytes target     prot opt in     out     source               destination
42682   39M PVEFW-FORWARD  all  --  *      *       0.0.0.0/0            0.0.0.0/0

Chain OUTPUT (policy ACCEPT 83 packets, 5044 bytes)
 pkts bytes target     prot opt in     out     source               destination
18346 5898K PVEFW-OUTPUT  all  --  *      *       0.0.0.0/0            0.0.0.0/0

Chain PVEFW-Drop (13 references)
 pkts bytes target     prot opt in     out     source               destination
  164 36412 PVEFW-DropBroadcast  all  --  *      *       0.0.0.0/0            0.0.0.0/0
    0     0 ACCEPT     icmp --  *      *       0.0.0.0/0            0.0.0.0/0            icmptype 3 code 4
    0     0 ACCEPT     icmp --  *      *       0.0.0.0/0            0.0.0.0/0            icmptype 11
    0     0 DROP       all  --  *      *       0.0.0.0/0            0.0.0.0/0            ctstate INVALID
    0     0 DROP       udp  --  *      *       0.0.0.0/0            0.0.0.0/0            multiport dports 135,445
    0     0 DROP       udp  --  *      *       0.0.0.0/0            0.0.0.0/0            udp dpts:137:139
    0     0 DROP       udp  --  *      *       0.0.0.0/0            0.0.0.0/0            udp spt:137 dpts:1024:65535
    0     0 DROP       tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            multiport dports 135,139,445
    0     0 DROP       udp  --  *      *       0.0.0.0/0            0.0.0.0/0            udp dpt:1900
    6  6510 DROP       tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            tcp flags:!0x17/0x02
    0     0 DROP       udp  --  *      *       0.0.0.0/0            0.0.0.0/0            udp spt:53
    6  5208            all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* PVESIG:83WlR/a4wLbmURFqMQT3uJSgIG8 */

Chain PVEFW-DropBroadcast (2 references)
 pkts bytes target     prot opt in     out     source               destination
   68 15402 DROP       all  --  *      *       0.0.0.0/0            0.0.0.0/0            ADDRTYPE match dst-type BROADCAST
   84  9292 DROP       all  --  *      *       0.0.0.0/0            0.0.0.0/0            ADDRTYPE match dst-type MULTICAST
    0     0 DROP       all  --  *      *       0.0.0.0/0            0.0.0.0/0            ADDRTYPE match dst-type ANYCAST
    0     0 DROP       all  --  *      *       0.0.0.0/0            224.0.0.0/4
   12 11718            all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* PVESIG:NyjHNAtFbkH7WGLamPpdVnxHy4w */

Chain PVEFW-FORWARD (1 references)
 pkts bytes target     prot opt in     out     source               destination
    0     0 DROP       all  --  *      *       0.0.0.0/0            0.0.0.0/0            ctstate INVALID
39162   38M ACCEPT     all  --  *      *       0.0.0.0/0            0.0.0.0/0            ctstate RELATED,ESTABLISHED
 1645  515K PVEFW-FWBR-IN  all  --  *      *       0.0.0.0/0            0.0.0.0/0            PHYSDEV match --physdev-in fwln+ --physdev-is-bridged
  140 18416 PVEFW-FWBR-OUT  all  --  *      *       0.0.0.0/0            0.0.0.0/0            PHYSDEV match --physdev-out fwln+ --physdev-is-bridged
 2745  812K            all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* PVESIG:qnNexOcGa+y+jebd4dAUqFSp5nw */

Update 1

Alternatively, iptables-save output.

# Generated by iptables-save v1.8.7 on Thu Jun  8 13:42:23 2023
*raw
:PREROUTING ACCEPT [2543038:2106118137]
:OUTPUT ACCEPT [342788:70396335]
COMMIT
# Completed on Thu Jun  8 13:42:23 2023
# Generated by iptables-save v1.8.7 on Thu Jun  8 13:42:23 2023
*filter
:INPUT ACCEPT [33063:2272680]
:FORWARD ACCEPT [242768:74134125]
:OUTPUT ACCEPT [5374:325800]
:PVEFW-Drop - [0:0]
:PVEFW-DropBroadcast - [0:0]
:PVEFW-FORWARD - [0:0]
:PVEFW-FWBR-IN - [0:0]
:PVEFW-FWBR-OUT - [0:0]
:PVEFW-HOST-IN - [0:0]
:PVEFW-HOST-OUT - [0:0]
:PVEFW-INPUT - [0:0]
:PVEFW-OUTPUT - [0:0]
:PVEFW-Reject - [0:0]
:PVEFW-SET-ACCEPT-MARK - [0:0]
:PVEFW-logflags - [0:0]
:PVEFW-reject - [0:0]
:PVEFW-smurflog - [0:0]
:PVEFW-smurfs - [0:0]
:PVEFW-tcpflags - [0:0]
-A INPUT -j PVEFW-INPUT
-A FORWARD -j PVEFW-FORWARD
-A OUTPUT -j PVEFW-OUTPUT
-A PVEFW-Drop -j PVEFW-DropBroadcast
-A PVEFW-Drop -p icmp -m icmp --icmp-type 3/4 -j ACCEPT
-A PVEFW-Drop -p icmp -m icmp --icmp-type 11 -j ACCEPT
-A PVEFW-Drop -m conntrack --ctstate INVALID -j DROP
-A PVEFW-Drop -p udp -m multiport --dports 135,445 -j DROP
-A PVEFW-Drop -p udp -m udp --dport 137:139 -j DROP
-A PVEFW-Drop -p udp -m udp --sport 137 --dport 1024:65535 -j DROP
-A PVEFW-Drop -p tcp -m multiport --dports 135,139,445 -j DROP
-A PVEFW-Drop -p udp -m udp --dport 1900 -j DROP
-A PVEFW-Drop -p tcp -m tcp ! --tcp-flags FIN,SYN,RST,ACK SYN -j DROP
-A PVEFW-Drop -p udp -m udp --sport 53 -j DROP
-A PVEFW-Drop -m comment --comment "PVESIG:83WlR/a4wLbmURFqMQT3uJSgIG8"
-A PVEFW-DropBroadcast -m addrtype --dst-type BROADCAST -j DROP
-A PVEFW-DropBroadcast -m addrtype --dst-type MULTICAST -j DROP
-A PVEFW-DropBroadcast -m addrtype --dst-type ANYCAST -j DROP
-A PVEFW-DropBroadcast -d 224.0.0.0/4 -j DROP
-A PVEFW-DropBroadcast -m comment --comment "PVESIG:NyjHNAtFbkH7WGLamPpdVnxHy4w"
-A PVEFW-FORWARD -m conntrack --ctstate INVALID -j DROP
-A PVEFW-FORWARD -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A PVEFW-FORWARD -m physdev --physdev-in fwln+ --physdev-is-bridged -j PVEFW-FWBR-IN
-A PVEFW-FORWARD -m physdev --physdev-out fwln+ --physdev-is-bridged -j PVEFW-FWBR-OUT
-A PVEFW-FORWARD -m comment --comment "PVESIG:qnNexOcGa+y+jebd4dAUqFSp5nw"
-A PVEFW-FWBR-IN -m conntrack --ctstate INVALID,NEW -j PVEFW-smurfs
-A PVEFW-FWBR-IN -m comment --comment "PVESIG:AExd1AckobhMIrEf5xVy0JhkW6g"
-A PVEFW-FWBR-OUT -m comment --comment "PVESIG:3Ocbg4kF01au/LYAeIPRKLGUbOE"
-A PVEFW-HOST-IN -i lo -j ACCEPT
-A PVEFW-HOST-IN -m conntrack --ctstate INVALID -j DROP
-A PVEFW-HOST-IN -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A PVEFW-HOST-IN -m conntrack --ctstate INVALID,NEW -j PVEFW-smurfs
-A PVEFW-HOST-IN -p igmp -j RETURN
-A PVEFW-HOST-IN -p udp -m udp --dport 53 -j RETURN
-A PVEFW-HOST-IN -p tcp -m tcp --dport 53 -j RETURN
-A PVEFW-HOST-IN -p tcp -m set --match-set PVEFW-0-management-v4 src -m tcp --dport 443 -j RETURN
-A PVEFW-HOST-IN -p tcp -m set --match-set PVEFW-0-management-v4 src -m tcp --dport 80 -j RETURN
-A PVEFW-HOST-IN -p tcp -m set --match-set PVEFW-0-management-v4 src -m tcp --dport 8006 -j RETURN
-A PVEFW-HOST-IN -p tcp -m set --match-set PVEFW-0-management-v4 src -m tcp --dport 5900:5999 -j RETURN
-A PVEFW-HOST-IN -p tcp -m set --match-set PVEFW-0-management-v4 src -m tcp --dport 3128 -j RETURN
-A PVEFW-HOST-IN -p tcp -m set --match-set PVEFW-0-management-v4 src -m tcp --dport 22 -j RETURN
-A PVEFW-HOST-IN -p tcp -m set --match-set PVEFW-0-management-v4 src -m tcp --dport 60000:60050 -j RETURN
-A PVEFW-HOST-IN -j PVEFW-Drop
-A PVEFW-HOST-IN -j DROP
-A PVEFW-HOST-IN -m comment --comment "PVESIG:Rej56Owxz0NP3pG3ek441Blmvh0"
-A PVEFW-HOST-OUT -o lo -j ACCEPT
-A PVEFW-HOST-OUT -m conntrack --ctstate INVALID -j DROP
-A PVEFW-HOST-OUT -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A PVEFW-HOST-OUT -p igmp -j RETURN
-A PVEFW-HOST-OUT -d 192.168.1.0/24 -p tcp -m tcp --dport 8006 -j RETURN
-A PVEFW-HOST-OUT -d 192.168.1.0/24 -p tcp -m tcp --dport 22 -j RETURN
-A PVEFW-HOST-OUT -d 192.168.1.0/24 -p tcp -m tcp --dport 5900:5999 -j RETURN
-A PVEFW-HOST-OUT -d 192.168.1.0/24 -p tcp -m tcp --dport 3128 -j RETURN
-A PVEFW-HOST-OUT -j RETURN
-A PVEFW-HOST-OUT -m comment --comment "PVESIG:CWlvhPG9j+jUt46LpfMTQuSJT7A"
-A PVEFW-INPUT -j PVEFW-HOST-IN
-A PVEFW-INPUT -m comment --comment "PVESIG:+5iMmLaxKXynOB/+5xibfx7WhFk"
-A PVEFW-OUTPUT -j PVEFW-HOST-OUT
-A PVEFW-OUTPUT -m comment --comment "PVESIG:LjHoZeSSiWAG3+2ZAyL/xuEehd0"
-A PVEFW-Reject -j PVEFW-DropBroadcast
-A PVEFW-Reject -p icmp -m icmp --icmp-type 3/4 -j ACCEPT
-A PVEFW-Reject -p icmp -m icmp --icmp-type 11 -j ACCEPT
-A PVEFW-Reject -m conntrack --ctstate INVALID -j DROP
-A PVEFW-Reject -p udp -m multiport --dports 135,445 -j PVEFW-reject
-A PVEFW-Reject -p udp -m udp --dport 137:139 -j PVEFW-reject
-A PVEFW-Reject -p udp -m udp --sport 137 --dport 1024:65535 -j PVEFW-reject
-A PVEFW-Reject -p tcp -m multiport --dports 135,139,445 -j PVEFW-reject
-A PVEFW-Reject -p udp -m udp --dport 1900 -j DROP
-A PVEFW-Reject -p tcp -m tcp ! --tcp-flags FIN,SYN,RST,ACK SYN -j DROP
-A PVEFW-Reject -p udp -m udp --sport 53 -j DROP
-A PVEFW-Reject -m comment --comment "PVESIG:h3DyALVslgH5hutETfixGP08w7c"
-A PVEFW-SET-ACCEPT-MARK -j MARK --set-xmark 0x80000000/0x80000000
-A PVEFW-SET-ACCEPT-MARK -m comment --comment "PVESIG:Hg/OIgIwJChBUcWU8Xnjhdd2jUY"
-A PVEFW-logflags -j DROP
-A PVEFW-logflags -m comment --comment "PVESIG:MN4PH1oPZeABMuWr64RrygPfW7A"
-A PVEFW-reject -m addrtype --dst-type BROADCAST -j DROP
-A PVEFW-reject -s 224.0.0.0/4 -j DROP
-A PVEFW-reject -p icmp -j DROP
-A PVEFW-reject -p tcp -j REJECT --reject-with tcp-reset
-A PVEFW-reject -p udp -j REJECT --reject-with icmp-port-unreachable
-A PVEFW-reject -p icmp -j REJECT --reject-with icmp-host-unreachable
-A PVEFW-reject -j REJECT --reject-with icmp-host-prohibited
-A PVEFW-reject -m comment --comment "PVESIG:Jlkrtle1mDdtxDeI9QaDSL++Npc"
-A PVEFW-smurflog -j DROP
-A PVEFW-smurflog -m comment --comment "PVESIG:2gfT1VMkfr0JL6OccRXTGXo+1qk"
-A PVEFW-smurfs -s 0.0.0.0/32 -j RETURN
-A PVEFW-smurfs -m addrtype --src-type BROADCAST -g PVEFW-smurflog
-A PVEFW-smurfs -s 224.0.0.0/4 -g PVEFW-smurflog
-A PVEFW-smurfs -m comment --comment "PVESIG:HssVe5QCBXd5mc9kC88749+7fag"
-A PVEFW-tcpflags -p tcp -m tcp --tcp-flags FIN,SYN,RST,PSH,ACK,URG FIN,PSH,URG -g PVEFW-logflags
-A PVEFW-tcpflags -p tcp -m tcp --tcp-flags FIN,SYN,RST,PSH,ACK,URG NONE -g PVEFW-logflags
-A PVEFW-tcpflags -p tcp -m tcp --tcp-flags SYN,RST SYN,RST -g PVEFW-logflags
-A PVEFW-tcpflags -p tcp -m tcp --tcp-flags FIN,SYN FIN,SYN -g PVEFW-logflags
-A PVEFW-tcpflags -p tcp -m tcp --sport 0 --tcp-flags FIN,SYN,RST,ACK SYN -g PVEFW-logflags
-A PVEFW-tcpflags -m comment --comment "PVESIG:CMFojwNPqllyqD67NeI5m+bP5mo"
COMMIT
# Completed on Thu Jun  8 13:42:23 2023

I'm also attaching the iptables traces (sudo iptables -t raw -A PREROUTING -p tcp --source 192.168.1.20 --sport 80 -j TRACE && sudo iptables -t raw -A OUTPUT -p tcp --destination 192.168.1.20 --dport 80 -j TRACE) for the requests in question (linebreaks added by me).

0 6 - 07/Jun/2023:23:14:36 +1000 TRACE: raw:OUTPUT:policy:2 OUT=vmbr0 SRC=192.168.1.18 DST=192.168.1.20 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=7089 DF PROTO=TCP SPT=51342 DPT=80 SEQ=3153030423 ACK=0 WINDOW=64240 SYN 
0 6 - 07/Jun/2023:23:14:36 +1000 TRACE: filter:OUTPUT:rule:1 OUT=vmbr0 SRC=192.168.1.18 DST=192.168.1.20 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=7089 DF PROTO=TCP SPT=51342 DPT=80 SEQ=3153030423 ACK=0 WINDOW=64240 SYN 
0 6 - 07/Jun/2023:23:14:36 +1000 TRACE: filter:PVEFW-OUTPUT:rule:1 OUT=vmbr0 SRC=192.168.1.18 DST=192.168.1.20 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=7089 DF PROTO=TCP SPT=51342 DPT=80 SEQ=3153030423 ACK=0 WINDOW=64240 SYN 
0 6 - 07/Jun/2023:23:14:36 +1000 TRACE: filter:PVEFW-HOST-OUT:return:9 OUT=vmbr0 SRC=192.168.1.18 DST=192.168.1.20 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=7089 DF PROTO=TCP SPT=51342 DPT=80 SEQ=3153030423 ACK=0 WINDOW=64240 SYN 
0 6 - 07/Jun/2023:23:14:36 +1000 TRACE: filter:PVEFW-OUTPUT:rule:2 OUT=vmbr0 SRC=192.168.1.18 DST=192.168.1.20 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=7089 DF PROTO=TCP SPT=51342 DPT=80 SEQ=3153030423 ACK=0 WINDOW=64240 SYN 
0 6 - 07/Jun/2023:23:14:36 +1000 TRACE: filter:PVEFW-OUTPUT:return:3 OUT=vmbr0 SRC=192.168.1.18 DST=192.168.1.20 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=7089 DF PROTO=TCP SPT=51342 DPT=80 SEQ=3153030423 ACK=0 WINDOW=64240 SYN 
0 6 - 07/Jun/2023:23:14:36 +1000 TRACE: filter:OUTPUT:policy:2 OUT=vmbr0 SRC=192.168.1.18 DST=192.168.1.20 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=7089 DF PROTO=TCP SPT=51342 DPT=80 SEQ=3153030423 ACK=0 WINDOW=64240 SYN 

0 6 - 07/Jun/2023:23:14:36 +1000 TRACE: raw:PREROUTING:policy:2 IN=fwbr106i0 PHYSIN=tap106i0 MAC=<redacted> SRC=192.168.1.20 DST=192.168.1.18 LEN=60 TOS=0x00 PREC=0x00 TTL=63 ID=0 DF PROTO=TCP SPT=80 DPT=51342 SEQ=2204635599 ACK=3153030424 WINDOW=65160 ACK SYN 
0 6 - 07/Jun/2023:23:14:36 +1000 TRACE: filter:FORWARD:rule:1 IN=fwbr106i0 OUT=fwbr106i0 PHYSIN=tap106i0 PHYSOUT=fwln106i0 MAC=<redacted> SRC=192.168.1.20 DST=192.168.1.18 LEN=60 TOS=0x00 PREC=0x00 TTL=63 ID=0 DF PROTO=TCP SPT=80 DPT=51342 SEQ=2204635599 ACK=3153030424 WINDOW=65160 ACK SYN 
0 6 - 07/Jun/2023:23:14:36 +1000 TRACE: filter:PVEFW-FORWARD:rule:2 IN=fwbr106i0 OUT=fwbr106i0 PHYSIN=tap106i0 PHYSOUT=fwln106i0 MAC=<redacted> SRC=192.168.1.20 DST=192.168.1.18 LEN=60 TOS=0x00 PREC=0x00 TTL=63 ID=0 DF PROTO=TCP SPT=80 DPT=51342 SEQ=2204635599 ACK=3153030424 WINDOW=65160 ACK SYN 

0 6 - 07/Jun/2023:23:14:36 +1000 TRACE: raw:PREROUTING:policy:2 IN=vmbr0 PHYSIN=fwpr106p0 MAC=<redacted> SRC=192.168.1.20 DST=192.168.1.18 LEN=60 TOS=0x00 PREC=0x00 TTL=63 ID=0 DF PROTO=TCP SPT=80 DPT=51342 SEQ=2204635599 ACK=3153030424 WINDOW=65160 ACK SYN 
0 6 - 07/Jun/2023:23:14:36 +1000 TRACE: filter:FORWARD:rule:1 IN=vmbr0 OUT=vmbr0 PHYSIN=fwpr106p0 PHYSOUT=enp2s0 MAC=<redacted> SRC=192.168.1.20 DST=192.168.1.18 LEN=60 TOS=0x00 PREC=0x00 TTL=63 ID=0 DF PROTO=TCP SPT=80 DPT=51342 SEQ=2204635599 ACK=3153030424 WINDOW=65160 ACK SYN 
0 6 - 07/Jun/2023:23:14:36 +1000 TRACE: filter:PVEFW-FORWARD:rule:2 IN=vmbr0 OUT=vmbr0 PHYSIN=fwpr106p0 PHYSOUT=enp2s0 MAC=<redacted> SRC=192.168.1.20 DST=192.168.1.18 LEN=60 TOS=0x00 PREC=0x00 TTL=63 ID=0 DF PROTO=TCP SPT=80 DPT=51342 SEQ=2204635599 ACK=3153030424 WINDOW=65160 ACK SYN 

0 6 - 07/Jun/2023:23:14:37 +1000 TRACE: raw:PREROUTING:policy:2 IN=fwbr106i0 PHYSIN=tap106i0 MAC=<redacted> SRC=192.168.1.20 DST=192.168.1.18 LEN=60 TOS=0x00 PREC=0x00 TTL=63 ID=0 DF PROTO=TCP SPT=80 DPT=51342 SEQ=2204635599 ACK=3153030424 WINDOW=65160 ACK SYN 
0 6 - 07/Jun/2023:23:14:37 +1000 TRACE: filter:FORWARD:rule:1 IN=fwbr106i0 OUT=fwbr106i0 PHYSIN=tap106i0 PHYSOUT=fwln106i0 MAC=<redacted> SRC=192.168.1.20 DST=192.168.1.18 LEN=60 TOS=0x00 PREC=0x00 TTL=63 ID=0 DF PROTO=TCP SPT=80 DPT=51342 SEQ=2204635599 ACK=3153030424 WINDOW=65160 ACK SYN 
0 6 - 07/Jun/2023:23:14:37 +1000 TRACE: filter:PVEFW-FORWARD:rule:2 IN=fwbr106i0 OUT=fwbr106i0 PHYSIN=tap106i0 PHYSOUT=fwln106i0 MAC=<redacted> SRC=192.168.1.20 DST=192.168.1.18 LEN=60 TOS=0x00 PREC=0x00 TTL=63 ID=0 DF PROTO=TCP SPT=80 DPT=51342 SEQ=2204635599 ACK=3153030424 WINDOW=65160 ACK SYN 

0 6 - 07/Jun/2023:23:14:37 +1000 TRACE: raw:PREROUTING:policy:2 IN=vmbr0 PHYSIN=fwpr106p0 MAC=<redacted> SRC=192.168.1.20 DST=192.168.1.18 LEN=60 TOS=0x00 PREC=0x00 TTL=63 ID=0 DF PROTO=TCP SPT=80 DPT=51342 SEQ=2204635599 ACK=3153030424 WINDOW=65160 ACK SYN 
0 6 - 07/Jun/2023:23:14:37 +1000 TRACE: filter:FORWARD:rule:1 IN=vmbr0 OUT=vmbr0 PHYSIN=fwpr106p0 PHYSOUT=enp2s0 MAC=<redacted> SRC=192.168.1.20 DST=192.168.1.18 LEN=60 TOS=0x00 PREC=0x00 TTL=63 ID=0 DF PROTO=TCP SPT=80 DPT=51342 SEQ=2204635599 ACK=3153030424 WINDOW=65160 ACK SYN 
0 6 - 07/Jun/2023:23:14:37 +1000 TRACE: filter:PVEFW-FORWARD:rule:2 IN=vmbr0 OUT=vmbr0 PHYSIN=fwpr106p0 PHYSOUT=enp2s0 MAC=<redacted> SRC=192.168.1.20 DST=192.168.1.18 LEN=60 TOS=0x00 PREC=0x00 TTL=63 ID=0 DF PROTO=TCP SPT=80 DPT=51342 SEQ=2204635599 ACK=3153030424 WINDOW=65160 ACK SYN

enp2s0 is the physical NIC of the host. vmbr0 is configured like below.

auto vmbr0
iface vmbr0 inet static
    address 192.168.1.18/24
    gateway 192.168.1.1
    bridge-ports enp2s0
    bridge-stp off
    bridge-fd 0

Update 2

Bridge information.

5: vmbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000
    link/ether <redacted> brd ff:ff:ff:ff:ff:ff promiscuity 0 minmtu 68 maxmtu 65535
    bridge forward_delay 0 hello_time 200 max_age 2000 ageing_time 30000 stp_state 0 priority 32768 vlan_filtering 0 vlan_protocol 802.1Q bridge_id 8000.<redacted> designated_root 8000.<redacted> root_port 0 root_path_cost 0 topology_change 0 topology_change_detected 0 hello_timer    0.00 tcn_timer    0.00 topology_change_timer    0.00 gc_timer   98.68 vlan_default_pvid 1 vlan_stats_enabled 0 vlan_stats_per_port 0 group_fwd_mask 0 group_address <redacted> mcast_snooping 1 mcast_router 1 mcast_query_use_ifaddr 0 mcast_querier 0 mcast_hash_elasticity 16 mcast_hash_max 4096 mcast_last_member_count 2 mcast_startup_query_count 2 mcast_last_member_interval 100 mcast_membership_interval 26000 mcast_querier_interval 25500 mcast_query_interval 12500 mcast_query_response_interval 1000 mcast_startup_query_interval 3124 mcast_stats_enabled 0 mcast_igmp_version 2 mcast_mld_version 1 nf_call_iptables 0 nf_call_ip6tables 0 nf_call_arptables 0 addrgenmode eui64 numtxqueues 1 numrxqueues 1 gso_max_size 64000 gso_max_segs 64

Some stuff I have tried:

  1. The Proxmox firewall of the guest has always been off. I tried turning off the firewall of the host, but this doesn't seem to fix the problem.
  2. The problem is recent. The setup has always been the same, and was working before. The only things changed are regular OS updates through apt-get. I tried rolling back the packages for Proxmox firewall and the Linux kernel to the versions prior to apt-get update. But it didn't help either.
193
1 + 5
tcp
Nikita Kipriyanov avatar
za flag
Nikita Kipriyanov
What system is 192.168.1.18? Show a complete current state of its firewall (e.g. output of `iptables-save` if it's Linux and iptables is used).
1
Reply
Ivan avatar
in flag
Ivan
@NikitaKipriyanov thanks again for the answer. I've updated the question with more details.
0
Reply
Nikita Kipriyanov avatar
za flag
Nikita Kipriyanov
Fine then, you also have a bridge? Also, I am not very proficient with Proxmox's firewall (always disabled it), but I can try to read the firewall setup. Notice that I specifically asked for `iptables-save` which is easier to read (for me), **not** `iptables -L` which reportedly hides important bits sometimes. Another issue might be iptables filtering bridged packets; check `ip -d link show dev vmbr0`, especially `nf_call_iptables`.
1
Reply
Ivan avatar
in flag
Ivan
@NikitaKipriyanov thank you for your patience, I understand the setup is a bit complicated. Regarding Proxmox, I have tried turning the firewall off through the web UI but it doesn't help. I will update the question with `iptables-save`. Yes, the bridge does have `nf_call_iptables`.
0
Reply
Nikita Kipriyanov avatar
za flag
Nikita Kipriyanov
I see nothing in your firewall that can block. Can you try to disable the bridge packet filtering for a moment? Run (as root) `echo 0 > /sys/class/net/vmbr0/bridge/nf_call_iptables` and see if this helps. You can always reverse this by echoing 1. Also, notice how it captures `fwln+`, these are created when you enable firewall for VMs or CTs. Do you have any interfaces with such name?
0
Reply
Score:3
Nikita Kipriyanov avatar Server
za flag
Nikita Kipriyanov

That Wireshark sees the return "ACK, SYN" doesn't mean OS TCP stack had processed it. It only means the packet reached the NIC. Wireshark taps "on the outer side" of the firewall.

Check the firewall on 192.168.1.18. Most likely it is blocking the return packet so the stack doesn't see it even though hardware received it (packet is dropped before TCP had a chance to see it).

+ 2
Ivan avatar
in flag
Ivan
Thanks for the prompt reply. I have added TRACE to iptables raw PREROUTING, and can see the incoming requests ended up in a rule that ACCEPTs RELATED and ESTABLISHED.
0
Reply
Ivan avatar
in flag
Ivan
What would you suggest as the next step in terms of iptables debugging? I couldn’t just drop all rules on that machine, since the VM network relies on the host firewall to function.
0
Reply
ธงชาติไทย
Elon Musk
I sit in a Tesla and translated this thread with Ai:

mangohost

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.