Join Log in
Score:0
Younes avatar Server

How to investigate not received TCP packets sent from VPN on the same LAN?

it flag
Younes

I'm setting up a VLAN on the cloud where many servers will connect to a remote host via VPN. The setup is as follows:


           Their Host d.d.d.72
                     |
                     |
                     |
       Their VPN Public IP b.b.b.116
                     |
                     |
                  Internet
                     |
                     |
        Our VPN Public IP a.a.a.101
                     |
         Our VPN Local IP s.s.s.31
                     |
  Our VLAN ----------+-----------+-------------------+--------...--------+--
                                 |                   |                   |
                             s.s.s.111           s.s.s.112    ...    s.s.s.nnn
                                 |                   |                   |
                                UAT1                UAT2      ...       UATn

strongswan is used by Our VPN and the target is to allow UAT1 to UATn hosts within Our VLAN to connect to Their Host using a raw TCP socket.

SYN packets are sent from UAT2 and reach Their Host which replies with SYN ACK. SYN ACK are received by Our VPN then forwarded to UAT2. THE ISSUE IS that SYN ACK are not received by UAT2!

Here is the tcpdump logs on UAT2:

root@uat2:~# tcpdump -ttnnvvS -i any host d.d.d.72
tcpdump: data link type LINUX_SLL2
tcpdump: listening on any, link-type LINUX_SLL2 (Linux cooked v2), snapshot length 262144 bytes
1687385350.426771 ens7  Out IP (tos 0x10, ttl 64, id 38370, offset 0, flags [DF], proto TCP (6), length 60)
    s.s.s.112.54838 > d.d.d.72.9990: Flags [S], cksum 0x324d (incorrect -> 0x3ba1), seq 4126082045, win 62720, options [mss 8960,sackOK,TS val 4002865714 ecr 0,nop,wscale 7], length 0
1687385351.445963 ens7  Out IP (tos 0x10, ttl 64, id 38371, offset 0, flags [DF], proto TCP (6), length 60)
    s.s.s.112.54838 > d.d.d.72.9990: Flags [S], cksum 0x324d (incorrect -> 0x37a6), seq 4126082045, win 62720, options [mss 8960,sackOK,TS val 4002866733 ecr 0,nop,wscale 7], length 0
1687385353.461982 ens7  Out IP (tos 0x10, ttl 64, id 38372, offset 0, flags [DF], proto TCP (6), length 60)
    s.s.s.112.54838 > d.d.d.72.9990: Flags [S], cksum 0x324d (incorrect -> 0x2fc6), seq 4126082045, win 62720, options [mss 8960,sackOK,TS val 4002868749 ecr 0,nop,wscale 7], length 0

Here is the tcpdump logs on Our VPN:

root@vpn1:~# tcpdump -ttnnvvS -i any host d.d.d.72
tcpdump: data link type LINUX_SLL2
tcpdump: listening on any, link-type LINUX_SLL2 (Linux cooked v2), snapshot length 262144 bytes
1687385350.426647 ens4  In  IP (tos 0x10, ttl 64, id 38370, offset 0, flags [DF], proto TCP (6), length 60)
    s.s.s.112.54838 > d.d.d.72.9990: Flags [S], cksum 0x3ba1 (correct), seq 4126082045, win 62720, options [mss 8960,sackOK,TS val 4002865714 ecr 0,nop,wscale 7], length 0
1687385350.468411 ens3  In  IP (tos 0x0, ttl 57, id 11319, offset 0, flags [DF], proto TCP (6), length 60)
    d.d.d.72.9990 > s.s.s.112.54838: Flags [S.], cksum 0x94f0 (correct), seq 1695227873, ack 4126082046, win 65535, options [mss 1460,nop,wscale 2,nop,nop,TS val 1699598776 ecr 4002865714], length 0
1687385350.469803 ens4  Out IP (tos 0x0, ttl 56, id 11319, offset 0, flags [DF], proto TCP (6), length 60)
    d.d.d.72.9990 > s.s.s.112.54838: Flags [S.], cksum 0x94f0 (correct), seq 1695227873, ack 4126082046, win 65535, options [mss 1460,nop,wscale 2,nop,nop,TS val 1699598776 ecr 4002865714], length 0
1687385351.445122 ens4  In  IP (tos 0x10, ttl 64, id 38371, offset 0, flags [DF], proto TCP (6), length 60)
    s.s.s.112.54838 > d.d.d.72.9990: Flags [S], cksum 0x37a6 (correct), seq 4126082045, win 62720, options [mss 8960,sackOK,TS val 4002866733 ecr 0,nop,wscale 7], length 0
1687385351.484746 ens3  In  IP (tos 0x0, ttl 57, id 11635, offset 0, flags [DF], proto TCP (6), length 60)
    d.d.d.72.9990 > s.s.s.112.54838: Flags [S.], cksum 0x90f3 (correct), seq 1695227873, ack 4126082046, win 65535, options [mss 1460,nop,wscale 2,nop,nop,TS val 1699598778 ecr 4002866733], length 0
1687385351.484783 ens4  Out IP (tos 0x0, ttl 56, id 11635, offset 0, flags [DF], proto TCP (6), length 60)
    d.d.d.72.9990 > s.s.s.112.54838: Flags [S.], cksum 0x90f3 (correct), seq 1695227873, ack 4126082046, win 65535, options [mss 1460,nop,wscale 2,nop,nop,TS val 1699598778 ecr 4002866733], length 0
1687385353.461170 ens4  In  IP (tos 0x10, ttl 64, id 38372, offset 0, flags [DF], proto TCP (6), length 60)
    s.s.s.112.54838 > d.d.d.72.9990: Flags [S], cksum 0x2fc6 (correct), seq 4126082045, win 62720, options [mss 8960,sackOK,TS val 4002868749 ecr 0,nop,wscale 7], length 0
1687385354.421463 ens3  In  IP (tos 0x0, ttl 57, id 13219, offset 0, flags [DF], proto TCP (6), length 60)
    d.d.d.72.9990 > s.s.s.112.54838: Flags [S.], cksum 0x90ee (correct), seq 1695227873, ack 4126082046, win 65535, options [mss 1460,nop,wscale 2,nop,nop,TS val 1699598783 ecr 4002866733], length 0
1687385354.421495 ens4  Out IP (tos 0x0, ttl 56, id 13219, offset 0, flags [DF], proto TCP (6), length 60)
    d.d.d.72.9990 > s.s.s.112.54838: Flags [S.], cksum 0x90ee (correct), seq 1695227873, ack 4126082046, win 65535, options [mss 1460,nop,wscale 2,nop,nop,TS val 1699598783 ecr 4002866733], length 0
1687385359.466543 ens3  In  IP (tos 0x10, ttl 253, id 37551, offset 0, flags [DF], proto TCP (6), length 40)
    d.d.d.72.9990 > s.s.s.112.54838: Flags [R.], cksum 0x019a (correct), seq 1695227874, ack 4126082046, win 0, length 0
1687385359.466573 ens4  Out IP (tos 0x10, ttl 252, id 37551, offset 0, flags [DF], proto TCP (6), length 40)
    d.d.d.72.9990 > s.s.s.112.54838: Flags [R.], cksum 0x019a (correct), seq 1695227874, ack 4126082046, win 0, length 0

As you can see:

  • UAT2 has sent SYN with id 38370
  • That packet has been received by Our VPN which has forward it to Their Host
  • Their Host has replied with SYN ACK with id 11319
  • That packet has been received by Our VPN which has forward it to UAT2
  • THE ISSUE IS: UAT2 didn't receive SYN ACK

Any idea how to investigate this issue?

During the above test:

  • Our VPN, UAT1 to UATn are all Debian 11 VMs on the cloud

  • iptables on UAT2 was empty:

    root@uat2:~# iptables -L -v --line-numbers
Chain INPUT (policy ACCEPT 65 packets, 26451 bytes)
num   pkts bytes target     prot opt in     out     source               destination         

Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
num   pkts bytes target     prot opt in     out     source               destination         

Chain OUTPUT (policy ACCEPT 64 packets, 26539 bytes)
num   pkts bytes target     prot opt in     out     source               destination         
root@uat2:~# telnet d.d.d.72 9990
Trying d.d.d.72...
^C

  • A static route was defined within UAT2 to reach Their Host

  • For Our VPN, ip_forward is activated:

    net.ipv4.ip_forward = 1

  • The ipsec tunnel is well established:

    root@vpn1:~# ipsec status
Security Associations (1 up, 0 connecting):
our_conn[2]: ESTABLISHED 24 minutes ago, a.a.a.101[a.a.a.101]...b.b.b.116[b.b.b.116]
our_conn{2}:  INSTALLED, TUNNEL, reqid 1, ESP SPIs: c38c258f_i c727e8a0_o
our_conn{2}:   s.s.s.0/24 === d.d.d.72/32
root@vpn1:~#

  • Our VPN pings successfully UAT2

111
1 + 7
tcp
cn flag
ecdsa
What about the firewall rules on `Our VPN`? (Please use `iptables-save` to dump them.) You also mentioned that the hosts are "on the cloud". Some of these cloud services might apply filtering/firewall rules that have to be modified/disabled (e.g. AVPC's source/destination check).
1
Reply
Younes avatar
it flag
Younes
@ecdsa: I have edited the question to add firewall rules on `Our VPN` node. For the cloud, I have another setup much elaborated (multiple zones separated by iptables-based firewalls) than this one with the same Cloud Provider and it is working. The main difference is the distro (Centos7 instead of Debian 11) and region (two different regions within the same data-center).
0
Reply
A.B avatar
cl flag
A.B
If you're using compression, there's a 3rd iteration inside the Linux routing stack that uses the IPIP protocol for small packets. So in addition to ESP can you check enabling IPIP in the firewall? ref: https://wiki.strongswan.org/projects/strongswan/wiki/IPComp/#Linux . And indeed `iptables-save` is more readable that iptables -L
1
Reply
A.B avatar
cl flag
A.B
Bah at the same time never mind: policy drop with 0 dropped packets and no other drop visible. So I don't see where it would be dropped anyway. Maybe (again!) `iptables-save` would tell more?
1
Reply
Younes avatar
it flag
Younes
@A.B: I have added the output of `iptables-save` on `Our VPN`. And I don't use compression as you can see it in `ipsec.conf`.
0
Reply
Younes avatar
it flag
Younes
@ecdsa: sorry for the misunderstanding. I have added the output of `iptables-save -c`.
0
Reply
Younes avatar
it flag
Younes
@ecdsa: you are right. Packets are dropped by the cloud service.
0
Reply
Score:0
Younes avatar Server
it flag
Younes

As suggested by @ecdsa, the packets are dropped by the infra (cloud service).

To prove it, I have added a NAT rule to change the source address based on destination port. When I telnet from My VPN to, say, UAT2 using the NATed port, packets are dropped. When I telnet using any other port, packets are not dropped.

The solution I come up with is to implement port forwarding at the level of Our VPN. As such, UATx machines will communicate through Our VLAN with Our VPN and this later will forward the traffic to d.d.d.72 machine.

Note: with this solution, the static route on UATx machines are no more required.

+ 0
ธงชาติไทย
Elon Musk
I sit in a Tesla and translated this thread with Ai:

mangohost

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.