ipsec/strongswan - tunnel is up, traffic is sent and received but replies are ignored

I need some help, I set up a strongswan IPsec tunnel with ESP and IKEv2, the tunnel is UP and remote sees packets coming and answers them, but my server is ignoring? answers.

The tunnel is between my Debian 11 server holding both the IP 10.0.1.1 and the public IP 100.100.100.100 (no NAT on this side) and the remote is a Checkpoint VPN endpoint.

On my side, I configured a swanctl config like this (All IPs have been changed):

cat /etc/swanctl.d/conf.d/test.conf
connections {
  test {
    local_addrs = 100.100.100.100
    local {
      id = 100.100.100.100
      auth = psk
    }
    remote_addrs = 200.200.200.200
    remote {
      id = 200.200.200.200
    }
    children {
      test_child {
        local_ts = 10.0.1.0/24
        remote_ts = 10.0.2.0/24
        esp_proposals = aes256-sha256-modp2048
        dpd_action = restart
        start_action = start
        ike {
          proposals = aes256-sha256-modp2048
        }
      }
    }
    version = 2
    dpd_delay = 300s
  }
}

secrets {
  ike-psk {
    secret = "MY_PSK_KEY"
  }
}

The remote is configured by another person on a Checkpoint VPN endpoint.

I can see that the tunnel is up with but it shows 0 packets in the "in" part, only out packets

swanctl --list-sas
test: #1, ESTABLISHED, IKEv2, c7742fc57b157ccd_i* b75ae63221a8c6d7_r
  local  '100.100.100.100' @ 100.100.100.100[4500]
  remote '200.200.200.200' @ 200.200.200.200[4500]
  AES_CBC-256/HMAC_SHA2_256_128/PRF_HMAC_SHA2_256/MODP_2048
  established 2326s ago, rekeying in 11899s
  test_child: #1, reqid 1, INSTALLED, TUNNEL, ESP:AES_CBC-256/HMAC_SHA2_256_128
    installed 2326s ago, rekeying in 916s, expires in 1634s
    in  cb90145c,      0 bytes,     0 packets
    out 883097a1,  15600 bytes,   260 packets,     2s ago
    local  10.0.1.0/24
    remote 10.0.2.0/24

On the remote side, they confirm they are seeing trafic coming in and confirm they are answering (I test with nc 10.0.2.1 4444). To try to understand what's going on, I captured the traffic with tcpdump and I was able to decrypt everything with Wireshark. In Wireshark, the deencapsulated packets shows my "nc 10.0.2.1 4444" with a TCP SYN from 10.0.1.1 to 10.0.2.1, remote answers with a TCP SYN ACK from 10.0.2.1 to 10.0.1.1 but no TCP ACK is sent from my side after. It just starts again with TCP Restransmission TCP SYN from 10.0.1.1 to 10.0.2.1 then remote answers with a TCP SYN ACK from 10.0.2.1 to 10.0.1.1, ....

With the firewall (UFW) disabled, a ICMP Destination unreachable (Port unreachable) is sent after the received SYN-ACK from my server public IP 100.100.100.100 (not from my local IP 10.0.1.1 weirdly (again I see this in the deencapsulated and decrypted traffic))

In my Firewall (UFW), the only rules are

ufw allow from 200.200.200.200 port 500,4500 proto udp to any
ufw allow from 10.0.2.0/24 to any

I have not configured any NAT rules as I think I don't need to as it is my server holding the IP 10.0.1.1 in the local_ts subnet and the traffic sent is correctly using 10.0.1.1 as the from address.

So my question is why is the SYN-ACK ignored, why swanctl --list-sas shows 0 packets in when I can see them in Wireshark ?

Some additional information

ip xfrm policy
src 10.0.1.0/24 dst 10.0.2.0/24
        dir out priority 371839 ptype main
        tmpl src 100.100.100.100 dst 200.200.200.200
                proto esp spi 0x649d7f4d reqid 1 mode tunnel
src 10.0.2.0/24 dst 10.0.1.0/24
        dir fwd priority 371839 ptype main
        tmpl src 200.200.200.200 dst 100.100.100.100
                proto esp reqid 1 mode tunnel
src 10.0.2.0/24 dst 10.0.1.0/24
        dir in priority 371839 ptype main
        tmpl src 200.200.200.200 dst 100.100.100.100
                proto esp reqid 1 mode tunnel


ip route show table 220
10.0.2.0/24 via 100.100.100.255 dev enp33s0f0 proto static src 10.0.1.1


ip a
2: enp33s0f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether ab:cd:ef:gh:ij:kl brd ff:ff:ff:ff:ff:ff
    inet 100.100.100.100/24 brd 100.100.100.255 scope global dynamic enp33s0f0
       valid_lft 81651sec preferred_lft 81651sec
    inet 10.0.1.1/24 brd 10.0.1.255 scope global enp33s0f0:0
       valid_lft forever preferred_lft forever