Considering the strongswan wiki, this appears to be a standard problem, but I can't get it working quite right.
Network layout

The local site (client and gateway) is under my control, the remote site (remote gateway and remote server) is not. The IPSec tunnel is a split tunnel such that only requests to the 10.10.0.0/16 subnet are sent through the IPSec tunnel.
Goal
I would like the client to communicate with the remote server, e.g. create a ssh or a smb connection.
What I already did
I've established an IPSec tunnel between the gateway and the remote gateway.
I've enabled ip forwarding on the gateway:
sysctl net.ipv4.ip_forward=1
I've created a NAT on the gateway:
iptables -t nat -I POSTROUTING -m policy --pol ipsec --dir out -j ACCEPT
iptables -t nat -A POSTROUTING -j MASQUERADE
On the client I've routed the traffic through the gateway:
ip route del default
ip route add default via 192.168.144.4
# 192.168.144.4 is the gateway
What does work
- The IPSec tunnel is established and stable.
- When logged-in into the
gateway, I can ping the remote gateway and the remote server successfully. I can also ping the client. And I can ping google.com.
- When logged-in into the
client, I can ping google.com and I can ping the gateway. With tcpdump icmp on the gateway I see that the ping google.com from the client is going through the gateway.
What does not work, yet
I cannot ping the remote server from the client by its IP.
client$ ping -c 1 10.10.12.7
PING 10.10.12.7 (10.10.12.7): 56 data bytes
--- 10.10.12.7 ping statistics ---
1 packets transmitted, 0 packets received, 100% packet loss
From the tcpdump on the gateway, it looks like the ping is sent, but not forwarded through the tunnel:
gateway$ tcpdump icmp
13:19:18.122999 IP 192.168.144.7 > 10.10.12.7: ICMP echo request, id 15, seq 0, length 64
13:19:18.123038 IP gateway > 10.10.12.7: ICMP echo request, id 15, seq 0, length 64
13:19:18.127534 IP ac5.nue3.m-online.net > gateway: ICMP net 10.10.12.7 unreachable, length 36
13:19:18.127556 IP ac5.nue3.m-online.net > 192.168.144.7: ICMP net 10.10.12.7 unreachable, length 36
As ac5.nue3.m-online.net is the internet-service provider of the local site, I think the packet is not routed through the IPSec tunnel, but through the internet connection of the gateway, which, of course, can't reach the remote server. (If I make the IPSec tunnel a full tunnel rather than a split tunnel, I get the same result.)
Any help or insight would be really appreciated!
EDIT: ipsec statusall on the gateway
gateway > ipsec statusall
Status of IKE charon daemon (strongSwan 5.8.2, Linux 5.4.0-88-generic, x86_64):
uptime: 7 minutes, since Oct 08 08:18:24 2021
malloc: sbrk 3112960, mmap 0, used 1081456, free 2031504
worker threads: 11 of 16 idle, 5/0/0/0 working, job queue: 0/0/0/0, scheduled: 4
loaded plugins: charon test-vectors ldap pkcs11 tpm aesni aes rc2 sha2 sha1 md5 mgf1 rdrand random nonce x509 revocation constraints pubkey pkcs1 pkcs7 pkcs8 pkcs12 pgp dnskey sshkey pem openssl gcrypt af-alg fips-prf gmp curve25519 agent chapoly xcbc cmac hmac ctr ccm gcm ntru drbg curl attr kernel-netlink resolve socket-default connmark farp stroke updown eap-identity eap-aka eap-md5 eap-gtc eap-mschapv2 eap-dynamic eap-radius eap-tls eap-ttls eap-peap eap-tnc xauth-generic xauth-eap xauth-pam tnc-tnccs dhcp lookip error-notify certexpire led addrblock unity counters
Listening IP addresses:
192.168.144.4
Connections:
example-ipsec: %any...vpn1.example.com IKEv2, dpddelay=300s
example-ipsec: local: [example@example.com] uses pre-shared key authentication
example-ipsec: remote: [example@example.com] uses pre-shared key authentication
example-ipsec: child: dynamic === 0.0.0.0/0 TUNNEL, dpdaction=clear
Security Associations (1 up, 0 connecting):
example-ipsec[1]: ESTABLISHED 7 minutes ago, 192.168.144.4[example@example.com]...<public-ip-of-the-remote-gateway>[example@example.com]
example-ipsec[1]: I: 9d7c74f670bbda86_i* c12b3b4a236b7018_r, pre-shared key reauthentication in 2 hours
example-ipsec[1]: IKE proposal: AES_CBC_256/HMAC_SHA2_256_128/PRF_HMAC_SHA2_256/MODP_2048
example-ipsec{1}: INSTALLED, TUNNEL, reqid 1, ESP in UDP SPIs: cf66ad72_i af3c9348_o
example-ipsec{1}: AES_CBC_256/HMAC_SHA2_256_128, 442 bytes_i (4 pkts, 434s ago), 485 bytes_o (6 pkts, 433s ago), rekeying in 38 minutes
example-ipsec{1}: 10.10.102.235/32 === 0.0.0.0/0
This is the output of ipsec statusall on the gateway after unsuccessfully sending the ping from client to the remote server. The ping from the gateway does not change the "bytes" in the output. The "bytes" in the output correspond to a ping I've sent from the gateway to the remote server.
EDIT: /etc/ipsec.conf on gateway:
# /etc/ipsec.conf
conn example-ipsec
left = %defaultroute
leftsourceip = %config
leftid = "example@example.com"
right = vpn1.example.com
rightid = "example@example.com"
rightsubnet = 0.0.0.0/0
leftfirewall = yes
installpolicy = yes
keyexchange = ikev2
type = tunnel
auto = start
leftauth = psk
rightauth = psk
dpdaction = clear
dpddelay = 300s