Container bridge networking with subnet on Hetzner and CNI

I am trying to create a private network of VMs and containers on Hetzner Cloud. I've tested this setup on my local home network and all works fine.

The plan is to have a private network for the VMs (10.0.0.0/8 just for testing). For my test I am using 10.10.0.1 and 10.10.0.2 as VMs. And each VM, will have a CNI bridge network, (eg: 172.20.0.0/16, for container). I want to make the bridge networks accesible by any VM on the 10.0.0.0/8 network, with static routes.

On Hetzner, I've configured a static route 172.20.0.0/16 to 10.10.0.1. On 10.10.0.1 I have a CNI bridge network for Podman, configured on the same range 172.20.0.0/16.

Any container that gets placed on that network, has no trouble pinging or reaching out to: local, other containers, host or internet, and the host (10.10.0.1) has no trouble reaching the containers (172.20.0.X).

The problem is when I want to ping the container from 10.10.0.2. I've monitored traffic with tcpdump and iftop, and the Hetzner route seems to work just fine, as connections reach the VM on 10.10.0.1 (ens10). Which makes me wonder if it's a routing issue between the ens10 and podman-vlan bridge interfaces?

Here are the routes from 10.10.0.1

default via 172.31.1.1 dev eth0 proto dhcp src X.Y.Z.W metric 100 
10.0.0.0/8 via 10.0.0.1 dev ens10 
10.0.0.1 dev ens10 scope link 
172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1 
172.20.0.0/16 dev podman-vlan proto kernel scope link src 172.20.0.1 
172.31.1.1 dev eth0 proto dhcp scope link src X.Y.Z.W metric 100 

On the 10.10.0.2 VM, I've only done a ip r add 172.20.0.0/16 via 10.0.0.1 (which seems to work as).

My expectation was to get 10.10.0.2 -> 10.0.0.1 -> 10.10.0.1 -> 172.20.0.1 -> 172.20.0.X. Instead everything seems to get lost at 10.10.0.1, including if I try ping -I ens10 172.20.0.X

This is the CNI config:

{
  "cniVersion": "0.4.0",
  "name": "podman",
  "plugins": [
    {
      "type": "bridge",
      "bridge": "podman-vlan",
      "isGateway": true,
      "ipMasq": true,
      "promiscMode": true,
      "ipam": {
        "type": "host-local",
        "routes": [{ "dst": "0.0.0.0/0" }],
        "ranges": [
          [
            {
              "subnet": "172.20.0.0/16",
              "gateway": "172.20.0.1"
            }
          ]
        ]
      }
    },
    {
      "type": "portmap",
      "capabilities": {
        "portMappings": true
      }
    },
    {
      "type": "firewall"
    },
    {
      "type": "tuning"
    }
  ]
}

Thanks in advance.