Unable to use fwmark on Debian 11 (bulleyes) to change routing behavior

I have a recipe I already use on many cases, but this time doesn't works on Debian 11 (kernel 5.10.0-10-amd64)

my setup is basically an internal interface eth0 for a RFC1918 LAN, and two external interfaces connected to some ISP's Box:

eth1 for ISP1 as default router at 10.0.0.254 with public IP 1.2.3.4 (figuratively)

eth2 for ISP2 has a router at 10.0.3.254 with public 2.3.4.5

I have different possible route. I want to control which route my packet takes, so I create some rule and fwmark. First I append 2<tab>secondrouter in /etc/iproute2/rt_tables

ip rule add fwmark 0x3 lookup secondrouter
ip route add default via 10.0.3.254 table secondrouter

everythings is fine regarding ip route list table secondrouter and ip rule list

at this time I am able to do:

curl -4 ifconfig.me
1.2.3.4 #<- public ip address of my default route

Then I do

iptables-legacy -t mangle -A OUTPUT -d 34.117.59.81 -j MARK --set-mark 0x3 

now if I do

curl -4 ifconfig.me
<timeout>

Where I expected 2.3.4.5 as public IP. So clearly the marked packet do not take the route from the ip route table, worse, it timeouts.

If I do this exactly the same way on older Debian, its works perfectly.

NB:if I do a

ip route add 34.117.59.81 via <second router IP>

my curl test works perfectly as expected

curl -4 ifconfig.me
2.3.4.5 #<- Pub ip address of my second router

My problem occurs when using iptables or iptables-legacy to mark packets to route. BTW I have plenty of iptables rules that works fine, so it do not looks like an iptables issue.

iptables-legacy-save 
# Generated by iptables-save v1.8.7 on Sun Jan 23 22:35:06 2022
*mangle
:PREROUTING ACCEPT [41:5019]
:INPUT ACCEPT [41:5019]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [44:4752]
:POSTROUTING ACCEPT [44:4752]
-A OUTPUT -d 192.168.0.0/16 -j RETURN
-A OUTPUT -d 172.16.0.0/12 -j RETURN
-A OUTPUT -d 10.0.0.0/8 -j RETURN
-A OUTPUT -d 34.117.59.81/32 -j MARK --set-xmark 0x3/0xffffffff
COMMIT
# Completed on Sun Jan 23 22:35:06 2022
# Generated by iptables-save v1.8.7 on Sun Jan 23 22:35:06 2022
*filter
:INPUT DROP [19:7746]
:FORWARD DROP [0:0]
:OUTPUT ACCEPT [846:62420]
-A INPUT -m state --state INVALID -j DROP
-A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT
-A INPUT -i lo -j ACCEPT
-A INPUT -p icmp -j ACCEPT
-A INPUT -p udp -m udp --dport 500 -j ACCEPT
-A INPUT -p esp -j ACCEPT
-A INPUT -p ah -j ACCEPT
-A INPUT -p tcp -m tcp --dport 22 -j ACCEPT
-A INPUT -s 192.168.0.0/16 -p tcp -m tcp --dport 3128 -j ACCEPT
-A INPUT -s 172.16.0.0/12 -p tcp -m tcp --dport 3128 -j ACCEPT
-A INPUT -s 10.0.0.0/8 -p tcp -m tcp --dport 3128 -j ACCEPT
-A FORWARD -m state --state INVALID -j DROP
-A FORWARD -m state --state RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -s 192.168.0.0/16 -d 10.0.6.0/24 -j ACCEPT
-A FORWARD -s 10.0.6.0/24 -d 192.168.0.0/16 -j ACCEPT
COMMIT
# Completed on Sun Jan 23 22:35:06 2022

When the packet get out of the machine it is given a source IP address corresponding to the default output interface.

So it is required to add a MASQUERADE rule on the output interface because the mark rules gives our packet another destination than the planned one.

iptables-legacy -t nat -A POSTROUTING -o eth2 -j MASQUERADE

for the sake of understanding netfilter order of processing : (via Plouf)

                     (input interface)
                             |
                             v
                     /-------+-------\
                     | sanity checks |
                     \-------+-------/
                             |
                             v hook NF_IP_PRE_ROUTING
                      +------+------+
                      | conntrack   |
                      | defrag.     |
                      +-------------+
                      | mangle      |
                      | PREROUTING  |
                      +-------------+
                      | nat (dst)   |
                      | PREROUTING  |
                      +------+------+
                             |
                             v
        local delivery  /----+----\    forward
       +---------------+   route   +---------------+
       |                \---------/                |
       v                                           v
 /-----+-----\                               /-----+-----\
 |  defrag.  |                               | TTL check |
 \-----+-----/                               | dec. TTL  |
       |                                     \-----+-----/
       v hook NF_IP_LOCAL_IN                       |
+------+------+                                    |
| mangle  (1) |                                    |
| INPUT       |                                    |
+-------------+                                    |
| filter      |                                    v
| INPUT       |                              /-----+-----\
+-------------+                              | DF check  |
| nat (src)(2)|                              \-----+-----/
+-------------+                                    |
| conntrack   |                                    |
| confirm     |                                    | 
+------+------+                               hook v NF_IP_FORWARD
       |                                    +------+------+
       v                                    | mangle  (1) |
/------+--------\                           | FORWARD     |
| local process |                           +-------------|
\------+--------/                           | filter      |
       |                                    | FORWARD     |
       v                                    +------+------+
 /-----+-----\                                     |
 |   route   |                                     |
 |   frag.   |                                     |
 \-----+-----/                                     |
       |                                           |
       v hook NF_IP_LOCAL_OUT                      |
+------+------+                                    |
| conntrack   |                                    |
| defrag.     |                                    |
+-------------+                                    |
| mangle      |                                    |
| OUTPUT      |                                    |
+-------------+                                    v
| nat (dst)(3)|                              /-----+-----\
| OUTPUT      |                              |   frag.   |
+-------------+                              \-----+-----/
| filter      |                                    |
| OUTPUT      |                                    |
+------+------+                                    |
       |                                           |
       v                                           |
 /-----+-----\                                     |
 |  reroute  |                                     |
 \-----+-----/                                     |
       |                                           |
       +------------------+     +------------------+
                          |     |
                          v     v hook NF_IP_POST_ROUTING
                      +---+-----+---+
                      | mangle  (1) |
                      | POSTROUTING |
                      +-------------+
                      | nat (src)   |
                      | defrag.     |
                      | POSTROUTING |
                      +-------------+
                      | conntrack   |
                      | confirm     |
                      | frag.       |
                      +------+------+
                             |
                             v
                   (output interface)