I have the following network:

.________.             ._______.
| vm-cli | <==> <enp1s0| vm-gw |enp2s0> <==> internet
·--------·             ·-------·

vm-cli has address 192.168.255.2; enp1s0@vm-gw has 192.168.255.1; enp2s0@vm-gw, doesn't matter.

I want all the internet traffic to go through Tor. For that, I configure Tor to use a transparent socket, and use policy routing to forward traffic to it. I first add a routing rule and define an alternative routing table that route everything to localhost:

ip rule add fwmark 1 table 1
ip route add local 0.0.0.0/0 dev lo table 1

Then I configure first the firewall to redirect vm-gw output to the proxy:

table proxy {
    # pass the diverted traffic to the proxy
    chain divert {
        type filter hook prerouting priority filter; policy accept;
        ip protocol tcp mark 1 tproxy to :9040
    }
}

table local {
    # use destination NAT for the DNS traffic as it doesn't work with tproxy
    chain dstNAT {
        type nat hook output priority mangle; policy accept;
        udp dport domain redirect to :9053
    }

    # mark all non-Tor traffic going to the internet connected interface to be rerouted
    chain reroute {
        type route hook output priority mangle; policy accept;
        oif enp2s0 skuid != tor ip protocol tcp meta mark set 1
    }
}

Nice, I test it with dig and curl: everything goes through Tor.

What doesn't work

Now I add the following rule to redirect the traffic from vm-cli or any other host using vm-gw as gateway:

table remote {
    chain reroute {
        type nat hook prerouting priority mangle; policy accept;
        # destination NAT'ing DNS traffic
        udp dport domain redirect to :9053
        # mark tcp traffic from enp1s0 for policy routing
        iif enp1s0 ip protocol tcp meta mark set 1
    }
}

I try making a DNS query, it works. I try to make an HTTP request, no answer…

Tracing packets

Tracing packets and looking at Tor log, I can see that vm-cli TCP traffic does get to Tor. But curl on vm-cli seems that it doesn't see any HTTP answer.

So I try to trace the HTTP server response. As it comes from Tor I won't see any HTTP traffic in the input, but I will see it in vm-gw output. Then I add the following rule for tracing:

table local {
    chain output {
        type filter hook output priority filter; policy accept;
        tcp sport 80 ip daddr 192.168.255.2 meta nftrace set 1
    }

    […]
}

Doing curl http://perdu.com on vm-cli I get the following trace on vm-gw:

trace id 2fc29112 ip local output packet: oif "enp1s0" ip saddr 208.97.177.124 ip daddr 192.168.255.2 ip dscp cs0 ip ecn not-ect ip ttl 64 ip id 0 ip length 60 tcp sport 80 tcp dport 57082 tcp flags == 0x12 tcp window 65160
trace id 2fc29112 ip local output rule tcp sport 80 ip daddr 192.168.255.2 meta nftrace set 1 (verdict continue)
trace id 2fc29112 ip local output verdict continue
trace id 2fc29112 ip local output policy accept

And there, everything looks correct to me… there's an answer from the HTTP server, the destination is vm-cli using the enp1s0 interface.

I try to trace the packet on vm-cli. There we can see the outgoing packet and its response:

trace id d9101d30 ip filter output packet: oif "enp2s0" ip saddr 192.168.255.2 ip daddr 208.97.177.124 ip dscp cs0 ip ecn not-ect ip ttl 64 ip id 40142 ip length 60 tcp sport 57124 tcp dport 80 tcp flags == syn tcp window 64240 
trace id d9101d30 ip filter output rule tcp dport 80 meta nftrace set 1 (verdict continue)
trace id d9101d30 ip filter output verdict continue
trace id d9101d30 ip filter output policy accept

trace id 51163d64 ip filter prerouting packet: iif "enp2s0" ether saddr 52:54:00:bd:38:6d ether daddr 52:54:00:f0:a2:51 ip saddr 208.97.177.124 ip daddr 192.168.255.2 ip dscp cs0 ip ecn not-ect ip ttl 64 ip id 0 ip length 60 tcp sport 80 tcp dport 57124 tcp flags == 0x12 tcp window 65160 
trace id 51163d64 ip filter prerouting rule meta nftrace set 1 (verdict continue)
trace id 51163d64 ip filter prerouting verdict continue 
trace id 51163d64 ip filter prerouting policy accept

I can see:

1. an outgoing packet from 192.168.255.2:57124 to 208.97.177.124:80
2. an answer from 208.97.177.124:80 to 192.168.255.2:57124

Addresses and ports match… curl --verbose tells that the connection is established, but why the HTTP request doesn't get any answer?

netcat confirms me that the HTTP response doesn't get to the application. I also tried hping, and with the tproxy setup I get duplicates:

hping perdu.com --destport 80 --syn --count 10
HPING perdu.com (enp2s0 208.97.177.124): S set, 40 headers + 0 data bytes
len=44 ip=208.97.177.124 ttl=64 DF id=0 sport=80 flags=SA seq=0 win=64240 rtt=9.8 ms
len=44 ip=208.97.177.124 ttl=64 DF id=0 sport=80 flags=SA seq=1 win=64240 rtt=9.8 ms
DUP! len=44 ip=208.97.177.124 ttl=64 DF id=0 sport=80 flags=SA seq=0 win=64240 rtt=1049.9 ms
len=44 ip=208.97.177.124 ttl=64 DF id=0 sport=80 flags=SA seq=2 win=64240 rtt=9.8 ms
DUP! len=44 ip=208.97.177.124 ttl=64 DF id=0 sport=80 flags=SA seq=1 win=64240 rtt=1099.9 ms
len=44 ip=208.97.177.124 ttl=64 DF id=0 sport=80 flags=SA seq=3 win=64240 rtt=29.8 ms
DUP! len=44 ip=208.97.177.124 ttl=64 DF id=0 sport=80 flags=SA seq=2 win=64240 rtt=1059.9 ms
DUP! len=44 ip=208.97.177.124 ttl=64 DF id=0 sport=80 flags=SA seq=0 win=64240 rtt=3120.0 ms
len=44 ip=208.97.177.124 ttl=64 DF id=0 sport=80 flags=SA seq=4 win=64240 rtt=29.8 ms
DUP! len=44 ip=208.97.177.124 ttl=64 DF id=0 sport=80 flags=SA seq=3 win=64240 rtt=1079.9 ms

--- perdu.com hping statistic ---
5 packets tramitted, 10 packets received, -100% packet loss
round-trip min/avg/max = 9.8/749.9/3120.0 ms