Score:0

Why is my policy-routing setup not working?

br flag

I have an EC2 machine running Ubuntu 20.04 with 2 ethernet interfaces. They are both connected to the same subnet and both are reachable from the outside. I have created 2 custom route tables, one for each interface and both just containing the route entries for the machine itself, the immediate subnet and a default gateway via the corresponding interface. I can add a rule for all traffic to use either of the custom route tables and it works: the machine will use the specified interface and the corresponding ip address for outbound communication. However if I add rules to use either custom route table based on packet markings that I apply via iptables, everything stops working and no traffic is leaving my machine.

Here are the main bits of relevant configuration:

$ sudo ip route
10.0.11.0/24 dev ens6 proto kernel scope link src 10.0.11.52
10.0.11.0/24 dev ens5 proto kernel scope link src 10.0.11.201

$ sudo ip route show table if1
default via 10.0.11.1 dev ens5 proto static metric 100
10.0.11.0/24 dev ens5 proto static scope link src 10.0.11.52
10.0.11.201 dev ens5 proto static scope link metric 100

$ sudo ip route show table if2
default via 10.0.11.1 dev ens6 proto static metric 200
10.0.11.0/24 dev ens6 proto static scope link src 10.0.11.201
10.0.11.52 dev ens6 proto static scope link metric 200

$ ip rule
0:      from all lookup local
46:     from all fwmark 0x3 lookup if1
47:     from all fwmark 0x2 lookup if2
50:     from 10.0.11.201 lookup if1
100:    from 10.0.11.52 lookup if2
32766:  from all lookup main
32767:  from all lookup default

$ sudo iptables -S -t mangle
-P PREROUTING ACCEPT
-P INPUT ACCEPT
-P FORWARD ACCEPT
-P OUTPUT ACCEPT
-P POSTROUTING ACCEPT
-A OUTPUT -m owner --uid-owner 1002 -j MARK --set-xmark 0x3/0xffffffff
-A OUTPUT -m owner --uid-owner 1003 -j MARK --set-xmark 0x2/0xffffffff

My goal is to route traffic from the user with uid 1002 over interface ens5 (if1) and traffic from user with uid 1003 via ens6 (if2) but I'm doing something wrong somewhere and it drives me nuts.... I can see that marked packets are being treated differently, so something is happening. If I assign a default route to the main routing table and then just mark traffic for uid 1002, all users can make outbound connections except for uid 1002. So I know the marking works, I know marked packets get routed differently and I know my custom route tables work but somewhere in this constellation there's an error and I can't find it. Any help would be greatly appreciated at this point!

--edit--

I arrived at this by basing my setupn on the approach outlined here: Trying to route outgoing http requests through VPN on Ubuntu Server 12.04 and some customizations as I don't wanna route traffic based on specific ports but originating uid.

br flag
I found a solution to my problem by doing it differently. Instead of marking packets with iptables and then using ip rules to activate different routing tables as described above I ditched iptables entirely and just put this statement into the ip rules: ``sudo ip rule add uidrange 1000-1000 lookup if1`` and ``sudo ip rule add uidrange 1001-1004 lookup if2`` works like a charm! :) I'm still interested to find out why my previous example didn't work...
mangohost

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.