Score:0

Multipath routing on centos adding the second route breaks the first

de flag

I have a Centos box with 2 interfaces that have routes to the internet. One 192.168.1.254, the other 192.168.133.11. I can route through 133.11 perfectly OK when it is the only route specified. When I add the other route to the table, the first stops working.

[root@sinister2 yum.repos.d]# ip route
default via 192.168.133.11 dev ens2u3
192.168.1.0/24 dev eno1 proto kernel scope link src 192.168.1.70 metric 100
192.168.133.0/24 dev ens2u3 proto kernel scope link src 192.168.133.108
[root@sinister2 yum.repos.d]# curl ifconfig.me
92.40.187.117
[root@sinister2 yum.repos.d]# route del default
[root@sinister2 yum.repos.d]# ip route add default scope global nexthop via 192.168.133.11 dev ens2u3 weight 1 nexthop via 192.168.1.254 dev eno1 weight 1
[root@sinister2 yum.repos.d]# curl ifconfig.me
86.133.6.21
[root@sinister2 yum.repos.d]# ip route
default
        nexthop via 192.168.133.11 dev ens2u3 weight 1
        nexthop via 192.168.1.254 dev eno1 weight 1
192.168.1.0/24 dev eno1 proto kernel scope link src 192.168.1.70 metric 100
192.168.133.0/24 dev ens2u3 proto kernel scope link src 192.168.133.108
[root@sinister2 yum.repos.d]# curl https://developer.download.nvidia.com/
curl: (7) Failed to connect to developer.download.nvidia.com port 443: Connection timed out
[root@sinister2 yum.repos.d]# traceroute developer.download.nvidia.com
traceroute to developer.download.nvidia.com (152.199.20.126), 30 hops max, 60 byte packets
 1  * * *
...manually clipped all the stars...
30  * * *
[root@sinister2 yum.repos.d]# route del default
[root@sinister2 yum.repos.d]# route add default gw 192.168.133.11
[root@sinister2 yum.repos.d]# traceroute developer.download.nvidia.com
traceroute to developer.download.nvidia.com (152.199.20.126), 30 hops max, 60 byte packets
 1  _gateway (192.168.133.11)  2.525 ms  2.667 ms  2.942 ms
 2  * * *
 3  172.25.83.33 (172.25.83.33)  301.910 ms  307.052 ms  306.962 ms
 4  * * *
 5  * * *
 6  172.25.88.89 (172.25.88.89)  305.709 ms * 172.25.88.93 (172.25.88.93)  292.299 ms
 7  * * *
 8  172.25.67.146 (172.25.67.146)  48.475 ms 172.25.67.158 (172.25.67.158)  53.182 ms 172.25.67.146 (172.25.67.146)  53.096 ms
 9  * * *
10  185.153.238.161 (185.153.238.161)  46.396 ms  57.785 ms  57.785 ms
11  ae-65.core1.lhc.edgecastcdn.net (152.195.96.139)  29.347 ms  39.733 ms  51.525 ms
12  152.199.20.126 (152.199.20.126)  39.295 ms  39.182 ms  39.041 ms
13  152.199.20.126 (152.199.20.126)  39.250 ms  39.251 ms  39.015 ms

That nvidia site routes through ens2u3, I can see the traffic on the interface :

tcpdump -i ens2u3
16:26:37.528360 ARP, Request who-has 192.168.133.11 tell sinister2, length 28
16:26:37.530385 ARP, Reply 192.168.133.11 is-at 76:e6:8d:8e:99:a8 (oui Unknown), length 28
16:26:39.256368 IP sinister2.42630 > 152.199.20.126.https: Flags [S], seq 1095059191, win 29200, options [mss 1460,sackOK,TS val 1024650559 ecr 0,nop,wscale 7], length 0
16:26:47.768368 IP sinister2.42630 > 152.199.20.126.https: Flags [S], seq 1095059191, win 29200, options [mss 1460,sackOK,TS val 1024659071 ecr 0,nop,wscale 7], length 0
etc...

None of my syns ever got an ack. Earlier, when I was routing everything through ens2u3, it did :

16:23:18.233472 IP sinister2.55236 > 152.199.20.126.https: Flags [S], seq 1471928043, win 29200, options [mss 1460,sackOK,TS val 466587424 ecr 0,nop,wscale 7], length                    0
16:23:18.276348 IP 152.199.20.126.https > sinister2.55236: Flags [S.], seq 2554681057, ack 1471928044, win 14600, options [mss 1220,nop,wscale 12,sackOK,TS val 371740                   3495 ecr 466587424], length 0
16:23:18.276428 IP sinister2.55236 > 152.199.20.126.https: Flags [.], ack 1, win 229, options [nop,nop,TS val 466587467 ecr 3717403495], length 0

Traceroute just shows stars for any address over ens2u3 and a connection to anything on eno1.

Anyone got any good ideas? Spot anything I've done wrong?

Do I need to do some sort of clearing of existing routes when changing them? In the first example, I can get to ifconfig.me via the new route, but not use the old route to nvidia, which makes me think it isn't that simple...

vn flag
https://access.redhat.com/solutions/53031 - use rp_filter if you are cool with asymmetric routing or use policy based routes.
mangohost

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.