Score:0

On Linux, how to send packets with local destination NOT through the loopback interface

cn flag

I have a Linux server with two ethernet interfaces which are connected to the same switch and are in the same subnetwork. The topology is as following

+--------------------+
|                    |
|         +----------+        +----------+
|         | enp1s0f0 |<======>|          |
|         +----------+        | ethernet |
|  Server            |        |          |
|         +----------+        |  switch  |
|         | enp1s0f1 |<======>|          |
|         +----------+        +----------+
|                    |
+--------------------+

The IP address of these two interfaces are 172.16.0.100 and 172.16.1.100 respectively.

$ ip addr
...
4: enp1s0f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 10:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff
    inet 172.16.0.100/16 brd 172.16.255.255 scope global noprefixroute enp1s0f0
       valid_lft forever preferred_lft forever
5: enp1s0f1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 10:00:00:00:01:00 brd ff:ff:ff:ff:ff:ff
    inet 172.16.1.100/16 brd 172.16.255.255 scope global noprefixroute enp1s0f1
       valid_lft forever preferred_lft forever
...

Now I want to send packets from enp1s0f0 to enp1s0f1 through the external switch (for testing purpose, from switch's view there're two individual computer communicate to each other). So I bind local address of socket to 172.16.0.100 and connect it to 172.16.1.100 (for example, use telnet -b 172.16.0.100 172.16.1.100 22 command). However, I can only see these packets go though the lo interface (by inspecting captured packets), rather than these two ethernet interface and the switch.

So my first question is how can I force Linux to send these packets with local destination through external interface?

I think I have set addresses and routing table correctly. The routing table contains the following entries

$ ip route
...
172.16.0.0/16 dev enp1s0f0 proto kernel scope link src 172.16.0.100 metric 100
172.16.0.0/16 dev enp1s0f1 proto kernel scope link src 172.16.1.100 metric 101

And the ARP table also seems to be configured correctly

$ arp -n
Address                  HWtype  HWaddress           Flags Mask            Iface
172.16.1.100             ether   10:00:00:00:01:00   CM                    enp1s0f0
172.16.0.100             ether   10:00:00:00:00:00   CM                    enp1s0f1

However Linux still routes these packets via the lo interface

$ ip route get from 172.16.0.100  172.16.1.100
local 172.16.1.100 from 172.16.0.100 dev lo uid 1000
    cache <local>
$ ip route get from 172.16.1.100  172.16.0.100
local 172.16.0.100 from 172.16.1.100 dev lo uid 1000
    cache <local>

I also tried pinging from these interfaces, e.g. ping -I enp1s0f0 172.16.1.100. The captured packets shows that the ICMP packets sent can arrive destination via the switch as expected, however there're no reply packets seen on enp1s0f1 interface. I've searched for solution for this problem and found this answer, and it says that this could relate to reverse path filtering. So I tried to disable rp_filter by setting default and interfaces' rp_filter value to either 0 and 2, but it didn't solve the problem. My additional question is is my configuration correct and how should I make Linux to reply to ICMP ping packets in this situation?

My OS is Ubuntu 20.04 and ICMP replies works well on other interfaces, and there's no iptable entries configured.

Donlon avatar
cn flag
I'm testing performance of the external Ethernet switch, so these packets must be sent through the interfaces. I've considered isolating different NIC using multiple computers or with multiple virtual machines, but it seems to be less efficient and inconvenient to control.
Nikita Kipriyanov avatar
za flag
Use network namespaces of different VRFs for different interfaces.
mangohost

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.