It would be very difficult (but not impossible) to have a Linux system, using a single routing stack, perform multi-homed routing of packets from itself to itself without using the lo interface.
But it's very easy on Linux to create additional network stacks to simulate multiple systems within one system: by using network namespaces.
Here one can relinquish one of the two NICs to a new network namespace which will be a peer to the initial (host) network namespace. They won't communicate directly, but only through the external router. Let's assume interfaces are really named A and B and that the router uses addresses 10.0.1.1/24 + 10.0.2.1/24.
create a new network namespace with additional facilitated management when using iproute2 tools (under the hood, namespace pseudo-files are mounted to keep the resource around without a process, etc.).
ip netns add sideB
move interface B to the new network namespace:
ip link set dev B netns sideB
note: wireless interfaces require using the iw command instead.
configure the new network namespace:
All of its network settings are lost when an interface changes namespace (both on the host where the interface disappears triggering addresses disappearing triggering routes disappearing, and on the new network namespace):
ip -n sideB link set dev B up
ip -n sideB address add 10.0.2.5/24 dev B
Not needed but in case iperf3 gets confused, have a functional loopback interface:
ip -n sideB link set lo up
configure routes between the two sides (initial/host namespace might have it already through the default route, but let's be explicit)
ip route add 10.0.2.0/24 via 10.0.1.1 dev A
ip -n sideB route add 10.0.1.0/24 via 10.0.2.1 dev B
Now one can run (in two terminals):
iperf3 -s
ip netns exec sideB iperf3 -c 10.0.1.5
with the client running in the new network namespace.
Once measuring is over, deleting the network namespace will return the NIC to the host namespace. But if any process is left by mistake using this network namespace (something like ip netns exec side B setsid sleep 9999) the ip netns delete sideB below will only remove the namespace from the view of the iproute2 tools, but not actually remove the namespace until the process ends, leaving the NIC difficult to recover back. So better move it back first:
ip -n sideB link set dev B netns 1
ip netns delete sideB
where 1 means PID 1's network namespace: the initial/host namespace. Additionally this allows to perform the experiment inside a container having been given physical NICs (because there 1's network namespace is the container's namespace, instead of the actual host's namespace) or the physical NIC would reappear on the host and be lost forever to the container if there's no access to the host.
If there's no tool (like udev + ifupdown's allow-hotplug or NetworkManager) detecting the apparition of the NIC, it will again have to be configured again on its return.
