Score:1

Server

Routing issue on Debian 11 VM

Granwille

2/16/24, 2:49 PM

I updated my OPnsense "router/gateway" to 23.1 a few days ago, and now I am experiencing an issue with one of my machines. I have one machine with OPnsense installed that acts as a basic NAT router for my private subnet 192.168.50.0/24. This machine/router uses the private IP 192.168.50.1, which is the gateway for my 192.168.50.0/24 network.

I have a physical KVM machine with 1x Public NIC and 1x Private NIC. On this machine, I have a VM running also equipped with 1x Public IP and then 1x Private IP falling in the 192.168.50.0/24 subnet as its private network. I am trying to mount an NFS share on this VM. I have two NFS shares and only ONE NFS share mounts successfully while the second refuses.

This is what the NFS mounts look like:

Successful Mount

root@s-145-VM:~# mount -t nfs -vvv 102.165.XXX.YYY:/data/secondary /mnt/SecStorage/test
mount.nfs: timeout set for Thu Feb 16 14:07:58 2023
mount.nfs: trying text-based options 'vers=4.2,addr=102.165.XXX.YYY,**clientaddr=197.189.XXX.YYY**'
root@s-145-VM:~#

Failed Mount:

root@s-145-VM:~# mount -t nfs 102.165.XXX.ZZZ:/data/secondary /mnt/SecStorage/test
mount.nfs: access denied by server while mounting 102.165.XXX.ZZZ:/data/secondary
root@s-145-VM:~# mount -t nfs -vvv 102.165.XXX.ZZZ:/data/secondary /mnt/SecStorage/test
mount.nfs: timeout set for Thu Feb 16 14:07:11 2023
mount.nfs: trying text-based options 'vers=4.2,addr=102.165.XXX.ZZZ,**clientaddr=192.168.50.53**'
mount.nfs: mount(2): Operation not permitted
mount.nfs: trying text-based options 'addr=102.165.XXX.ZZZ'
mount.nfs: prog 100003, trying vers=3, prot=6
mount.nfs: trying 102.165.XXX.ZZZ prog 100003 vers 3 prot TCP port 2049
mount.nfs: prog 100005, trying vers=3, prot=17
mount.nfs: trying 102.165.XXX.ZZZ prog 100005 vers 3 prot UDP port 892
mount.nfs: mount(2): Permission denied
mount.nfs: access denied by server while mounting 102.165.XXX.ZZZ:/data/secondary

As you can see, in the failed mount attempt, the VM passes its private IP address, 192.168.50.53 when it tries to mount the second NFS. And not its public IP 197.189.XXX.YYY. And the reason this is happening is that the VM automatically after each reboot adds the following line to its IP routing table:

102.165.XXX.ZZZ via 192.168.50.1 dev eth1

When I delete this entry from the routing table, the NFS mounts successfully. Something in the OPnsense router or Linux kernel is causing this behaviour and making it create this entry in the IP route table. And I have no idea what it could be. What puzzles me, it only creates this entry for this IP and not for the other NFS, which is successful and they are basically on the same network space 102.165.XXX

I have double-checked the Opnsense server and rebooted it many times and I cannot pinpoint what is causing this behaviour in the VM.

Any suggestions for this will be extremely appreciated, please.

EDIT:

So after some more investigation, the IP route that is added (102.165.XXX.ZZZ via 192.168.50.1 dev eth1) is 100% correct and expected. When I remove this entry from the IP route, the VM tries to access NFS via the gateway of the public IP (default route), and then it works.

So essentially, it means that somehow my router is not passing the traffic accordingly. But if I run a ping test from my router its successful, see here:

# /sbin/ping -4 -S '192.168.50.1'  -c '3' '102.165.XXX.YYY'
PING 102.165.XXX.YYY (102.165.XXX.YYY) from 192.168.50.1: 56 data bytes
64 bytes from 102.165.XXX.YYY: icmp_seq=0 ttl=60 time=1.140 ms
64 bytes from 102.165.XXX.YYY: icmp_seq=1 ttl=60 time=1.200 ms
64 bytes from 102.165.XXX.YYY: icmp_seq=2 ttl=60 time=1.348 ms

So what am I missing? Should I add some sort of route manually to the KVM host maybe?

129

0 + 9

debian

routing

linux-networking

linux-kernel

opnsense

Granwille

2/16/24, 3:10 PM

Hi @Appleoddity, I trust you're well. Sorry for tagging you so randomly but you're the only person that I know that could possibly help with this Will appreciate it if you can have a look at this for me, thank you in advance.

A.B

2/18/24, 11:13 PM

The route could be amended to `102.165.XXX.ZZZ via 192.168.50.1 dev eth1 src 197.189.XXX.YYY` . That's what to do when an internal glue-only private IP LAN is involved in routing: use the public IP address rather than the interface's non-public IP address. The remaining problem is: where is it done to change it?

A.B

2/18/24, 11:18 PM

Also I suspect that when you remove the route, you get asymmetric routing: system public IP -> internet -> target -> private LAN -> back to system. And I'm surprised it works (usually systems (here the NFS server or whatever involved part receiving traffic on the wrong interface) use [strict reverse path forwarding](https://www.rfc-editor.org/rfc/rfc3704#section-2.2) which would not allow asymmetric routes)

Granwille

2/20/24, 6:29 AM

@A.B, adding `102.165.XXX.ZZZ via 192.168.50.1 dev eth1 src 197.189.XXX.YYY` does not seem to work either. It only works by not having any route in the IP table. So I am really puzzled by this issue and not sure I understand why this is happening.

A.B

2/20/24, 7:45 AM

Can you check what I wrote about asymmetric routing? When it's working, are packets both sent and received on the same interface, or not (you'll need two tcpdump commands to check)?

Granwille

2/21/24, 6:49 AM

@A.B thank you for helping me thus far, I am a bit still new to networking/routing concepts, do you mind giving me the tcpdump command that you expect me to run in the VM, please?

A.B

2/21/24, 7:18 AM

I'm sorry but I don't think I understood the layout correctly so I'm not sure what I wrote made much sense. Anyway, that would be one "tcpdump -n -i eth0" + one "tcpdump -n -i eth1". If accessing using ssh adding "not port 22" would probably be useful.

Granwille

2/21/24, 3:10 PM

Hi @A.B I think I may have solved it. I will give full details shortly. I am, however, still getting an error to mount the NFS storage to my VM. The error now reads, "Access/Permissions Denied". And I do a manual mount with the -vvv parameter, client is trying to connect to my NFS from a private IP address such as 192.168.50.78. And I believe this is the reason it's being denied. So I add a NAT outbound rule on OPNsense, from 192.168.50.78/32 to the NFS IP, and tested a manual mount again via -vvv now the mount process just hangs. And for some reason, I can no longer ping the NFS server. Ideas

Granwille

2/21/24, 8:57 PM

@A.B here's a details description of the issue behind this one: https://serverfault.com/questions/1123448/nfs-mouting-failing-due-to-illegal-port

Elon Musk

I sit in a Tesla and translated this thread with Ai:

EN: Routing issue on Debian 11 VM

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.