Score:0

strange NAT issues with pfSense to vagrant VM

tr flag

This one has got me confused:

I have a pfSense firewall (let's call it pfs) and behind it several servers. I NAT several services from my public IP to different servers on the LAN without any issues.

On one of the servers (let's call it s1) I'm running a vagrant (with libvirt) VM (let's call it v1) with a public network configured, which gets IP 192.168.1.159 via pfss DHCP server.

Now I configure a simple NAT on pfs to access s1's SSH, say <wan>:6622 -> s1:22 and access it on mydomain.com:6622. No problem.

I can also access v1:22 (or the equivalent 192.168.1.159:22) with a valid ssh user from within the LAN without issue.

Now I add a simple NAT on pfs, say <wan>:6722 -> v1:22. Now trying to access mydomain.com:6722 does not work?!

The objective is to add even "another layer": running containers with public ports, e.g. --publish 9980:80 on v1 and access them as e.g. v1:9980 and from mydomain.com:9980 with the corresponding NAT on pfs like <wan>:9980 -> v1:9980. From the LAN this is also working as expected (i.e. I can access v1:9980 from the LAN), but a NAT via pfs is not.

I have similar setups working within the same network on different machines without problems. I even have another (non-vagrant, but also libvirt) VM on s1 to which I can ssh over NAT via my public IP perfectly fine. But somehow the above doesn't work with the vagrant machine, and I'm really at a loss what might be causing this issue. (FWIW I have net.ipv4.forward enabled on v1).

EDIT:

I got one step closer: if I nuke the first existing NIC of the vagrant VM using virt-manager, and set the second VM to rtl8139 instead of virtio (and then restart), I lose vagrant ssh capability but NAT then works. So the question then becomes: how to configure via vagrant provisioning such that we have a similar config, I suppose that means that the public network must be on the default interface then?

Score:0
tr flag

Solution:

The cause is that vagrant requires (and therefore configures) its local interface (private network) as primary, with no standard method to override that. (Some info is here, but the vagrant people admit that they're themselves confused by the topic...)

A (more robust) variation on the concept to tweak the default routes brought the solution. I'm using an ansible provisioner, where I execute the following (on the guest, via the provisioning playbook):

  - name: remove wrong default route on eth0 (again)
    shell: |
      eval $(route -n | awk '$0~/[.0]{4}/ && $3~/[.0]{4}/ && $8~/eth0/ { printf "ip route del default via %s dev %s; ",$3,$8 }')

(interestingly it needs to be executed twice (or after a while?), possibly because the network hasn't come up fully when the provision script started?)

This removes the default route on eth0 (the public_network would only be automatically configured on eth1 by vagrant) and makes external connections work as expected. This is presumably (speculating here) due to the fact that responses to incoming NATed requests are routed to the firewall via the default route on eth0 by default (which by vagrant default has priority), which confuses the NAT FW because it came in on the VM's eth1. So removing eth0 as default replies to external requests on the same interface as they came in on.

mangohost

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.