I am having an issue getting ports to forward to the VM's via MASQUERADE iptable rules. I had this working on the same servers with Proxmox. But now with XCP-NG I cannot get this to work. I did once in the past on one of these servers. But I have not been able to replicate it again. Network connection for outbound and getting updates and http/https request on the VM's is working just fine. I just cant get the ports to forward to the VoyOs/Pfsense VM's so I can pass them on to the internal VM's.
In short I can get network to the VM's but I cannot get the ports to forward.
I have two server both running XCP-NG but one is running VoyOS and the other OPNsense:
- XCP-NG : 8.2
- OPNsense : 21.7.1
- VoyOS : Latest
I followed these guides:
xenserver-single-ip-howto
setting-up-gateway-using-iptables-and-route-on-linux
create-private-network-bridge-proxmox-with-nat
xen-host-and-guest-sharing-same-ip
Network Map
- Management Interface
NIC 0 --> xenbr0
- MASQUERADE to xenbr1 (10.0.0.254) --> Wan- VoyOs/Pfsense (10.0.0.2) --> Lan Internal (10.0.1.1) --> VM's (10.0.1.x)
Tried:
iptables -t nat -A POSTROUTING -s '10.0.0.0/24' -o xenbr0 -j MASQUERADE
iptables -A PREROUTING -t nat -p tcp -i xenbr0 -d XXX.XXX.XXX.XX --dport 2727 -j DNAT --to 10.0.0.2:2727
and removing -d XXX.XXX.XXX.XX external IPaddress
iptables -A PREROUTING -t nat -p tcp -i xenbr0 --dport 2727 -j DNAT --to 10.0.0.2:2727
Then I removed those rules and tried:
iptables -I FORWARD 1 -i xenbr1 -j ACCEPT
iptables -t nat -A POSTROUTING -o xenbr0 -j MASQUERADE
And lastly tried:
iptables --append FORWARD --in-interface xenbr1 -j ACCEPT
iptables -D FORWARD --in-interface xenbr1 -j ACCEPT
IP Tables Rules
# iptables --list-rules
-P INPUT ACCEPT
-P FORWARD ACCEPT
-P OUTPUT ACCEPT
-N RH-Firewall-1-INPUT
-N xapi_nbd_input_chain
-N xapi_nbd_output_chain
-A INPUT -p tcp -m tcp --dport 10809 -j xapi_nbd_input_chain
-A INPUT -p gre -j ACCEPT
-A INPUT -j RH-Firewall-1-INPUT
-A FORWARD -j RH-Firewall-1-INPUT
-A OUTPUT -p tcp -m tcp --sport 10809 -j xapi_nbd_output_chain
-A RH-Firewall-1-INPUT -i lo -j ACCEPT
-A RH-Firewall-1-INPUT -p icmp -m icmp --icmp-type any -j ACCEPT
-A RH-Firewall-1-INPUT -i xenapi -p udp -m udp --dport 67 -j ACCEPT
-A RH-Firewall-1-INPUT -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A RH-Firewall-1-INPUT -p udp -m conntrack --ctstate NEW -m udp --dport 694 -j ACCEPT
-A RH-Firewall-1-INPUT -p tcp -m conntrack --ctstate NEW -m tcp --dport 22 -j ACCEPT
-A RH-Firewall-1-INPUT -p tcp -m conntrack --ctstate NEW -m tcp --dport 80 -j ACCEPT
-A RH-Firewall-1-INPUT -p tcp -m conntrack --ctstate NEW -m tcp --dport 443 -j ACCEPT
-A RH-Firewall-1-INPUT -p tcp -m tcp --dport 21064 -j ACCEPT
-A RH-Firewall-1-INPUT -p udp -m multiport --dports 5404,5405 -j ACCEPT
-A RH-Firewall-1-INPUT -j REJECT --reject-with icmp-host-prohibited
-A xapi_nbd_input_chain -j REJECT --reject-with icmp-port-unreachable
-A xapi_nbd_output_chain -j REJECT --reject-with icmp-port-unreachable
IP tables
# iptables -L && iptables -t nat -L
Chain INPUT (policy ACCEPT)
target prot opt source destination
xapi_nbd_input_chain tcp -- anywhere anywhere tcp dpt:nbd
ACCEPT gre -- anywhere anywhere
RH-Firewall-1-INPUT all -- anywhere anywhere
Chain FORWARD (policy ACCEPT)
target prot opt source destination
RH-Firewall-1-INPUT all -- anywhere anywhere
Chain OUTPUT (policy ACCEPT)
target prot opt source destination
xapi_nbd_output_chain tcp -- anywhere anywhere tcp spt:nbd
Chain RH-Firewall-1-INPUT (2 references)
target prot opt source destination
ACCEPT all -- anywhere anywhere
ACCEPT icmp -- anywhere anywhere icmp any
ACCEPT udp -- anywhere anywhere udp dpt:bootps
ACCEPT all -- anywhere anywhere ctstate RELATED,ESTABLISHED
ACCEPT udp -- anywhere anywhere ctstate NEW udp dpt:ha-cluster
ACCEPT tcp -- anywhere anywhere ctstate NEW tcp dpt:ssh
ACCEPT tcp -- anywhere anywhere ctstate NEW tcp dpt:http
ACCEPT tcp -- anywhere anywhere ctstate NEW tcp dpt:https
ACCEPT tcp -- anywhere anywhere tcp dpt:21064
ACCEPT udp -- anywhere anywhere multiport dports hpoms-dps-lstn,netsupport
REJECT all -- anywhere anywhere reject-with icmp-host-prohibited
Chain xapi_nbd_input_chain (1 references)
target prot opt source destination
REJECT all -- anywhere anywhere reject-with icmp-port-unreachable
Chain xapi_nbd_output_chain (1 references)
target prot opt source destination
REJECT all -- anywhere anywhere reject-with icmp-port-unreachable
Chain PREROUTING (policy ACCEPT)
target prot opt source destination
DNAT tcp -- anywhere anywhere tcp dpt:mgcp-callagent to:10.0.0.2:2727
Chain INPUT (policy ACCEPT)
target prot opt source destination
Chain OUTPUT (policy ACCEPT)
target prot opt source destination
Chain POSTROUTING (policy ACCEPT)
target prot opt source destination
MASQUERADE all -- 10.0.0.0/24 anywhere
/etc/sysconfig/iptables
# cat /etc/sysconfig/iptables
# Generated by iptables-save v1.4.21 on Mon Aug 16 20:14:40 2021
*filter
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [6179:9587238]
:RH-Firewall-1-INPUT - [0:0]
:xapi_nbd_input_chain - [0:0]
:xapi_nbd_output_chain - [0:0]
-A INPUT -p tcp -m tcp --dport 10809 -j xapi_nbd_input_chain
-A INPUT -p gre -j ACCEPT
-A INPUT -j RH-Firewall-1-INPUT
-A FORWARD -j RH-Firewall-1-INPUT
-A OUTPUT -p tcp -m tcp --sport 10809 -j xapi_nbd_output_chain
-A RH-Firewall-1-INPUT -i lo -j ACCEPT
-A RH-Firewall-1-INPUT -p icmp -m icmp --icmp-type any -j ACCEPT
-A RH-Firewall-1-INPUT -i xenapi -p udp -m udp --dport 67 -j ACCEPT
-A RH-Firewall-1-INPUT -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A RH-Firewall-1-INPUT -p udp -m conntrack --ctstate NEW -m udp --dport 694 -j ACCEPT
-A RH-Firewall-1-INPUT -p tcp -m conntrack --ctstate NEW -m tcp --dport 22 -j ACCEPT
-A RH-Firewall-1-INPUT -p tcp -m conntrack --ctstate NEW -m tcp --dport 80 -j ACCEPT
-A RH-Firewall-1-INPUT -p tcp -m conntrack --ctstate NEW -m tcp --dport 443 -j ACCEPT
-A RH-Firewall-1-INPUT -p tcp -m tcp --dport 21064 -j ACCEPT
-A RH-Firewall-1-INPUT -p udp -m multiport --dports 5404,5405 -j ACCEPT
-A RH-Firewall-1-INPUT -j REJECT --reject-with icmp-host-prohibited
-A xapi_nbd_input_chain -j REJECT --reject-with icmp-port-unreachable
-A xapi_nbd_output_chain -j REJECT --reject-with icmp-port-unreachable
COMMIT
# Completed on Mon Aug 16 20:14:40 2021
# Generated by iptables-save v1.4.21 on Mon Aug 16 20:14:40 2021
*nat
:PREROUTING ACCEPT [5:322]
:INPUT ACCEPT [3:164]
:OUTPUT ACCEPT [1:73]
:POSTROUTING ACCEPT [1:73]
-A PREROUTING -i xenbr0 -p tcp -m tcp --dport 2727 -j DNAT --to-destination 10.0.0.2:2727
-A POSTROUTING -s 10.0.0.0/24 -o xenbr0 -j MASQUERADE
COMMIT
# Completed on Mon Aug 16 20:14:40 2021
Ifconfig
# ifconfig
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
ether 78:2b:cb:3c:81:65 txqueuelen 1000 (Ethernet)
RX packets 24705 bytes 3641727 (3.4 MiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 47612 bytes 37970921 (36.2 MiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
eth1: flags=4099<UP,BROADCAST,MULTICAST> mtu 1500
ether 78:2b:cb:3c:81:66 txqueuelen 1000 (Ethernet)
RX packets 0 bytes 0 (0.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 0 bytes 0 (0.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536
inet 127.0.0.1 netmask 255.0.0.0
loop txqueuelen 1000 (Local Loopback)
RX packets 5951 bytes 33860750 (32.2 MiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 5951 bytes 33860750 (32.2 MiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
vif1.0: flags=4291<UP,BROADCAST,RUNNING,NOARP,MULTICAST> mtu 1500
ether fe:ff:ff:ff:ff:ff txqueuelen 32 (Ethernet)
RX packets 6 bytes 444 (444.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 0 bytes 0 (0.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
vif1.1: flags=4291<UP,BROADCAST,RUNNING,NOARP,MULTICAST> mtu 1500
ether fe:ff:ff:ff:ff:ff txqueuelen 32 (Ethernet)
RX packets 1442 bytes 95984 (93.7 KiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 1336 bytes 143058 (139.7 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
xapi0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
ether be:16:43:65:a2:4f txqueuelen 1000 (Ethernet)
RX packets 6 bytes 444 (444.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 0 bytes 0 (0.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
xenbr0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet XXX.XXX.XXX.XX netmask 255.255.255.252 broadcast XXX.XXX.XXX.XX
ether 78:2b:cb:3c:81:65 txqueuelen 1000 (Ethernet)
RX packets 23841 bytes 3101836 (2.9 MiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 33197 bytes 36975647 (35.2 MiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
xenbr1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 10.0.0.254 netmask 255.255.255.0 broadcast 10.0.0.255
ether 78:2b:cb:3c:81:66 txqueuelen 1000 (Ethernet)
RX packets 1442 bytes 95984 (93.7 KiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 1336 bytes 143058 (139.7 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
/etc/sysctl.conf
# sysctl settings are defined through files in
# /usr/lib/sysctl.d/, /run/sysctl.d/, and /etc/sysctl.d/.
#
# Vendors settings live in /usr/lib/sysctl.d/.
# To override a whole file, create a new file with the same in
# /etc/sysctl.d/ and put new settings there. To override
# only specific settings, add a file with a lexically later
# name in /etc/sysctl.d/ and put new settings there.
#
# For more information, see sysctl.conf(5) and sysctl.d(5).
net.ipv4.ip_forward = 1
