Join Log in
Score:0
gstlouis avatar Server

reaching apache public reverse proxy virtualhost url from localhost behind a NAT using iptables

bo flag
gstlouis

I have an issue with a guest using virsh behind a server running with iptables firewall. This guest hosts websites, one is mattermost with reverse proxy.

Everything is working well. I then installed collabora online 1: https://www.collaboraoffice.com/code/. Super cool to open documents and they have a plugin for mattermost

I tested this plugin from a remote server also running mattermost that reaches out to this local box behind my iptables, and it is working perfectly. However, when I test this plugin from the guest local server itself, behind iptables, so NAT basically, it cannot find itself. I get timed out.

So, my guest behind iptables, with proper rules setup to pass traffic, holds mattermost AND CollaboraOnline, but if I point the public URL from mattermost to fetch CollaboraOnline, both localhost they cannot find each other. I can't do localhost:9980 or 127.0.0.1:9980 (which is where CollaboraOnline is) in the plugin, it does not like the port in the address...

If I do curl https://CollaboraOnline.domain.ca I can see it times out.

If I edit /etc/hosts file on localhost server to 127.0.0.1 CollaboraOnline.domain.ca, then curl works, mattermost-plugin does find CollaboraOnline, but it still will not work because I end up with an SSL type verification error like this.

AH02032: Hostname provided via SNI and hostname provided via HTTP have no compatible SSL setup how to bypass

So now I am running out of bright ideas. I have another public box not behind virsh with iptables, if I do curl to one of its localhost, everything is fine. This only leads me to believe iptables might need a rule for my guest running mattermost and CollaboraOnline to be able to loop back to itself when requesting a public URL it serves itself?!?

Does anyone have any idea about this?

My guest VM is 192.168.122.126 and my parent server hosting Vrish guests and iptables is 192.168.122.1

Here are the iptables rules ( removed some clutter like fail2ban stuff)

iptables -L
Chain FORWARD (policy ACCEPT)
target     prot opt source               destination         
ACCEPT     all  --  anywhere             192.168.122.126      state NEW,RELATED,ESTABLISHED
ACCEPT     all  --  anywhere             192.168.122.0/24     ctstate RELATED,ESTABLISHED
ACCEPT     all  --  192.168.122.0/24     anywhere            
ACCEPT     all  --  anywhere             anywhere            
REJECT     all  --  anywhere             anywhere             reject-with icmp-port-unreachable
REJECT     all  --  anywhere             anywhere             reject-with icmp-port-unreachable

iptables -L -t nat 
Chain PREROUTING (policy ACCEPT)
target     prot opt source               destination         
DNAT       tcp  --  anywhere             148.59.149.79        tcp dpt:9980 to:192.168.122.126:9980
DNAT       tcp  --  anywhere             148.59.149.79        tcp dpt:12000 to:192.168.122.126:12000
DNAT       tcp  --  anywhere             148.59.149.79        tcp dpt:11000 to:192.168.122.126:11000
DNAT       tcp  --  anywhere             148.59.149.79        tcp dpt:submission to:192.168.122.126:587
DNAT       tcp  --  anywhere             148.59.149.79        tcp dpt:imap2 to:192.168.122.126:143
DNAT       tcp  --  anywhere             148.59.149.79        tcp dpt:smtp to:192.168.122.126:25
DNAT       tcp  --  anywhere             148.59.149.79        tcp dpt:webmin to:192.168.122.126:10000
DNAT       tcp  --  anywhere             148.59.149.79        tcp dpt:4443 to:192.168.122.126:4443
DNAT       udp  --  anywhere             148.59.149.79        udp dpt:10000 to:192.168.122.126:10000
DNAT       tcp  --  anywhere             148.59.149.79        tcp dpt:http to:192.168.122.126:80
DNAT       tcp  --  anywhere             148.59.149.79        tcp dpt:https to:192.168.122.126:443
34
0 + 0
Martin avatar
kz flag
Martin
If I understood you correctly, your host system holds the iptables rules, and redirects incoming traffic correctly to your virtual machine. Did you execute the curl commands inside the VM, or on the host system?
0
Reply
gstlouis avatar
bo flag
gstlouis
@Martin I did curl from the guest VM, and not the server that holds iptables and hosting my guests. So my guest it 192.168.122.126 and my server/iptables/vrish is 192.168.122.1
0
Reply
Martin avatar
kz flag
Martin
Ah okay, VM and host are inside the same subnet... can you post your iptables rules, please?
0
Reply
gstlouis avatar
bo flag
gstlouis
@Martin updated original post
0
Reply
Martin avatar
kz flag
Martin
and the VM does not have any firewall rules in place? because if the nat is taking place on the server, I do not see any reason why a curl from within the VM to localhost should fail...
0
Reply
gstlouis avatar
bo flag
gstlouis
@Martin VM does have iptables for fail2ban purposes. FORWARD chain holds nothing, and nat table chains all empty. unless I need to tell iptables on VM something about looping the url public request back to itself, iptables holds nothing custom from its original config, except for extra fail2ban chains. I can post if you think its relevant.
0
Reply
Martin avatar
kz flag
Martin
Let us [continue this discussion in chat](https://chat.stackexchange.com/rooms/136222/discussion-between-martin-and-gstlouis).
0
Reply
gstlouis avatar
bo flag
gstlouis
Already had to leave. Maybe tomorrow? @martin
0
Reply
Martin avatar
kz flag
Martin
sure, I think the INPUT / OUTPUT chains are relevant - maybe fail2ban locked you out...
0
Reply
gstlouis avatar
bo flag
gstlouis
@Martin I have shutdown both f2b and it is still not working. I'm logged into chat, but may not respond quickly
0
Reply
Score:1
Martin avatar Server
kz flag
Martin

The culprit here has been a faulty/incomplete NAT rule. During a TCP connection, each endpoint has fixed source / destination IPs, and if an IP packet is received on this port with a different source / destination IP, the packet gets discarded. This is true for both endpoints: client ( curl ) and the server.

To understand the issue, I will follow the packet flow, starting from the client:

  • curl from within the VM sends a packet to the public IP address. Source IP: 192.168.122.126, Destination IP: 148.59.149.79
  • The kernel of the VM checks its routing table, and sends the ip packet to its default gateway, which is 192.168.122.1
  • the kernel of the host receives the packet and passes it through the iptables chains (remember which chain gets traversed first - PREROUTING goes before POSTROUTING )
  • inside the PREROUTING chain, the destination IP gets replaced with 192.168.122.126
  • here, the POSTROUTING chain would get traversed.
  • because of the destination IP, the packet goes back to the VM
  • at the server socket, the IP packet arrives. Destination AND source IP are 192.168.122.126. No Problem so far.
  • The server socket sends the reply - this time, since the destination IP is its own interface, the NAT rules of the host will NOT get applied.
  • The client socket receives the reply from 192.168.122.126 - which is not the IP, to which the client socket has sent its connection request, therefore the packet gets dropped.

The solution is the change of the source IP address in a way, that the reply from the VM's server socket has to pass the host NAT rules:

iptables -t nat -I POSTROUTING 1 -s 192.168.122.126 -d 192.168.122.126 -j SNAT --to-source 192.168.122.1

Remember, that the SNAT target is only applicable inside the POSTROUTING chain - therefore, the DNAT from the PREROUTING chain already has been applied.

With this rule, the server socket inside the VM receives an incoming connection request from 192.168.122.1 - sends the reply, and the reply will get the SNAT / DNAT rules reverse applied, as soon as it arrives at the host: source to 192.168.122.126, and destination to 148.59.149.79.

This reply matches the initial connection request, and the connection attempt succeeds.

0 + 0
ธงชาติไทย
Elon Musk
I sit in a Tesla and translated this thread with Ai:

mangohost

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.