Join Log in
Score:0
Peter C. Glade avatar Server

Route traffic to Kubernetes cluster using iptables

ug flag
Peter C. Glade

I have a kubernetes cluster (currently only 1 node) running on my root server (netcup) and try to route traffic to it. Im pretty bad in networking so hopefully someone can help me.

On the cluster metallb is installed and calico as CNI. Metallb is responsible for assignment and announcement of IP addresses to K8s-LoadBalancers which works fine internally. So i can reach a load balanced service from within the server. I only expose a single service with a fixed internal IP (reverse-proxy)

Now I want to expose it to the outside by routing traffic directly to the IP address from metallb using iptables

The following rules were applied in iptables:

iptables -A PREROUTING -t nat -p tcp -m tcp -j DNAT --to-destination <metallb-ip> -i eth0 --destination-port 80 -m comment --comment Redirect web traffic to cluster (80)
iptables -A PREROUTING -t nat -p tcp -m tcp -j DNAT --to-destination <metallb-ip> -i eth0 --destination-port 443 -m comment --comment Redirect web traffic to cluster (443)
iptables -A POSTROUTING -t nat -p tcp -m tcp -j SNAT --to-source <public-server-ip> -o eth0 --destination-port 80 -m comment --comment Redirect web traffic from cluster (80)
iptables -A POSTROUTING -t nat -p tcp -m tcp -j SNAT --to-source <public-server-ip> -o eth0 --destination-port 443 -m comment --comment Redirect web traffic from cluster (443)

(maybe the commands are not 100% correct because i extracted them from my ansible script. You can find the ansible part at the bottom)

As far as i understand this should send all incoming TCP traffic to the internal IP using DNAT and all outgoing traffic will have the public server address using SNAT. So for a connecting client it should look like he is talking directly to the public IP/Port?

Unfortunately, trying to access the service from the Internet results in the following:

$> curl <public-server-ip>
curl: (7) Failed to connect to <public-server-ip> port 80 after 647 ms: Network is down

Access from the server works:

$> curl <metallb-ip>
404 page not found # which the expected answer since its a blank Traefik reverse proxy

An nmap port scan currently throws the following:

$> nmap -Pn -p 80,443 <public-server-ip>
Starting Nmap 7.93 ( https://nmap.org ) at 2022-12-15 15:57 CET
Strange SO_ERROR from connection to <public-server-ip> (50 - 'Network is down') -- bailing scan
QUITTING!

But it is up and running since im connected via SSH.

There is currently no ufw enabled.

I appreciate any help since as I said I am not very good at networking. ;)

Ansible script:

- name: DNAT port 80 to cluster
  iptables:
    table: nat
    chain: PREROUTING
    in_interface: eth0
    protocol: tcp
    match: tcp
    jump: DNAT # REDIRECT
    to_destination: <metallb-ip>
    destination_port: 80
    comment: Redirect web traffic to cluster (80)
  become: yes

- name: DNAT port 443 to cluster
  iptables:
    table: nat
    chain: PREROUTING
    in_interface: eth0
    protocol: tcp
    match: tcp
    destination_port: 443
    jump: DNAT # REDIRECT
    to_destination: <metallb-ip>
    comment: Redirect web traffic to cluster (443)
  become: yes 

- name: SNAT port 80 from cluster
  iptables:
    table: nat
    chain: POSTROUTING
    out_interface: eth0
    protocol: tcp
    match: tcp
    destination_port: 80
    jump: SNAT # REDIRECT
    to_source: <public-server-ip>
    comment: Redirect web traffic from cluster (443)
  become: yes

- name: SNAT port 443 from cluster
  iptables:
    table: nat
    chain: POSTROUTING
    out_interface: eth0
    protocol: tcp
    match: tcp
    destination_port: 443
    jump: SNAT # REDIRECT
    to_source: <public-server-ip>
    comment: Redirect web traffic from cluster (443)
  become: yes

- name: Configure IP Masquerading
  iptables: 
    table: nat
    chain: POSTROUTING
    out_interface: eth0
    jump: MASQUERADE

UPDATE: With the help of this article i tried to understand the flow. A added logging statements to investigate where thing are going wrong. That's what I found so far:

To check where packets get lost I added the following rules:

iptables -t raw -A PREROUTING -p tcp --dport 80 -j LOG --log-prefix "[raw:PREROUTING]: "
iptables -t mangle -A PREROUTING -p tcp --dport 80 -j LOG --log-prefix "[mangle:PREROUTING]: "
iptables -t nat -A PREROUTING -p tcp --dport 80 -j LOG --log-prefix "[nat:PREROUTING]: "
iptables -t mangle -A POSTROUTING -p tcp --sport 80 -j LOG --log-prefix "[mangle:PREROUTING]: "
iptables -t nat -A POSTROUTING -p tcp --sport 80 -j LOG --log-prefix "[nat:PREROUTING]: "

Under /var/log/kern.log just the raw and mangle log statements appearing. So it seems that the nat-step is not triggered. Checking the mangle rules with: sudo iptables -t mangle -L but couldn't identify any issue here.

IP-Forward is enabled:

$> sudo sysctl net.ipv4.ip_forward
net.ipv4.ip_forward = 1

So why the packets don't come to the nat-step?

$> sudo iptables -t mangle -L
Chain PREROUTING (policy ACCEPT)
target     prot opt source               destination
cali-PREROUTING  all  --  anywhere             anywhere             /* cali:6gwbT8clXdHdC1b1 */
LOG        tcp  --  anywhere             anywhere             tcp dpt:http LOG level warning prefix "[mangle:PREROUTING]: "

Chain INPUT (policy ACCEPT)
target     prot opt source               destination

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination

Chain POSTROUTING (policy ACCEPT)
target     prot opt source               destination
cali-POSTROUTING  all  --  anywhere             anywhere             /* cali:O3lYWMrLQYEMJtB5 */

Chain KUBE-IPTABLES-HINT (0 references)
target     prot opt source               destination

Chain KUBE-KUBELET-CANARY (0 references)
target     prot opt source               destination

Chain KUBE-PROXY-CANARY (0 references)
target     prot opt source               destination

Chain cali-from-host-endpoint (1 references)
target     prot opt source               destination

Chain cali-to-host-endpoint (1 references)
target     prot opt source               destination

Chain cali-PREROUTING (1 references)
target     prot opt source               destination
ACCEPT     all  --  anywhere             anywhere             /* cali:6BJqBjBC7crtA-7- */ ctstate RELATED,ESTABLISHED
ACCEPT     all  --  anywhere             anywhere             /* cali:KX7AGNd6rMcDUai6 */ mark match 0x10000/0x10000
cali-from-host-endpoint  all  --  anywhere             anywhere             /* cali:wNH7KsA3ILKJBsY9 */
ACCEPT     all  --  anywhere             anywhere             /* cali:Cg96MgVuoPm7UMRo */ /* Host endpoint policy accepted packet. */ mark match 0x10000/0x10000

Chain cali-POSTROUTING (1 references)
target     prot opt source               destination
RETURN     all  --  anywhere             anywhere             /* cali:NX-7roTexQ3fGRfU */ mark match 0x10000/0x10000
MARK       all  --  anywhere             anywhere             /* cali:nnqPh8lh2VOogSzX */ MARK and 0xfff0ffff
cali-to-host-endpoint  all  --  anywhere             anywhere             /* cali:nquN8Jw8Tz72pcBW */ ctstate DNAT
RETURN     all  --  anywhere             anywhere             /* cali:jWrgvDQ0xEZHmta3 */ /* Host endpoint policy accepted packet. */ mark match 0x10000/0x10000
453
0 + 7
nat
Ron Maupin avatar
us flag
Ron Maupin
NAT is not routing.
0
Reply
Peter C. Glade avatar
ug flag
Peter C. Glade
@RonMaupin: Thanks for clarifying, wasn't really aware of the differences. I have a IP in a server-internal network which I want to be accessible from the outside, wouldn't this be the right use case for NATing? In this case only my wording is misleading. If you have any other ideas, please let me know ;)
0
Reply
Ron Maupin avatar
us flag
Ron Maupin
You only NAT where you must: public<->private or overlapping addressing. NAT is not a substitute for routing, it is a kludge to extend the life of IPv4 until IPv6 id ubiquitous. It take more resource and is slower than simple routing.
0
Reply
Peter C. Glade avatar
ug flag
Peter C. Glade
Ok, I see. But any time i hit google like "route traffic to server internal ip address" I end up in a thread which explains how to set set NAT rules with `iptables`. What's the correct way routing the traffic to my internal IP?
0
Reply
Ron Maupin avatar
us flag
Ron Maupin
You can enable routing on the host. The WAN router will need to know how to reach that network. There are three ways a router learns how to reach a network: directly connected, statically configured, or dynamically with a routing protocol. For something simple, statically configuring a route for that network pointing to the host should suffice, but static routing does not scale, so larger situations should use a routing protocol.
0
Reply
Peter C. Glade avatar
ug flag
Peter C. Glade
But this is only possible if I can manage the WAN router? Bc. everything runs on a root server from my hosting provider (netcup). This servers has a static ip address, of cause. On this server there is a bare-metal kubernetes instance running, which per design does not listen on port 80/443. With metallb i can provide load balancing and ip provisioning locally. It assigns internal IP addresses which routes to the kubernetes services within the cluster. And now I "just have to" route all traffic to this internal IP. Im do not understand how the WAN router should help here?
0
Reply
Peter C. Glade avatar
ug flag
Peter C. Glade
Sometimes it's so simple. Thanks to the comments from @RonMaupin i noticed that I was completely on the wrong track. I ordered an additional IP address from my hoster and assigned it to the server. This IP is assigned with metallb and correctly routes to the reverse proxy. At least i learned a lot about networking ^^
0
Reply
ธงชาติไทย
Elon Musk
I sit in a Tesla and translated this thread with Ai:

mangohost

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.