I am attempting to make a local request to the kubernetes cluster that is hosted on my server, the cluster's NodePort is listening at the following address 172.20.120.1:30280. External client in production are required to make requests to 172.20.0.1:8000 (this cannot change), so I am attempting to add a DNAT rule to nat the traffic from:
172.20.0.1:8000 -> 172.20.120.1:30280 (k8s NodePort)
I added the following DNAT PREROUTING rule:
Chain PREROUTING (policy ACCEPT 2614 packets, 170K bytes)
num pkts bytes target prot opt in out source destination
...
26 27462 1648K DNAT tcp -- * * 0.0.0.0/0 172.20.0.1 tcp dpt:8000 to:172.20.120.1:30280
and the following OUTPUT rule:
# iptables -v -L OUTPUT -n --line-numbers | head -10
Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes)
num pkts bytes target prot opt in out source destination
...
4 943 156K tcp -- * * 0.0.0.0/0 172.20.120.1 tcp dpt:30280
I am able to make curl request to 172.20.120.1:30280 directly and get a successful response back. However, when I make a curl request to 172.20.0.1:8000 it just hangs with the following message:
# curl -vvvk https://172.20.0.1:8000/v1/my-api
* About to connect() to 172.20.0.1 port 8000 (#0)
* Trying 172.20.0.1...
* Connected to 172.20.0.1 (172.20.0.1) port 8000 (#0)
* Initializing NSS with certpath: sql:/etc/pki/nssdb
And then it eventually times out.
My tcpdump show that the traffic is getting forwarded to that correct IP when trigger the curl request to 172.20.0.1:8000:
# tcpdump -nnvvv -i any src 172.20.0.1 and dst 172.20.120.1
tcpdump: listening on any, link-type LINUX_SLL (Linux cooked), capture size 262144 bytes
08:47:07.108364 IP (tos 0x0, ttl 64, id 27172, offset 0, flags [DF], proto TCP (6), length 60)
172.20.0.1.52910 > 172.20.120.1.30280: Flags [S], cksum 0xd85d (incorrect -> 0x84e2), seq 1825443523, win 43690, options [mss 65495,sackOK,TS val 549020903 ecr 0,nop,wscale 7], length 0
08:47:07.108771 IP (tos 0x0, ttl 64, id 27173, offset 0, flags [DF], proto TCP (6), length 52)
172.20.0.1.52910 > 172.20.120.1.30280: Flags [.], cksum 0xd855 (incorrect -> 0x3029), seq 1825443524, ack 3881482736, win 342, options [nop,nop,TS val 549020904 ecr 549020904], length 0
08:47:07.206994 IP (tos 0x0, ttl 64, id 27174, offset 0, flags [DF], proto TCP (6), length 229)
...
Also I have added TRACE to the iptable rules and I see stuff outputted when making the curl to 172.20.0.1:8000:
Mar 25 09:01:38.570 x kernel: TRACE: raw:PREROUTING:policy:4 IN=lo OUT= MAC=00:00:00:00:00:00:00:00:00:00:00:00:08:00 SRC=172.20.0.1 DST=172.20.120.1 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=44467 DF PROTO=TCP SPT=40096 DPT=30280 SEQ=191700158 ACK=0 WINDOW=43690 RES=0x00 SYN URGP=0 OPT (0204FFD70402080A20C6B10D0000000001030307)
Mar 25 09:01:38.570 x kernel: TRACE: mangle:PREROUTING:rule:1 IN=lo OUT= MAC=00:00:00:00:00:00:00:00:00:00:00:00:08:00 SRC=172.20.0.1 DST=172.20.120.1 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=44467 DF PROTO=TCP SPT=40096 DPT=30280 SEQ=191700158 ACK=0 WINDOW=43690 RES=0x00 SYN URGP=0 OPT (0204FFD70402080A20C6B10D0000000001030307)
Mar 25 09:01:38.570 x kernel: TRACE: mangle:cali-PREROUTING:rule:3 IN=lo OUT= MAC=00:00:00:00:00:00:00:00:00:00:00:00:08:00 SRC=172.20.0.1 DST=172.20.120.1 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=44467 DF PROTO=TCP SPT=40096 DPT=30280 SEQ=191700158 ACK=0 WINDOW=43690 RES=0x00 SYN URGP=0 OPT (0204FFD70402080A20C6B10D0000000001030307)
Mar 25 09:01:38.570 x kernel: TRACE: mangle:cali-from-host-endpoint:return:1 IN=lo OUT= MAC=00:00:00:00:00:00:00:00:00:00:00:00:08:00 SRC=172.20.0.1 DST=172.20.120.1 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=44467 DF PROTO=TCP SPT=40096 DPT=30280 SEQ=191700158 ACK=0 WINDOW=43690 RES=0x00 SYN URGP=0 OPT (0204FFD70402080A20C6B10D0000000001030307)
And when I make a request directly to 172.20.120.1:30280 it works and I get a successful response back, so I just can't wrap my head around why the DNAT rule doesn't work.
I also tried opening up the firewall to ACCEPT all but that didn't work either.
iptables -P INPUT ACCEPT
iptables -P OUTPUT ACCEPT
I can also see the OUTPUT rule packet size increase when making the curl request to 172.20.0.1:8000 so I know that rule is getting hit.
Does anyone know why I can't curl 172.20.0.1:8000 and get a successful response back but when curl 172.20.120.1:30280 directly it works fine?
