Rolling updates of backend with udp server sockets

I have a backend system used for IOT devices which use UDP protocol for communications. And there are certain TCP (HTTP2) based APIs for mobile apps from the same backend.

I am trying to build a rolling update feature to enable 0 downtime patching of the backend services.

My setup is like this.

Instead of directly exposing the sockets to the apps, I am trying to do transparent proxy to my processes. I have exposed 2 sockets (1 udp and 1 tcp) to the public internet using firewall.

My production server is opening different set of ports for udp and tcp (which are changeable via environment variables without changing the underlying binary).

Step 1:

I am trying to create transparent proxy in the same machine from udp port 16002 to 17002. For udp, my server will also initiate some communication with the devices in the wild. Server should see the source IP/port as well as communicate back with these devices which could be under some NATs (typically, WiFi router) by respecting same origin policy of the NATs.

And same for tcp. From port 16012 to port 17012. This is the typical deployment by externalizing the real ports.

I am not able to make this work.

Step 2:

Whenever there is new code to be patched, I want to bring up the new code on two different set of ports as shown in the below picture (P2 - Process 2).

When process 2 is up and running, I will change the IP address mapping to the new Process (P2). After giving certain time for P1 to finish off any pending IO operations, we will bring down the Process P1.

For the next patch, we will bring up P1' and the process inverses.

Is there any flaw in this design? Can this be achieved technically by using iptables and tproxy or any other linux tools?

I have considered building an L7 router and relaying the packets back and forth by defining high level objects. But I am curious if this can be done using low level routing L3/L4 since it can be more efficient and battle tested. For sure, these nft, iptables tools have usability issues and not very intuitive, especially nft, for a developer.

You can do this with iptables. To create a port forwarding from 17X to 16X:

iptables -A PREROUTING -t nat -p tcp -m tcp --dport 16012 -j DNAT --to-destination :17012
iptables -A PREROUTING -t nat -p udp -m udp --dport 16002 -j DNAT --to-destination :17002

Then when you want to switch which port it is pointing to after you have started up the updated version of your app:

iptables -D PREROUTING -t nat -p tcp -m tcp --dport 16012 -j DNAT --to-destination :17012
iptables -D PREROUTING -t nat -p udp -m udp --dport 16002 -j DNAT --to-destination :17002
iptables -A PREROUTING -t nat -p tcp -m tcp --dport 16012 -j DNAT --to-destination :18012
iptables -A PREROUTING -t nat -p udp -m udp --dport 16002 -j DNAT --to-destination :18002

If you are on a distro using an iptables.service and you want these changes to be permanent, you can add the -A lines to /etc/sysconfig/iptables or wherever your config file is located. If you are using a distro that makes use of a firewalld.service then you can achieve the same goal and have it be permanent across reboots this way:

This first line is only needed once to enable masquerading in general

firewall-cmd --permanent --add-masquerade

Then to create your initial forwardings:

firewall-cmd --permanent --add-forward-port=port=16002:proto=udp:toport=17002
firewall-cmd --permanent --add-forward-port=port=16012:proto=tcp:toport=17012

When you want to change the application to listening on a different port, just run:

firewall-cmd --permanent --remove-forward-port=port=16002:proto=udp:toport=17002
firewall-cmd --permanent --remove-forward-port=port=16012:proto=tcp:toport=17012
firewall-cmd --permanent --add-forward-port=port=16002:proto=udp:toport=18002
firewall-cmd --permanent --add-forward-port=port=16012:proto=tcp:toport=18012