Since Linux kernel 4.17, there's no need to use iptables at all: the routing stack can directly choose a different route depending on port without having it marked, rerouted and requiring a NAT band-aid (but usually only for UDP or for TCP as client, not for TCP as server like here):
With the nginx server running on the VPN client, assuming there's no iptables ruleset and no additional routing rule yet in place...
Add the alternate route in the routing table, preceded by the LAN route before (it will at least prevent connections from LAN to uselessly go through the router which will have to emit ICMP redirects). This table will be flushed if the interface goes down then up, so this should be integrated with the tool managing the interface:
ip route add 192.168.1.0/24 dev eth0 table 80
ip route add default via 192.168.1.1 dev eth0 table 80
Choose locally initiated (that's the special role of iif lo
here) reply traffic from port 80 (to anywhere) to use this table:
ip rule add iif lo ipproto tcp sport 80 lookup 80
Same for incoming HTTP traffic, ie the original direction (not really needed since SRPF won't be enabled, but for consistency anyway):
ip rule add iif eth0 ipproto tcp dport 80 lookup 80
Loosen SRPF in case it was enabled because I'm not certain TCP related traffic such as ICMP errors during PMTUD might be accepted in case it was set as strict:
sysctl -w net.ipv4.conf.eth0.rp_filter=2
HTTP/3 support (which uses QUIC and is thus over UDP) just requires the same rules for UDP:
ip rule add iif lo ipproto udp sport 80 lookup 80
ip rule add iif eth0 ipproto udp dport 80 lookup 80
Should some IP addresses be made an exception to this exception on port 80, one method among others is to add them in the routing table as not matching using the throw keyword, thus falling through to the main routing table. For example to have traffic from 10.8.0.0/24 which is guaranteed to be OpenVPN traffic, to still work with nginx:
throw
- a special control route used together with policy rules. If
such a route is selected, lookup in this table is terminated
pretending that no route was found. Without policy routing it is
equivalent to the absence of the route in the routing table. The
packets are dropped and the ICMP message net unreachable is generated.
The local senders get an ENETUNREACH error.
ip route add throw 10.8.0.0/24 table 80