Similarly to this Cloudflare blog post, I am trying to setup Nginx with a transparent socket (with the IP_TRANSPARENT socket option). I want to do this to implement a reverse TCP proxy that effectively binds to all ports.
The IP_TRANSPARENT socket option is not natively supported by Nginx for listening so I am trying to create the socket using a Systemd socket unit and then pass it to the Nginx process using the NGINX environment variable.
The socket unit is as follows (unimportant lines have been removed):
[Socket]
ListenStream=127.0.0.1:1234
Transparent=true
[Install]
WantedBy=sockets.target
My Nginx service unit is as follows (unimportant lines have been removed):
[Service]
# Configure Nginx to use the socket inherited from Systemd
Environment=NGINX=3;
ExecStart=/usr/sbin/nginx -c /etc/nginx/nginx.conf
And my nginx.conf is as follows (unimportant lines have been removed and upstream is the hostname of the upstream host I want to proxy to):
stream {
server {
proxy_pass upstream:443
listen 127.0.0.1:1234;
}
}
Nginx is running on a GCP compute instance behind an internal passthrough load balancer with IP address 10.11.12.13 so I have added the following iptables rule to redirect traffic to the transparent socket:
iptables -t mangle -I PREROUTING -d 10.11.12.13/32 -p tcp -j TPROXY --on-port=1234 --on-ip=127.0.0.1
As described here, GCP installs the local route for the load balancer, so there is no need to do that myself. In other words, the following command returns RTNETLINK answers: File exists:
ip route add local 10.11.12.13/32 dev lo src 127.0.0.1
However, the following curl command hangs:
curl https://10.11.12.13
The TCP connection is being established correctly from the perspective of curl, however when stracing the Nginx worker process it seems to be hanging on the accept4 system call.
Running tcpdump from the instance that Nginx is running on shows the following:
15.59.51.564902 IP [REDACTED].50028 > 10.11.12.13.443: Flags [S], seq 3519046460, win 65535, options [mss 1320,nop,wscale 6,nop,nop,TS val 3905128064 ecr 0,sackOK,eol], length 0
15.59.51.564973 IP 10.11.12.13.443 > [REDACTED].50028: Flags [S.], seq 2512605282, ack 3519046461, win 28400, options [mss 1420,nop,nop,sackOK,nop,wscale 7], length 0
15.59.51.601898 IP [REDACTED].50028 > 10.11.12.13.443: Flags [.], ack 1, win 4096, length 0
15.59.51.601898 IP [REDACTED].50028 > 10.11.12.13.443: Flags [P.], seq 1:370, ack 1, win 4096, length 369
15.59.51.601898 IP 10.11.12.13.443 > [REDACTED].50028: Flags [.], ack 370, win 231, length 0
As I understand, this means that the client is sending data, the server is acknowledging it, but then the client is not sending any more data, presumably because it's expecting a response from the server that it is not receiving. For this reason, I suspect Nginx is just ignoring the data sent by the client.
I hope I have provided enough context to understand the problem. I am not too sure where to go from here or if there is something fundamentally wrong with what I am trying to do. Any help would be greatly appreciated.
