Detecting 502 responses in Nginx Load Balancer

I have the following setup - Nginx load balancer that receives https traffic and passes through to nodes. On each node there is a reverse proxy that handles https traffic and passes data to App1, App2 in plain text.

--> LB --> RP -> App1, App2
       `-> RP -> App1, App2

Now the issue is that if App1 is down on a node, load balancer doesn't detect that and is happily serving 502 back to the client. I guess it is because reverse proxy is still up and is encrypting the traffic and therefore load balancer simply passes through data. How can I inform load balancer that the App1 on node1 is down and go to the other node?

Simplified LB nginx.conf:

stream {
    map "$ssl_preread_server_name:$server_port" $namehttps {
        hostnames;
        some-address.io:8443 test_site;
    }

    upstream test_site {
        server 192.168.1.10;
        server 192.168.1.11
    }
    server {
        proxy_pass $namehttps;
        ssl_preread on;
    }
}

Reverse proxy nginx.conf acts as a standard reverse proxy terminating ssl traffic and passing un-encrypted traffic to an app.

If I correctly understood what you mean, you can use active health checks, which are basically periodic HTTP requests.

As per the NGINX documentation (https://docs.nginx.com/nginx/admin-guide/load-balancer/http-health-check/), which I'm going to summarize here in case the page goes 404, you have to:

• define in the server portion of the configuration a location / (or similar path) section;

• include the proxy_pass and health_check directive in the location / (or similar) section;

• in the http upstream server group, define a shared memory by adding zone backend 64k (or similar value).

However, this doesn't work if you're using a stream block, because streams work at the TCP/UDP layer. The upstream must be contained in a http block, so you can benefit from the higher-layer features of the http module.

The problem is that you are using stream and should not be.

nginx has no visibility to the connection payload when using stream proxies. It has no way to even know that there is HTTP in there, nor does stream care anything about the payload.

You will need to use "a standard reverse proxy terminating ssl traffic".