Score:0

After some time Nginx reverse proxy stops functioning with SSL errors to upstream until configuration is reloaded

cn flag

I have a Nginx reverse proxy, which mysteriously stops functioning after some hours. Error log gets filled with SSL errors connecting to upstream. Runing "nginx -s reload" (restarting nginx service) fixes the issue until next failure. Any ideas where to dig further?

Config files does not change, there are no any other activity in logs. Cannot pinpoint "time to fail", sometimes it seems to be 3-4 hours, sometimes somewhere around 8.

Application is some third-party Java Swing based dynamic web app from the legacy enterprise horror cellar... :)

Snippet for one of a bunch of the Nginx config Locations for the specific URI location. Cannot use one / location as one of the location needs MTLS authorization and others simply has to rely on whitelist.

location ~* ^/app-location(/?|/.*|-.*)$ {
  include /etc/nginx/conf.d/include.APP.whitelist;
  proxy_pass https://__APP_LB_DNS__:6443;
  proxy_ssl_server_name on;
  proxy_ssl_name __APP_LB_DNS__;
  proxy_ssl_trusted_certificate __certificate-backend-ca-location__;
  proxy_ssl_verify       off;
  proxy_ssl_verify_depth 3;
  proxy_ssl_session_reuse on;
  proxy_ssl_certificate     __client-certificate-location__;
  proxy_ssl_certificate_key __client-certificate-key-location__;
}

After some time I start to get 502: Bad Gateway errors and error.log is filled with such errors. 10.0.0.10 is the upstream load balancer IP for APP_LB_DNS hostname.

2022/02/09 21:36:45 [error] 12371#12371: *59560 SSL_do_handshake() failed (SSL: error:14094410:SSL routines:ssl3_read_bytes:sslv3 alert handshake failure:SSL alert number 40) while SSL handshaking to upstream, client: CLIENT_IP, server: EXTERNAL_DNS_NAME, request: "GET /app-location/ HTTP/1.1", upstream: "https://10.0.0.10:6443/app-location/", host: "EXTERNAL_DNS_NAME", referrer: "http://CLIENT_DNS:53998/"

I have had to add options to ensure Nginx is communicating with Azure backend LB via its DNS name. Otherwise I was getting errors above constantly. Another thing that puzzles me is that it is only this weird app that requires such options. Other in house written apps does work well without those lines.

proxy_ssl_server_name on;
proxy_ssl_name __APP-DNS__;

Server is running Nginx on Centos 7 in an Azure VM nginx-1.18.0-2.el7.ngx.x86_64 nginx-modsecurity3-centos7-3.0.4-1.x86_64

mangohost

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.