Score:2

How to troubleshoot ERR_SSL_VERSION_OR_CIPHER_MISMATCH on server with nginx reverse proxy and let's encrypt?

to flag

I have a server (Debian 9.13) with several websites running in Docker containers with nginx (1.13.12) as a reverse proxy.

The websites are under two domain names : alchimie-web.com and lesamisdelachesnaie.fr

I have been using Let's encrypt certbot (with the Docker image) to issue and renew the certificates and it's been working just fine.

The certificates for alchimie-web.com were renewed on December 31, and since then it's impossible to connect to the services on alchimie-web.com (like www.alchimie-web.com).

With Chrome, I get ERR_SSL_VERSION_OR_CIPHER_MISMATCH, and with curl I get :

# curl "https://www.alchimie-web.com"
curl: (35) error:14077410:SSL routines:SSL23_GET_SERVER_HELLO:sslv3 alert handshake failure

The certificates for lesamisdelachesnaie.fr were renewed on December 1 and the websites are still working. But they were issued with a former version of the certbot Docker image. I have updated it to the latest one in between.

The SSL settings for nginx are set globally in the http section.

Here's the relevant snippet :

   # global SSL configuration
    ssl_certificate         /etc/nginx/ssl/fullchain.pem;
    ssl_certificate_key     /etc/nginx/ssl/privkey.pem;
    ssl_trusted_certificate   /etc/nginx/ssl/chain.pem;

    ssl_protocols           TLSv1.2 TLSv1.3; # TLSv1 TLSv1.1 

    # Diffie-Hellman
    ssl_ecdh_curve          secp384r1;

    # OCSP Stapling
    resolver 80.67.169.12 80.67.169.40 valid=300s;
    ssl_stapling on;
    ssl_stapling_verify on;
    ssl_verify_depth        3;

    ssl_ciphers EECDH+AESGCM:EECDH+CHACHA20:EECDH+AES;
    ssl_prefer_server_ciphers   on;

    # SSL cache
    ssl_session_cache   shared:SSL:10m;
    ssl_session_timeout 5m;
    ssl_session_tickets off;

I tried commenting out the ssl_ciphers directive and turning off ssl_prefer_server_ciphers but it doesn't change anything.

I get no useful information from ssllabs.com :

Assessment failed: Failed to communicate with the secure server

Any hint on what's wrong and/or where I should look to troubleshoot this issue ?

Edit : things I've tried based on comments from @steffenullrich, @lexli and @vidarlo

  • check the logs : nothing particular in nginx logs when I try to connect from Chrome, or curl or openssl s_client
    however, when I scan with cipherscan (nice tool, thanks @lexli), I get errors in the nginx log :
    • SSL_do_handshake() failed (SSL: error:1417D18C:SSL routines:tls_process_client_hello:version too low) while SSL handshaking
    • SSL_do_handshake() failed (SSL: error:1417D102:SSL routines:tls_process_client_hello:unsupported protocol) while SSL handshaking
    • cipherscan concludes Host does not seem to support SSL or TLS protocol
  • move the ssl certificate directive to the specific sites (server{...}) section (with an include directive) : it makes no difference
  • check the configuration and certificate paths : nginx -t says that syntax is correct and the test passed. I checked that it would throw an error with a wrong path or filename for the certificate files.
  • renew the certificate : no effect

I updated nginx to the latest Docker image (1.23.3) and it did change something, I'll dig into that...

Steffen Ullrich avatar
se flag
Please check the servers error log.
Lex Li avatar
vn flag
I will use a tool like Wireshark to actually look at the TLS handshake packets. Only that can explain enough details on the error messages.
Steffen Ullrich avatar
se flag
@LexLi: There is not much to see here. It is just a TLS alert from the server in response to a perfectly normal ClientHello. `openssl s_client ... -debug` will you give this information already.
Steffen Ullrich avatar
se flag
*"The SSL settings for nginx are set globally in the http section."* - the certificate should be set in the site specific section, not in the global section. Please compare the configurations of the working and non-working site in this regard.
Lex Li avatar
vn flag
I think `cipherscan` provided more information for your domains, https://github.com/mozilla/cipherscan If the same SSL settings apply to both domains, I will suspect the certificate you got for "alchimie-web.com" was somehow broken or misconfigured.
Score:0
ar flag

I would start with openssl:

$ openssl s_client -connect www.alchimie-web.com:443 -servername www.alchimie-web.com
CONNECTED(00000003)
140257366177600:error:14094410:SSL routines:ssl3_read_bytes:sslv3 alert handshake failure:ssl/record/rec_layer_s3.c:1544:SSL alert number 40
---
no peer certificate available
---
No client certificate CA names sent
---
SSL handshake has read 7 bytes and written 330 bytes
Verification: OK
---
New, (NONE), Cipher is (NONE)
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 0 (ok)
---

The server is for some reason not able to negotiate a connection with the client. This can be because the certificate is corrupt, it has a set of ciphers configured that doesn't overlap with what the client accepts, or similar. The next step would be to verify your certificates, and attempt a renewal even though your certs are not expired.

You should also validate your configuration and that your configured certificate paths is correct.

Steffen Ullrich avatar
se flag
*"This tells us that you don't send any certificates"* - not really. It means that either the server has nothing matching the offerings of the client (ciphers, curves, protocol) or that there is some problem on the server side. Both of these can result in the server not being able to continue, so the only thing the server send back is a TLS alert handshake failure.
vidarlo avatar
ar flag
@SteffenUllrich Thanks for the correction. I still think a valid next point would be to check that the certificate is present - and perform a manual renewal just to rule it out
Steffen Ullrich avatar
se flag
I would suggest to first look at the server error logs. If there is no usable certificate this information should be visible in the logs.
Manumie avatar
to flag
@SteffenUllrich The server logs are quite silent about that ; `nginx -t` says my settings are ok. I did move the sll_certificate settings to the site specific sections, according to your comment above, to match the settings for the working site. @vidarlo I renewed the certificate, to no avail. How could I verify my certificate ? Can I test it somehow to determine what ciphers are used by the certificate ? There might me be a mismatch with the ssl_ciphers directive in nginx.conf ?
Manumie avatar
to flag
@SteffenUllrich about the server logs : what is weird is that the nginx logs (level=warn) don't even show a request when I try to hit the url, either with a browser or `curl` or `openssl s_client` . It seems that the handshake fails without even reaching nginx...
Score:0
to flag

So it was that simple : update nginx...

It seems that the certificates issued by Letsencrypt could not be handled by older version of nginx.

Score:0
ru flag

In my case, I fixed this error by switching Cloudflare DNS from Proxied to DNS only and it worked. Nginx and Certbot were behind Cloudflare.

mangohost

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.