curl: (60) server certificate verification failed CRLfile: none

I'm slowly transitioning from an exclusive developer role and into more of a hybrid DevOps role at my company. Which means I'm new to a lot of this, please go easy on me... :-p

My client's server is running Ubuntu 16.04, with PHP 5.6.4 and there is a function in their site's administrative portal that runs a curl command (essentially) back to itself for some sort of file syncing. And it's been failing for some time (a few weeks/months). The problem (I think) is that certificate verification is failing, and thus, the function is dying on the vine.

When I ssh into the server, I can easily curl out to anywhere with no issues (Google, example.org, etc...). But trying to just a basic curl to the site's main url borks.

$ curl -v https://www.[my-site-name].com

*   Trying [my-site-IP]...
* Connected to [my-site-name] ([my-site-IP]) port 443 (#0)
* found 258 certificates in /etc/ssl/certs/ca-certificates.crt
* found 908 certificates in /etc/ssl/certs/
* ALPN, offering http/1.1
* SSL connection using TLS1.2 / ECDHE_RSA_AES_128_GCM_SHA256
* server certificate verification failed. CAfile: /etc/ssl/certs/ca-certificates.crt CRLfile: none
* Closing connection 0
curl: (60) server certificate verification failed. CAfile: /etc/ssl/certs/ca-certificates.crt CRLfile: none
More details here: http://curl.haxx.se/docs/sslcerts.html

curl performs SSL certificate verification by default, using a "bundle"
 of Certificate Authority (CA) public keys (CA certs). If the default
 bundle file isn't adequate, you can specify an alternate file
 using the --cacert option.
If this HTTPS server uses a certificate signed by a CA represented in
 the bundle, the certificate verification probably failed due to a
 problem with the certificate (it might be expired, or the name might
 not match the domain name in the URL).
If you'd like to turn off curl's verification of the certificate, use
 the -k (or --insecure) option.

I know that I can run curl with -k for it to be insecure, but I'm hesitant to do so. I guess the first question is, should I not worry about running this with the insecure flag since it's technically not leaving the server at all? I've tested this exact same curl command on one of our newer boxes running Ubuntu 18.x and also on a DigitalOcean running v20 with no issues at all -- both external and internal curls worked great.

I can even be on another server, and curl back to my one experiencing issues, and that is working fine too.

I've tried everything I could think of (which admittedly isn't much) and nothing seems to be working.

• run updates for curl and certbot packages
• forcing the update-ca-certificates
• added /etc/ssl/certs/cacert.pem to both the curl.cainfo and openssl.cafile vars in php.ini

I know this probably doesn't matter, but just for completeness, I've also run the site through various online verification services:

• https://www.ssllabs.com/ssltest
• https://chainchecker.certifytheweb.com
• https://decoder.link/sslchecker

All came back with positive results. The only negative (I guess) is that SSLLabs graded us with a 'B' because apparently TLS 1.0 is still enabled.

Any help would be greatly appreciated. I feel like reading the docs mentioned in the failure warning isn't really all that helpful.

Suggestions / tips / tricks ??

1,000 thank you's in advance!

Meta: not an answer, at least not yet, but some info and advice.

Okay, easy case doesn't apply. As you've encountered, curl doesn't give very helpful information when it gets a cert-verify error from the underlying SSL/TLS stack, at least when using GnuTLS as the Ubuntu build does. However, OpenSSL (which should be present because curl has it as a dependency, though I don't see why) provides a commandline program named simply openssl that is mostly a test tool, and is good deal more verbose as well as more error-tolerant. For example, on an intentionally bad site, in a docker version of Ubuntu 16.04:

# curl -v https://untrusted-root.badssl.com/
*   Trying 104.154.89.105...
* Connected to untrusted-root.badssl.com (104.154.89.105) port 443 (#0)
* found 129 certificates in /etc/ssl/certs/ca-certificates.crt
* found 516 certificates in /etc/ssl/certs
* ALPN, offering http/1.1
* SSL connection using TLS1.2 / ECDHE_RSA_AES_128_GCM_SHA256
* server certificate verification failed. CAfile: /etc/ssl/certs/ca-certificates.crt CRLfile: none
* Closing connection 0
curl: (60) server certificate verification failed. CAfile: /etc/ssl/certs/ca-certificates.crt CRLfile: none
More details here: http://curl.haxx.se/docs/sslcerts.html

[snip canned text -- same as yours]
# openssl s_client -connect untrusted-root.badssl.com:443 -servername untrusted-root.badssl.com
CONNECTED(00000003)
depth=1 C = US, ST = California, L = San Francisco, O = BadSSL, CN = BadSSL Untrusted Root Certificate Authority
verify error:num=19:self signed certificate in certificate chain
---
Certificate chain
 0 s:/C=US/ST=California/L=San Francisco/O=BadSSL/CN=*.badssl.com
   i:/C=US/ST=California/L=San Francisco/O=BadSSL/CN=BadSSL Untrusted Root Certificate Authority
 1 s:/C=US/ST=California/L=San Francisco/O=BadSSL/CN=BadSSL Untrusted Root Certificate Authority
   i:/C=US/ST=California/L=San Francisco/O=BadSSL/CN=BadSSL Untrusted Root Certificate Authority
---
Server certificate
-----BEGIN CERTIFICATE-----
MIIEmTCCAoGgAwIBAgIJAMJ1vCpOBAlkMA0GCSqGSIb3DQEBCwUAMIGBMQswCQYD
VQQGEwJVUzETMBEGA1UECAwKQ2FsaWZvcm5pYTEWMBQGA1UEBwwNU2FuIEZyYW5j
aXNjbzEPMA0GA1UECgwGQmFkU1NMMTQwMgYDVQQDDCtCYWRTU0wgVW50cnVzdGVk
IFJvb3QgQ2VydGlmaWNhdGUgQXV0aG9yaXR5MB4XDTIxMTIwNDAwMDgxOVoXDTIz
MTIwNDAwMDgxOVowYjELMAkGA1UEBhMCVVMxEzARBgNVBAgMCkNhbGlmb3JuaWEx
FjAUBgNVBAcMDVNhbiBGcmFuY2lzY28xDzANBgNVBAoMBkJhZFNTTDEVMBMGA1UE
AwwMKi5iYWRzc2wuY29tMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA
wgTs+IzuBMKz2FDVcFjMkxjrXKhoSbAitfmVnrErLHY+bMBLYExM6rK0wA+AtrD5
csmGAvlcQV0TK39xxEu86ZQuUDemZxxhjPZBQsVG0xaHJ5906wqdEVImIXNshEx5
VeTRa+gGPUgVUq2zKNuq/27/YJVKd2s58STRMbbdTcDE/FO5bUKttXz+rvUV0jNI
5yJxx8IUemwo6jdK3+pstXK0flqiFtxpsVdE2woSq97DD0d0XEEi4Zr5G5PmrSIG
KS6xukkcDCeeo/uL90ByAKySCNmMV4RTgQXL5v5rVJhAJ4XHELtzcO9pGEEHRVV8
+WQ/PSzDqXzrkxpMhtHKhQIDAQABozIwMDAJBgNVHRMEAjAAMCMGA1UdEQQcMBqC
DCouYmFkc3NsLmNvbYIKYmFkc3NsLmNvbTANBgkqhkiG9w0BAQsFAAOCAgEADVgB
Ias+fb/Ckdnw5iSYsfRIcfhnTBNgvroorUQe09Psx0tAbjlIYqOCtloNhvCbJYa7
tQQOO65s7RKArpAA1qoapWfDti2aHuMNvGwImwX538RiLf4Rm4MEF6vuF6MZMH/u
Ts1iugB3+d7oSWl/K+RvA4NMRNrlxOLelBJwaTsExsQ5QalpPamongnyWXHZ2Sna
dsw9hBku9ZmlRqYOCE/TajsydqCIhCc2QC5xdd3fxXlcfq5h7G0oOuCYvW7BscTk
AQZYkwS+y3mTHF9wSlxJB4iEGC0NovdM5GsVgfvZ5+jtXGuDlsphAIFLxpIJi1bR
+NsAkEthHoQZDNttvtVJPFPp83PdRDmL6IwrbbXvAwZWWgYmS6HpbF5bxR09JIOA
KttGK1wnd6bh2d9Xy6kfoxZ1gz6i2y5OpxbMoi6z8o1Y2hSOv2kgnD2fxtR9X4OO
wrwWZWnhwsiq7pnZbZiA9GFR4tKZVcJ5ny5aul/MZ0wb5MST7wHW/7qhydfBpOy6
hZ5BSbwfmBao+CJ7NxNJb5c03W5+/Vf1uxXZhodpag6Z5p0rru0v7ea9nMk5dNUm
qR+2XGwzDk9n8jCWwfvSKCa77rf2HqKi8ZaQN/NRp7uVqfY++JVI+h3CLyMk8wTL
FifbLPKbSCW7PAFEfM3wh76VQg1CHpHOPVp/wno=
-----END CERTIFICATE-----
subject=/C=US/ST=California/L=San Francisco/O=BadSSL/CN=*.badssl.com
issuer=/C=US/ST=California/L=San Francisco/O=BadSSL/CN=BadSSL Untrusted Root Certificate Authority
---
No client certificate CA names sent
Peer signing digest: SHA512
Server Temp Key: ECDH, P-256, 256 bits
---
SSL handshake has read 3561 bytes and written 425 bytes
---
New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES128-GCM-SHA256
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
    Protocol  : TLSv1.2
    Cipher    : ECDHE-RSA-AES128-GCM-SHA256
    Session-ID: AC5EB655FAEA1C1A320CA930E6A087FBFE020D3D9C451A72DD7CE77A8080AF0D
    Session-ID-ctx:
    Master-Key: 419065F69901D74454A224DD22F9D459FA85E6EB4D6F043C649C8FDFA2E5E1FD9C168AC270087B88511B0939927F9A0A
    Key-Arg   : None
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    TLS session ticket lifetime hint: 300 (seconds)
    TLS session ticket:
    0000 - cc 91 34 52 b1 ce c6 7c-8a 97 27 b0 b4 50 d6 9d   ..4R...|..'..P..
    0010 - 27 b8 87 13 0b 70 92 e7-23 be 57 5c 00 31 f7 90   '....p..#.W\.1..
    0020 - 65 e3 08 d5 63 14 e9 cd-cc 50 59 3f 2a 98 31 d1   e...c....PY?*.1.
    0030 - ab f8 ca a8 09 aa 66 bf-7c ff b6 66 42 10 8b 6e   ......f.|..fB..n
    0040 - c7 e8 7b f3 2b 35 b3 76-8c 95 b3 b5 37 85 09 42   ..{.+5.v....7..B
    0050 - 83 a9 c3 f7 54 f4 c5 f5-b2 0d d5 fa c6 24 a6 e2   ....T........$..
    0060 - fb 03 cf 35 9c 0d cc 48-e0 bc fc 43 11 3c 19 51   ...5...H...C.<.Q
    0070 - 0a 23 7d b2 6c ff 9b e2-bc 64 1d 74 34 cb 13 7b   .#}.l....d.t4..{
    0080 - c8 55 47 95 71 9d 94 00-33 a0 a0 87 d4 4a fb 81   .UG.q...3....J..
    0090 - 34 36 28 2e ac d7 22 36-36 5f 9c cb 3f 0d e7 00   46(..."66_..?...
    00a0 - e2 b7 d0 35 48 81 2f c7-9d a1 8a f6 52 a3 11 17   ...5H./.....R...
    00b0 - 67 89 94 d3 66 2b 5c c4-73 c5 72 1e 84 46 57 a5   g...f+\.s.r..FW.
    00c0 - 77 05 bc 49 a3 1a fe e6-da a8 0f 40 e2 17 f0 40   w..I.......@...@

    Start Time: 1644060097
    Timeout   : 300 (sec)
    Verify return code: 19 (self signed certificate in certificate chain)
---
Q
DONE

Observe that the first output after the CONNECTED line: depth=1 ... / verify error:num=19:.... indicates the cert verification failed; however openssl ignores this error and completes the handshake, resulting in the last section of output: SSL-Session: / Protocol: (valid) / Cipher: (valid) / ... / Verify return code: ... after which you must type Q and return or control-D (alone) to exit the program, or control-C to kill, unless you add </dev/null on the commandline which has the same effect as automatically typing control-D. For now you don't need to understand the specific values of Protocol and Cipher, just that they are NOT something like 'none' or 'error' or 'missing'. But be warned that OpenSSL's message text for verify=19 is incomplete to the point of being confusing; it is perfectly normal in general for the cert chain from an SSL/TLS server to include the root cert which is self-signed, and if that root is in the local truststore OpenSSL does (and curl/GnuTLS certainly should) accept such a chain. It is only if the root is in the chain AND NOT locally trusted that you get this verify code. Other verify codes like 'expired' have mostly clearer messages.

Also note I used -servername with the host name; this invokes SNI=ServerNameIndication which many SSL/TLS servers today, including badssl.com, need in order to serve the correct certificate(s). However, the version of curl in Ubuntu16.04 does NOT send SNI; without research, I don't know if this is due to the slightly (but not very) older version of curl, the use of GnuTLS, or just a mistake in the build. But anyway as a test try the openssl s_client command against your server both with and without the -servername host part, and save the output (with redirection or tee, preferably including stderr with &> or |& tee or equivalent) because we may need them. If the with-SNI version works and the without-SNI version gets a verify error, that's your problem -- lack of SNI. (It may be difficult to fix, but at least you know what it is.)

Otherwise, look at the output for without-SNI. If OpenSSL reports a verify error (agreeing with curl/GnuTLS), look at the related message -- it should at least point in the direction of the problem, if not exactly to it. If OpenSSL reports no verify error (where curl/GnuTLS does, both using the same system-supplied truststore in /etc/ssl) it's going to be more complicated, so I'll leave that for now and add later if needed.