versions
Ubuntu 20.04, OpenSSL 1.1.1f
My Openssl should have the feature (or "bug fix" if you prefer) that the trusted-first option is enabled by default. And this is confirmed:
17:57UTC:root@bard:/etc# openssl verify -help 2>&1| grep trusted_first
-trusted_first search trust store first (default)
The certificates and chain (below) work fine installed in a web server. This cert is installed and both a local curl from the command line and my web browser are happy with the cert and chain files (below).
The problem I'm trying to solve here is that I cannot verify this chain and certificate file using openssl from the command line.
ISRG Root X1 installed?
Yes. Installed, and ca-certificates is happy.
17:57UTC:root@bard:/etc# ls /usr/share/ca-certificates/mozilla/ISRG_Root_X1.crt
-rw-r--r-- 1 root root 1939 Sep 22 11:46 /usr/share/ca-certificates/mozilla/ISRG_Root_X1.crt
17:59UTC:root@bard:/etc# grep ISRG /etc/ca-certificates.conf
mozilla/ISRG_Root_X1.crt
17:59UTC:root@bard:/etc# update-ca-certificates
Updating certificates in /etc/ssl/certs...
0 added, 0 removed; done.
Running hooks in /etc/ca-certificates/update.d...
done.
the cert and chain
Cert is one file, and there's a cert chain file containing two certs.
Here's the cert…
-----BEGIN CERTIFICATE-----
MIIFKDCCBBCgAwIBAgISA1G6nqla/B5CEPZI6YO4Y7eIMA0GCSqGSIb3DQEBCwUA
MDIxCzAJBgNVBAYTAlVTMRYwFAYDVQQKEw1MZXQncyBFbmNyeXB0MQswCQYDVQQD
EwJSMzAeFw0yMTA5MTcwNjE1MjBaFw0yMTEyMTYwNjE1MTlaMBwxGjAYBgNVBAMT
EXdpbmRzb3IuYmxrYnguY29tMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKC
AQEA2vM7VD207G7s04jA5QSG9jmmYCO+xi+4B/nnID66BIA5YJ05ZU6iahuIWDHn
hqfHNW/E+DlouC6MqInuAPOuOMhkQ8sOJMXOwvMXwKbuH5TmkBzieMHcp1X2igAm
/3tGFXzJ4kCLvmm3OOttwAC+fS38F7+pLKQ/yEnNtV/nZbYlHWAJjee1pMc64oHz
jaG7bdXNBvFNFTAk3p9Td61IU0GiF5IG9dDFAFTdV3lYchH9mpKRcuIlupKHchXE
1ZS2MkFX6W4oRdiCT5/lDRACSpLS58LFUTpL/dCOxYAH5xuVHg0teLubiEmygPbT
/k/7JcQBnznBfwCBANJJG15PWQIDAQABo4ICTDCCAkgwDgYDVR0PAQH/BAQDAgWg
MB0GA1UdJQQWMBQGCCsGAQUFBwMBBggrBgEFBQcDAjAMBgNVHRMBAf8EAjAAMB0G
A1UdDgQWBBQTeSF5n3kz5+2Jk8YIS3WMf6EBOTAfBgNVHSMEGDAWgBQULrMXt1hW
y65QCUDmH6+dixTCxjBVBggrBgEFBQcBAQRJMEcwIQYIKwYBBQUHMAGGFWh0dHA6
Ly9yMy5vLmxlbmNyLm9yZzAiBggrBgEFBQcwAoYWaHR0cDovL3IzLmkubGVuY3Iu
b3JnLzAcBgNVHREEFTATghF3aW5kc29yLmJsa2J4LmNvbTBMBgNVHSAERTBDMAgG
BmeBDAECATA3BgsrBgEEAYLfEwEBATAoMCYGCCsGAQUFBwIBFhpodHRwOi8vY3Bz
LmxldHNlbmNyeXB0Lm9yZzCCAQQGCisGAQQB1nkCBAIEgfUEgfIA8AB3AESUZS6w
7s6vxEAH2Kj+KMDa5oK+2MsxtT/TM5a1toGoAAABe/KcFs0AAAQDAEgwRgIhANJ1
8OZD4y0iwpF7tWBRI+Mzo0m4jAr66TeG+zPFeGhKAiEA6YEv9lTAZNY9V8kmBWs7
FUWddQnd7Defgl+C0AQa+uEAdQB9PvL4j/+IVWgkwsDKnlKJeSvFDngJfy5ql2iZ
fiLw1wAAAXvynBbuAAAEAwBGMEQCIC3LXfXhust7RRcK+491g2UrVZ92m1BfbA19
71tzuTk0AiAhsz2A1NKEq7cjeDEb9i+XAopxh6A/rXvQrzLf0IjNLjANBgkqhkiG
9w0BAQsFAAOCAQEAb6FnAvJwOSWWFILY2E9lUJ/3fo/xDG7gtXfQfNkZfzrQ5uXa
tcc2YCV0SJve7OqRF3xAK/TYbIFpLjsBfetDoE+Tz2Vqa7sAYwH4J5IIVXt4LQef
PoHapxOmUO4+gfDqkVwKlXYXyKTCoIH0JUEfuwuA4hABBnbD6WDh0Zt4RO+ZYd9C
8Lh14YixuupKu3/Xm+yjT7OksVU9zX65gJ9Pades/qjHMb58SCli5ouljueLLNrJ
9/jhxM/5bcom4AgMvIybU2UnXaTDKkw5PhYRE9d6O8hSy1AZ4txKDyYEy9rESeR0
cDn5dB90nx74SJXHQfK2ScC2oVNvz4MXAPrYEg==
-----END CERTIFICATE-----
The chain file has two intermediate certificates. The first ("MIIFFj…") is signed by the newer "ISRG Root X1" CA certificate, and the second one ("MIIFYD…") is signed by the old/now-expired "DST whateveritwas X3".
Here's the chain file…
-----BEGIN CERTIFICATE-----
MIIFFjCCAv6gAwIBAgIRAJErCErPDBinU/bWLiWnX1owDQYJKoZIhvcNAQELBQAw
TzELMAkGA1UEBhMCVVMxKTAnBgNVBAoTIEludGVybmV0IFNlY3VyaXR5IFJlc2Vh
cmNoIEdyb3VwMRUwEwYDVQQDEwxJU1JHIFJvb3QgWDEwHhcNMjAwOTA0MDAwMDAw
WhcNMjUwOTE1MTYwMDAwWjAyMQswCQYDVQQGEwJVUzEWMBQGA1UEChMNTGV0J3Mg
RW5jcnlwdDELMAkGA1UEAxMCUjMwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEK
AoIBAQC7AhUozPaglNMPEuyNVZLD+ILxmaZ6QoinXSaqtSu5xUyxr45r+XXIo9cP
R5QUVTVXjJ6oojkZ9YI8QqlObvU7wy7bjcCwXPNZOOftz2nwWgsbvsCUJCWH+jdx
sxPnHKzhm+/b5DtFUkWWqcFTzjTIUu61ru2P3mBw4qVUq7ZtDpelQDRrK9O8Zutm
NHz6a4uPVymZ+DAXXbpyb/uBxa3Shlg9F8fnCbvxK/eG3MHacV3URuPMrSXBiLxg
Z3Vms/EY96Jc5lP/Ooi2R6X/ExjqmAl3P51T+c8B5fWmcBcUr2Ok/5mzk53cU6cG
/kiFHaFpriV1uxPMUgP17VGhi9sVAgMBAAGjggEIMIIBBDAOBgNVHQ8BAf8EBAMC
AYYwHQYDVR0lBBYwFAYIKwYBBQUHAwIGCCsGAQUFBwMBMBIGA1UdEwEB/wQIMAYB
Af8CAQAwHQYDVR0OBBYEFBQusxe3WFbLrlAJQOYfr52LFMLGMB8GA1UdIwQYMBaA
FHm0WeZ7tuXkAXOACIjIGlj26ZtuMDIGCCsGAQUFBwEBBCYwJDAiBggrBgEFBQcw
AoYWaHR0cDovL3gxLmkubGVuY3Iub3JnLzAnBgNVHR8EIDAeMBygGqAYhhZodHRw
Oi8veDEuYy5sZW5jci5vcmcvMCIGA1UdIAQbMBkwCAYGZ4EMAQIBMA0GCysGAQQB
gt8TAQEBMA0GCSqGSIb3DQEBCwUAA4ICAQCFyk5HPqP3hUSFvNVneLKYY611TR6W
PTNlclQtgaDqw+34IL9fzLdwALduO/ZelN7kIJ+m74uyA+eitRY8kc607TkC53wl
ikfmZW4/RvTZ8M6UK+5UzhK8jCdLuMGYL6KvzXGRSgi3yLgjewQtCPkIVz6D2QQz
CkcheAmCJ8MqyJu5zlzyZMjAvnnAT45tRAxekrsu94sQ4egdRCnbWSDtY7kh+BIm
lJNXoB1lBMEKIq4QDUOXoRgffuDghje1WrG9ML+Hbisq/yFOGwXD9RiX8F6sw6W4
avAuvDszue5L3sz85K+EC4Y/wFVDNvZo4TYXao6Z0f+lQKc0t8DQYzk1OXVu8rp2
yJMC6alLbBfODALZvYH7n7do1AZls4I9d1P4jnkDrQoxB3UqQ9hVl3LEKQ73xF1O
yK5GhDDX8oVfGKF5u+decIsH4YaTw7mP3GFxJSqv3+0lUFJoi5Lc5da149p90Ids
hCExroL1+7mryIkXPeFM5TgO9r0rvZaBFOvV2z0gp35Z0+L4WPlbuEjN/lxPFin+
HlUjr8gRsI3qfJOQFy/9rKIJR0Y/8Omwt/8oTWgy1mdeHmmjk7j1nYsvC9JSQ6Zv
MldlTTKB3zhThV1+XWYp6rjd5JW1zbVWEkLNxE7GJThEUG3szgBVGP7pSWTUTsqX
nLRbwHOoq7hHwg==
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
MIIFYDCCBEigAwIBAgIQQAF3ITfU6UK47naqPGQKtzANBgkqhkiG9w0BAQsFADA/
MSQwIgYDVQQKExtEaWdpdGFsIFNpZ25hdHVyZSBUcnVzdCBDby4xFzAVBgNVBAMT
DkRTVCBSb290IENBIFgzMB4XDTIxMDEyMDE5MTQwM1oXDTI0MDkzMDE4MTQwM1ow
TzELMAkGA1UEBhMCVVMxKTAnBgNVBAoTIEludGVybmV0IFNlY3VyaXR5IFJlc2Vh
cmNoIEdyb3VwMRUwEwYDVQQDEwxJU1JHIFJvb3QgWDEwggIiMA0GCSqGSIb3DQEB
AQUAA4ICDwAwggIKAoICAQCt6CRz9BQ385ueK1coHIe+3LffOJCMbjzmV6B493XC
ov71am72AE8o295ohmxEk7axY/0UEmu/H9LqMZshftEzPLpI9d1537O4/xLxIZpL
wYqGcWlKZmZsj348cL+tKSIG8+TA5oCu4kuPt5l+lAOf00eXfJlII1PoOK5PCm+D
LtFJV4yAdLbaL9A4jXsDcCEbdfIwPPqPrt3aY6vrFk/CjhFLfs8L6P+1dy70sntK
4EwSJQxwjQMpoOFTJOwT2e4ZvxCzSow/iaNhUd6shweU9GNx7C7ib1uYgeGJXDR5
bHbvO5BieebbpJovJsXQEOEO3tkQjhb7t/eo98flAgeYjzYIlefiN5YNNnWe+w5y
sR2bvAP5SQXYgd0FtCrWQemsAXaVCg/Y39W9Eh81LygXbNKYwagJZHduRze6zqxZ
Xmidf3LWicUGQSk+WT7dJvUkyRGnWqNMQB9GoZm1pzpRboY7nn1ypxIFeFntPlF4
FQsDj43QLwWyPntKHEtzBRL8xurgUBN8Q5N0s8p0544fAQjQMNRbcTa0B7rBMDBc
SLeCO5imfWCKoqMpgsy6vYMEG6KDA0Gh1gXxG8K28Kh8hjtGqEgqiNx2mna/H2ql
PRmP6zjzZN7IKw0KKP/32+IVQtQi0Cdd4Xn+GOdwiK1O5tmLOsbdJ1Fu/7xk9TND
TwIDAQABo4IBRjCCAUIwDwYDVR0TAQH/BAUwAwEB/zAOBgNVHQ8BAf8EBAMCAQYw
SwYIKwYBBQUHAQEEPzA9MDsGCCsGAQUFBzAChi9odHRwOi8vYXBwcy5pZGVudHJ1
c3QuY29tL3Jvb3RzL2RzdHJvb3RjYXgzLnA3YzAfBgNVHSMEGDAWgBTEp7Gkeyxx
+tvhS5B1/8QVYIWJEDBUBgNVHSAETTBLMAgGBmeBDAECATA/BgsrBgEEAYLfEwEB
ATAwMC4GCCsGAQUFBwIBFiJodHRwOi8vY3BzLnJvb3QteDEubGV0c2VuY3J5cHQu
b3JnMDwGA1UdHwQ1MDMwMaAvoC2GK2h0dHA6Ly9jcmwuaWRlbnRydXN0LmNvbS9E
U1RST09UQ0FYM0NSTC5jcmwwHQYDVR0OBBYEFHm0WeZ7tuXkAXOACIjIGlj26Ztu
MA0GCSqGSIb3DQEBCwUAA4IBAQAKcwBslm7/DlLQrt2M51oGrS+o44+/yQoDFVDC
5WxCu2+b9LRPwkSICHXM6webFGJueN7sJ7o5XPWioW5WlHAQU7G75K/QosMrAdSW
9MUgNTP52GE24HGNtLi1qoJFlcDyqSMo59ahy2cI2qBDLKobkx/J3vWraV0T9VuG
WCLKTVXkcGdtwlfFRjlBz4pYg1htmf5X6DYO8A4jqv2Il9DjXA6USbW1FzXSLr9O
he8Y4IWS6wY7bCkjCWDcRQJMEhg76fsO3txE+FiYruq9RUWhiF1myv4Q6W+CyBFC
Dfvp7OOGAN6dEOM4+qR9sdjoSYKEBpsr6GtPAQw4dy753ec5
-----END CERTIFICATE-----
what's wrong?
If I simply ask openssl to verify, it is unhappy:
18:09UTC:root@bard:~/TEMP# openssl verify -purpose sslserver -CAfile ./chain.pem ./cert.pem
C = US, O = Internet Security Research Group, CN = ISRG Root X1
error 2 at 2 depth lookup: unable to get issuer certificate
error ./cert.pem: verification failed
Wait wat?
…does that say the "ISRG Root X1" CA Certificate isn't installed locally?
muddying the waters
Letsencrypt is including those two certificates in the chain file just to support old Android systems. What if I try a simpler chain and just include the intermediate cert signed by the newer "ISRG Root X1". So, just the "MIIFFj…" in the chain file…
18:21UTC:root@bard:~/TEMP# openssl verify -purpose sslserver -CAfile ./chainISRGX1.pem ./cert.pem
./cert.pem: OK
18:22UTC:root@bard:~/TEMP# echo $?
0
Happy!
Which leads me to believe that it's the presence of the second, certificate in the chain file that causes the problem...
but my Openssl has the trusted_first option enabled.
