Score:0

LDAP replication to server with Let's Encrypt certificate fails, "unable to get issuer certificate"

in flag

I am currently trying to set up LDAP replication between to instances of 389 Directory Server (both running on Fedora 37), which I'll call $SUPPLIER and $CONSUMER in the following (serving at the domains supplier.mydomain.example and consumer.mydomain.example, respectively).

Both $SUPPLIER and $CONSUMER are configured identically and use Let's Encrypt certificates. I have successfully configured several clients of $SUPPLIER, which all work like a charm (all use ldaps:// to communicate with $SUPPLIER, and demand TLS certificates from it).

However, replicating from $SUPPLIER to $CONSUMER does not work. I receive the following error message:

[02/May/2023:14:28:07.389193770 +0200] - ERR - NSMMReplicationPlugin - bind_and_check_pwp - agmt="cn=Synchronize with $CONSUMER" (ldap3:636) - Replication bind with SIMPLE auth failed: LDAP error -1 (Can't contact LDAP server) (error:0A000086:SSL routines::certificate verify failed (unable to get issuer certificate))
[02/May/2023:14:28:10.408516918 +0200] - ERR - slapi_ldap_bind - Could not send bind request for id [cn=replication manager,cn=config] authentication mechanism [SIMPLE]: error -1 (Can't contact LDAP server), system error -5987 (Invalid function argument.), network error 0 (Unknown error, host "consumer.mydomain.example:636")

How do I resolve the “unable to get issuer certificate” error?

Most of the content I have already found about the problem assumes that self-signed certificates are used and that the CA needs to be manually installed; however I thought the entire joke of having a CA like Let's Encrypt issue a certificate was that the CA is already trusted by all operating systems / applications (e.g. like it is the case with browsers, where I can automatically connect to a Let's Encrypt-secured website).

Things that work

[email protected]$ dsconf -D "cn=replication manager,cn=config" ldaps://consumer.mydomain.example:636 monitor server
dn: cn=monitor
version: 389-Directory/2.3.3
...

Any other valid dsconf (or probably also dsidm) commands also work, i.e. there is no error with the encrypted connection here.

[email protected]$ echo -n "" | openssl s_client -showcerts consumer.mydomain.example:636
CONNECTED(00000003)
depth=2 C = US, O = Internet Security Research Group, CN = ISRG Root X1
verify return:1
depth=1 C = US, O = Let's Encrypt, CN = R3
verify return:1
depth=0 CN = consumer.mydomain.example
verify return:1
---
Certificate chain
 0 s:CN = consumer.mydomain.example
   i:C = US, O = Let's Encrypt, CN = R3
   a:PKEY: id-ecPublicKey, 256 (bit); sigalg: RSA-SHA256
   v:NotBefore: Apr 10 18:33:38 2023 GMT; NotAfter: Jul  9 18:33:37 2023 GMT
-----BEGIN CERTIFICATE-----
[...]
-----END CERTIFICATE-----
 1 s:C = US, O = Let's Encrypt, CN = R3
   i:C = US, O = Internet Security Research Group, CN = ISRG Root X1
   a:PKEY: rsaEncryption, 2048 (bit); sigalg: RSA-SHA256
   v:NotBefore: Sep  4 00:00:00 2020 GMT; NotAfter: Sep 15 16:00:00 2025 GMT
-----BEGIN CERTIFICATE-----
[...]
-----END CERTIFICATE-----
 2 s:C = US, O = Internet Security Research Group, CN = ISRG Root X1
   i:O = Digital Signature Trust Co., CN = DST Root CA X3
   a:PKEY: rsaEncryption, 4096 (bit); sigalg: RSA-SHA256
   v:NotBefore: Jan 20 19:14:03 2021 GMT; NotAfter: Sep 30 18:14:03 2024 GMT
-----BEGIN CERTIFICATE-----
[...]
-----END CERTIFICATE-----
---
Server certificate
subject=CN = consumer.mydomain.example
issuer=C = US, O = Let's Encrypt, CN = R3
---
No client certificate CA names sent
Requested Signature Algorithms: ECDSA+SHA256:ECDSA+SHA384:ECDSA+SHA512:RSA-PSS+SHA256:RSA-PSS+SHA384:RSA-PSS+SHA512:RSA+SHA256:RSA+SHA384:RSA+SHA512
Shared Requested Signature Algorithms: ECDSA+SHA256:ECDSA+SHA384:ECDSA+SHA512:RSA-PSS+SHA256:RSA-PSS+SHA384:RSA-PSS+SHA512:RSA+SHA256:RSA+SHA384:RSA+SHA512
Peer signing digest: SHA256
Peer signature type: ECDSA
Server Temp Key: X25519, 253 bits
---
SSL handshake has read 4205 bytes and written 424 bytes
Verification: OK
---
New, TLSv1.3, Cipher is TLS_AES_128_GCM_SHA256
Server public key is 256 bit
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 0 (ok)
---
DONE
[email protected]$ trust list
[...]
pkcs11:id=%79%B4%59%E6%7B%B6%E5%E4%01%73%80%08%88%C8%1A%58%F6%E9%9B%6E;type=cert
    type: certificate
    label: ISRG Root X1
    trust: anchor
    category: authority
[...]

Things that don't work

[email protected]$ echo -n | openssl s_client -connect consumer.mydomain.example:636 | sed -ne '/-BEGIN CERTIFICATE-/,/-END CERTIFICATE-/p' > /tmp/ldapserver.pem
[email protected]$ openssl verify /tmp/ldapserver.pem  
CN = consumer.mydomain.example
error 20 at 0 depth lookup: unable to get local issuer certificate
error ldapserver.pem: verification failed

This seems to be the same error message as in the LDAP server's log, so maybe the two problems are related?

Update: Per dave_thompson_085's comment, this seems to be more of a Red Herring, as this command will apparently always fail, so it failing here is no surprise.

Related

  1. Unable to establish replication with STARTTLS - 389-users - Fedora Mailing Lists: Problem seems to be very similar (and quite recent), not yet resolved.
  2. Old Let's Encrypt Root Certificate Expiration and OpenSSL 1.0.2: Might have been the problem if this was two years ago, however, I wouldn't expect it to cause any trouble on a modern OS in 2023.
  3. Unable to openssl verify letsencrypt certificate suggests to use the -untrusted option when using openssl verify, however, I cannot pass this option to 389 DS, and I also don't really want to put -untrusted flags anywhere near a production environment.
  4. 0A000086 seems to be an error code emitted by OpenSSL (see e.g. this web search), so the problem is probably one in OpenSSL and not with 389 Directory Server directly?
  5. A maybe related bug, discussing how 389 DS is linked to two crypto libraries, which might cause problems
  6. Source code for slapi_ldap_bind, which raises the error message
cn flag
Does your setup have the network access to perform validation of the certificates? E.g. access to the CDP/AIA/OCSP location? That should be the *first* thing to check.
TuringTux avatar
in flag
@GregAskew Yes, both supplier and consumer have full outgoing internet access, meaning that e.g. http://r3.o.lencr.org, http://r3.i.lencr.org/, the CRL at http://x1.c.lencr.org/ are all accessible (just verified it on the host with `curl`).
dave_thompson_085 avatar
jp flag
TLS cert validation in general, and LE in particular, requires intermediate/chain cert(s). Your `openssl s_client >file; openssl verify file` tries to verify using _only_ the leaf cert which NEVER works and gives no useful evidence; that's why `-untrusted` with the chain(s) is suggested. Try `openssl s_client -showcerts` and look at _all_ the received certs, and the verification callback log. @GregAskew+ note OpenSSL _never_ uses AIA to fetch chain, and by default does not check revocation at all; that must be requested by the calling code and I have no idea if 389 does or can.
dave_thompson_085 avatar
jp flag
PS: the problem with OpenSSL 1.0.2 vs LetsEncrypt 'compatibility' chain when DST Root X3 expired in 2021 had the system of reporting your cert _expired_ (when it shouldn't be) not 'unable to get issuer'. And we have _dozens_ of Qs on several Stacks about it you could reference :-)
TuringTux avatar
in flag
@dave_thompson_085 Thanks, I have provided the output of `openssl s_client -showcerts`. Seems fine to my (untrained) eye.
dave_thompson_085 avatar
jp flag
Yes that chain looks good and more important the callback log, which checked also the details you redacted, says it _is_ good. Either (1) the 389 program is getting a different chain somehow (a server _can_ do this though I don't see why it would here) or (2) it is using a different/wrong truststore. You might be able to check (1) with an external trace like wireshark; otherwise I think you need someone who knows about 389 (I don't) or to get source and start debugging it (not fun, sorry).
Score:1
lc flag

the error message indicates: unable to get issuer certificate which points to a PKI trust issue for the LDAP service and possibly the system.
I would add and trust all the issuers in the chain of trust that issued the LDAP SSL server certificate. Into the LDAP NSS db directory with a,

certutil -A -d xx -n xx -t CT -a -i some.ca.cert.pem.txt

as well as on the system with a trust anchor some.ca.cert.pem.txt from p11-kit, assuming this is RHEL-9 or RHEL-8, or Fedora.

This needs to be done on each replica, and the trust anchor is also needed on the LDAP clients.

Then run some test ldapsearch command with LDAPS and STARTTLS, locally, and across the replicas, and from some LDAP clients.

Add a -d 4" to a test ldapsearch command in case of issues, but I would expect the same issuer trust message to be provided.
Note the default LDAP SSL server certificate provided, is only a test "dummy" cert to show the features out of the box, and needs to be replaced. Note also when testing and comparing, each client tool has different requirements.

For example, using OpenSSL s_client will have a different behavior from a OpenLDAP client's ldapsearch, or from a Java client or a Python client, or SSSD, etc.

mangohost

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.