sendmail: server shows failures due to bad client certificate

My sendmail server (FreeBSD, compiled against OpenSSL) is refusing to negotiate TLS with one particular set of clients.

Log messages look like this:

sm-mta[92474]: STARTTLS=server, error: accept failed=-1, reason=sslv3 alert bad certificate, SSL_error=1, errno=0, retry=-1, relay=xx.example.com [XX.XX.XX.XX]

Other than these failures, which originate with a well-known ISP's mail infrastructure, everything is working great; the certificate on the server itself is up-to-date and valid.

Two questions, an answer to either one would be helpful:

1. How can I diagnose this? In particular, can I get a copy of the bad certificate from within the sendmail environment?
2. Can I update my sendmail config to either tell clients not to provide a certificate, or to tell sendmail to accept a bogus client certificate?

Thanks!

EDIT: I found the issue here, which was that my server's certificate didn't match my server's DANE TLSA record. Looking at a Wireshark decode of the transaction showed that the ISP (Comcast) was receiving my valid certificate but rejecting it anyway, and it turned out that the TLSA record was not synchronized. It looks like Comcast might be one of the few email senders on the Internet that refuses to send after a DANE verification failure.

You could use OpenSSL to query the remote server. For example, to see the certificate on one of Google's mail server:

echo quit | openssl s_client -connect aspmx.l.google.com:25 -starttls smtp

should show something similar to:

CONNECTED(00000003)
depth=2 C = US, O = Google Trust Services LLC, CN = GTS Root R1
verify return:1
depth=1 C = US, O = Google Trust Services LLC, CN = GTS CA 1C3
verify return:1
depth=0 CN = mx.google.com
verify return:1
---
Certificate chain
 0 s:CN = mx.google.com
   i:C = US, O = Google Trust Services LLC, CN = GTS CA 1C3
 1 s:C = US, O = Google Trust Services LLC, CN = GTS CA 1C3
   i:C = US, O = Google Trust Services LLC, CN = GTS Root R1
 2 s:C = US, O = Google Trust Services LLC, CN = GTS Root R1
   i:C = BE, O = GlobalSign nv-sa, OU = Root CA, CN = GlobalSign Root CA
---
Server certificate
-----BEGIN CERTIFICATE-----
MIIG9DCCBdygAwIBAgIQdfcJTB0GgVsKbOXnm+OJmTANBgkqhkiG9w0BAQsFADBG
MQswCQYDVQQGEwJVUzEiMCAGA1UEChMZR29vZ2xlIFRydXN0IFNlcnZpY2VzIExM
QzETMBEGA1UEAxMKR1RTIENBIDFDMzAeFw0yMzA0MDMwODIzNTRaFw0yMzA2MjYw
ODIzNTNaMBgxFjAUBgNVBAMTDW14Lmdvb2dsZS5jb20wWTATBgcqhkjOPQIBBggq
hkjOPQMBBwNCAASHHjjoYwgwyC6/DYMfRij2+kZyiywt8lVwAyy90RasYt3tDcmD
kqUhLf8Mv527u0akQVq00cpcRg9W1tFgSKl9o4IE1TCCBNEwDgYDVR0PAQH/BAQD
AgeAMBMGA1UdJQQMMAoGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwHQYDVR0OBBYE
FBneGLCbQL6PsUCbmin+e8j25tSYMB8GA1UdIwQYMBaAFIp0f6+Fze6VzT2c0OJG
FPNxNR0nMGoGCCsGAQUFBwEBBF4wXDAnBggrBgEFBQcwAYYbaHR0cDovL29jc3Au
cGtpLmdvb2cvZ3RzMWMzMDEGCCsGAQUFBzAChiVodHRwOi8vcGtpLmdvb2cvcmVw
by9jZXJ0cy9ndHMxYzMuZGVyMIIChgYDVR0RBIICfTCCAnmCDW14Lmdvb2dsZS5j
b22CD3NtdHAuZ29vZ2xlLmNvbYISYXNwbXgubC5nb29nbGUuY29tghdhbHQxLmFz
cG14LmwuZ29vZ2xlLmNvbYIXYWx0Mi5hc3BteC5sLmdvb2dsZS5jb22CF2FsdDMu
YXNwbXgubC5nb29nbGUuY29tghdhbHQ0LmFzcG14LmwuZ29vZ2xlLmNvbYIaZ21h
aWwtc210cC1pbi5sLmdvb2dsZS5jb22CH2FsdDEuZ21haWwtc210cC1pbi5sLmdv
b2dsZS5jb22CH2FsdDIuZ21haWwtc210cC1pbi5sLmdvb2dsZS5jb22CH2FsdDMu
Z21haWwtc210cC1pbi5sLmdvb2dsZS5jb22CH2FsdDQuZ21haWwtc210cC1pbi5s
Lmdvb2dsZS5jb22CGGdtci1zbXRwLWluLmwuZ29vZ2xlLmNvbYIdYWx0MS5nbXIt
c210cC1pbi5sLmdvb2dsZS5jb22CHWFsdDIuZ21yLXNtdHAtaW4ubC5nb29nbGUu
Y29tgh1hbHQzLmdtci1zbXRwLWluLmwuZ29vZ2xlLmNvbYIdYWx0NC5nbXItc210
cC1pbi5sLmdvb2dsZS5jb22CDW14MS5zbXRwLmdvb2eCDW14Mi5zbXRwLmdvb2eC
DW14My5zbXRwLmdvb2eCDW14NC5zbXRwLmdvb2eCFWFzcG14Mi5nb29nbGVtYWls
LmNvbYIVYXNwbXgzLmdvb2dsZW1haWwuY29tghVhc3BteDQuZ29vZ2xlbWFpbC5j
b22CFWFzcG14NS5nb29nbGVtYWlsLmNvbYIRZ21yLW14Lmdvb2dsZS5jb20wIQYD
VR0gBBowGDAIBgZngQwBAgEwDAYKKwYBBAHWeQIFAzA8BgNVHR8ENTAzMDGgL6At
hitodHRwOi8vY3Jscy5wa2kuZ29vZy9ndHMxYzMvUU92SjBOMXNUMkEuY3JsMIIB
AwYKKwYBBAHWeQIEAgSB9ASB8QDvAHUAejKMVNi3LbYg6jjgUh7phBZwMhOFTTvS
K8E6V6NS61IAAAGHRm4lhwAABAMARjBEAiAbFYSOoxVgUmcJ//sPa+hYjV4DrVfp
I3BK/Z6oBCwARQIgKKWKSA17g7eWZarLagY5oG4P+UxzBZd5uJHM+nj379IAdgDo
PtDaPvUGNTLnVyi8iWvJA9PL0RFr7Otp4Xd9bQa9bgAAAYdGbiVNAAAEAwBHMEUC
IQDINMfMYnK9vpptQYj9Ve6EOYa26GZV2TaM4Sw7J30ZkwIgSB8GtydIcaC+2pgJ
EifKh9ZkJvY/o6L24amB1gFUxIswDQYJKoZIhvcNAQELBQADggEBAAnbG+WzOmkA
FqJGasrOMQGIcLwfHSwEAeAjuQOCVxZ8I0e3bKV3+4jLMDEXz9SkhTCXq57p9pWu
YorP5fPlR9+5Z2cCCCbFU5sHXRsZ3olFqVIbTqtN0pxAOFSKah85KnEq2cMSeQ6j
fhhFT41nLvd+QotH5vYh0IUnSgEOGnOxNfgc/Ixk75ENZ7GBZopWzIQ75J+X8RE6
j/SC42wunfCeR6xywhlpDg70mwjLP6QslX/JHfsR6pg+MLyTgmrTincbWeStauvN
G2JZkth9MeorZoTmyx+vooeKQgIV8PKRs5Llc+lSjA4b9wlJNHA9hMd5mBRZL291
oMYlrLxMli8=
-----END CERTIFICATE-----
subject=CN = mx.google.com

issuer=C = US, O = Google Trust Services LLC, CN = GTS CA 1C3

---
No client certificate CA names sent
Peer signing digest: SHA256
Peer signature type: ECDSA
Server Temp Key: X25519, 253 bits
---
SSL handshake has read 5201 bytes and written 423 bytes
Verification: OK
---
New, TLSv1.3, Cipher is TLS_AES_256_GCM_SHA384
Server public key is 256 bit
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 0 (ok)
---
250 SMTPUTF8
DONE

You can add -showcerts to get all the certificates in the chain.

My generic way to approach certificate validation issue is:

1. Take a netowrk capture with tcpdump or similar
2. Open the capture in Wireshark and right click on the SSL/TLS ALERT packet and chose Follow -> TCP stream
3. Check the X509 certificate:
   • Dates (is the current date between issued date and expiration date?)
   • Subject ( the hostname used by the client is the same as the CN= field?)
   • Subject Alternative Name (similar to Subject)
   • Constraints
   • Intermediate certificates (are all intermediate certificates presented after the host certificate? are they valid?)

Usually is a missing intermediate certificate that gives this type of issue (some clients do not validate the certificate of your server).

It turned out that capturing and analyzing the TLS negotiation in Wireshark was the key here, and it led me to discover that the ISP (Comcast) was rejecting my domain certificate, despite the fact that almost every other sender on the Internet was accepting it.

Further investigation found that the TLSA record for the mail server in question was out-of-date, and that Comcast was rejecting the certificate due to a TLSA mismatch.

Unfortunately, the error mechanisms available in TLS negotiation do not provide a way for the sender to tell the receiver why the certificate was rejected. Maybe TLS 1.4 can define a specific alert for DANE certificate mismatch.