Why does my RSA DANE TLSA work, but my ECDSA DANE TLSA fail?

I've purchased two single domain, wildcard SSL certificates from Namecheap/Sectigo/Comodo. I generated my CSRs in the typical fashion using openssl.

$ openssl req -newkey rsa:4096 -keyout example.com.rsa.key -out example.com.rsa.csr
$ openssl genpkey -genparam -algorithm ec -pkeyopt ec_paramgen_curve:P-256 -out example.com.ecdsa.pem
$ openssl req -newkey ec:example.com.ecdsa.pem -keyout example.com.ecdsa.key -out example.com.ecdsa.csr

I submitted the CSR's and was issued each certificate package including the .crt and .ca-bundle, all as expected.

I was able to install both certificates for use by Postfix. Each required an unencrypted version of the private key, secure permissions having been maintained.

$ openssl rsa -in example.com.rsa.key -out example.com.rsa.key.unencrypted
$ openssl ec -in example.com.ecdsa.key -out example.com.ecdsa.key.unencrypted

I have correctly setup SPF, DKIM, DMARC, and DNSSEC, each confirmed by third party tools and having sent and received mail with passing scores in the headers.

I again used openssl to generate the hashes of the certificates for my DANE TLSA records. I created two sets, one set for each algorithm, and for both port 25 and 587. These were added to my zone file, checked with named-checkzone, signed with dnssec-signzone, and published in the DNS.

I began by using two external tools to check the configuration, the first at danecheck and the second at dane.sys4.de. Both reported overall success using the RSA cert but specific failure of the ECDSA cert.

The former succeeded, reporting in part:

DANE TLSA 3 1 1 [9679fc29..]: OK matched EE certificate
DANE TLSA 3 1 1 [ecd29ffd..]: FAIL did not match EE certificate

The latter succeeded, reporting in part:

3, 1, 1 9679fc296960a23c[...]149b990a680cad8b *in green*
3, 1, 1 ecd29ffd76d61326[...]dadbcfa42eae9158 - unable to get local issuer certificate: (20) *in red*

I attempted to fix this by changing the various usage, selector, and matching fields, 3 0 1, 3 1 1, etc. I tried generating the hashes locally with openssl and by using online tools like gen_tlsa. Both tools produced identical hashed results for both certificates. Again, the RSA TLSA's succeeded while the ECDSA records failed. I eventually found the chaingen script, which generates all of the 3 0 1, 3 1 1, 3 0 2, and 3 1 2 hashes which I included for each port with no improvement for the ECDSA records.

I spent several hours trying different permutations of each certificate, TLSA records, ports, etc. I tried adding TLSA records (2 0 1) for the parent keys of the ECDSA records to the DNS. I tried changing the certificate available to Postfix to include the ca-bundle along with the certificate in a .pem file.

Researching I found this post at exim-users titled "How to get ec cert used with DANE and ec+rsa certs" by Viktor Dukhovni, who needs no introduction in the Postfix/DANE circle. He suggested that perhaps the online tools were opportunistic and as soon as they succeeded with the RSA certs didn't bother to test with the ECDSA certs. He provided very valuable calls to openssl to show that indeed the keys were correct in the poster's situation. I duplicated that effort with two short shell scripts using Viktor's code substituting my information, one called checkrsa and the other called checkecdsa.

The two scripts execute two commands each with the following setup for checkrsa (for IPv4):

$ cat checkrsa

echo "quit" | openssl s_client -starttls smtp -connect mail.example.com:25 -4 -verify 9 \
    -dane_tlsa_domain mail.example.com \
    -dane_tlsa_rrdata "3 0 1 c487cdb079b49d12ee357d9547c6f67448a2ec2e789c86c65d213a57aaec4ac9" \
    -dane_tlsa_rrdata "3 1 1 9679fc296960a23c89fed73d8e6a6d0ab33f0abc99d48c44149b990a680cad8b" \
    -sigalgs rsa_pkcs1_sha256:rsa_pkcs1_sha384:rsa_pkcs1_sha512:rsa_pss_rsae_sha256:rsa_pss_rsae_sha384:rsa_pss_rsae_sha512:rsa_pss_pss_sha256:rsa_pss_pss_sha384:rsa_pss_pss_sha512

And checkecdsa (for IPv6):

$ cat checkecdsa
echo "quit" | openssl s_client -starttls smtp -connect mail.example.com:25 -6 -verify 9 \
    -dane_tlsa_domain mail.example.com \
    -dane_tlsa_rrdata "3 0 1 87a8c75581c9d022062416a37387de1ec30d311e4d7afbeb892be287fb13bfc5" \
    -dane_tlsa_rrdata "3 1 1 ecd29ffd76d6132629ddbf7ccd31460e23ac3b4a50d47bccdadbcfa42eae9158" \
    -sigalgs ecdsa_secp256r1_sha256:ecdsa_secp384r1_sha384:ecdsa_secp521r1_sha512

The results:

root@server:~/bin# ./checkrsa 
verify depth is 9
CONNECTED(00000003)
depth=0 CN = *.example.com
verify return:1
---
Certificate chain
 0 s:CN = *.example.com
   i:C = GB, ST = Greater Manchester, L = Salford, O = Sectigo Limited, CN = Sectigo RSA Domain Validation Secure Server CA
---
Server certificate
-----BEGIN CERTIFICATE-----
MIIHNDCCBhygAwIBAgIQAzfJZrX65NjG2FvLmk0I0jANBgkqhkiG9w0BAQsFADCB
...
Xw8UgZDYihDaIxT8SUQUgV9weQg5Lkru
-----END CERTIFICATE-----
subject=CN = *.example.com

issuer=C = GB, ST = Greater Manchester, L = Salford, O = Sectigo Limited, CN = Sectigo RSA Domain Validation Secure Server CA

---
No client certificate CA names sent
Peer signing digest: SHA256
Peer signature type: RSA-PSS
Server Temp Key: X25519, 253 bits
---
SSL handshake has read 2904 bytes and written 386 bytes
Verification: OK
Verified peername: *.example.com
DANE TLSA 3 1 1 ...99d48c44149b990a680cad8b matched EE certificate at depth 0
---
New, TLSv1.3, Cipher is TLS_AES_256_GCM_SHA384
Server public key is 4096 bit
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 0 (ok)
---
250 CHUNKING
DONE
root@server:~/bin# ./checkecdsa 
verify depth is 9
CONNECTED(00000003)
depth=0 CN = *.example.com
verify return:1
---
Certificate chain
 0 s:CN = *.example.com
   i:C = GB, ST = Greater Manchester, L = Salford, O = Sectigo Limited, CN = Sectigo 
ECC Domain Validation Secure Server CA
---
Server certificate
-----BEGIN CERTIFICATE-----
MIIEqDCCBE6gAwIBAgIRAIIKQBNWh2R9wwvn2/j30PgwCgYIKoZIzj0EAwIwgY8x
...
YemreHq/Cd5HPgIgE6InSF5ko6mWo9GMpR7w1ijpbsnShlS6EiYrpZozD0s=
-----END CERTIFICATE-----
subject=CN = *.example.com

issuer=C = GB, ST = Greater Manchester, L = Salford, O = Sectigo Limited, CN = Sectigo ECC Domain Validation Secure Server CA

---
No client certificate CA names sent
Peer signing digest: SHA256
Peer signature type: ECDSA
Server Temp Key: X25519, 253 bits
---
SSL handshake has read 1811 bytes and written 348 bytes
Verification: OK
Verified peername: *.example.com
DANE TLSA 3 1 1 ...50d47bccdadbcfa42eae9158 matched EE certificate at depth 0
---
New, TLSv1.3, Cipher is TLS_AES_256_GCM_SHA384
Server public key is 256 bit
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 0 (ok)
---
250 CHUNKING
DONE

So, both certificates checkout locally, but continue to fail externally.

Seeing that Viktor's message indicated that perhaps the tools, having success with RSA bailed on the ECDSA hashes, I also tried just publishing the ECDSA TLSA records. The results showed a complete failure with just the ECDSA hashes on their own.

So, I'm let with this situation: I have two sets of SSL certificates, one RSA, the other ECDSA. Both certs succeed in Postfix. Both certs checkout with openssl. Publishing both certs passes overall, based on the RSA certs, noting that the ECDSA certs fail. The ECDSA certs succeed locally, but fail externally.

I'm not sure where to go from here and am looking for help in getting my ECDSA certificate setup in parallel with my RSA records for DANE.

Yup, the tools you mentioned gave you misleading results.

A good test needs to examine the cross product of IP addresses times algorithms and then for each of those: check whether the presented cert chains up to a trusted CA and whether it chains up to or matches a certificate published via TLSA. That is 8 verification steps from 4 unique negotiation attempts, before even considering older protocol versions and SNI.

Some tests only try the first IP, some tests only try the first negotiated cipher suite. Most tests fail to report in unambiguous language whether the DANE or the PKI match failed.

The s_client command you tried is just fine.. but you want to run four not two commands (twice that, if you use IPv4 and IPv6). Because I suspect while your TLSA records are okay, you are not sending along (the correct) intermediates yet.

I'm no expert in DANE or TLSA (just learning the ropes here), but my guess is that @anx is quite correct: you have too many entries for each port. One should suffice, preferably 3 1 1 (or 3 1 2 if you prefer SHA512) since that ought to allow the sending end to check the full chain of trust.

This is mostly because the default behaviour of DNS resolvers I'm aware of (that means there may be many others that I'm not aware of!) is to ask for all records matching a query (for example: all records for _25._tcp.mail) and randomly pick one among the possible options. Some DNS RR allow for a weighted choice (MX records, for instance); some conventions regarding TXT fields might have their own way of encoding priorities and preferences, but, by default, most resolvers will simply pick one reply at random and use it.

Thus, such resolvers might only get a 1 x x reply (because it was the one chosen at random) and be strict about DANE checking... and reject it because it cannot validate the full chain of trust.

Mind you, I'm just speculating here. My above reasoning fails to explain why you manage RSA to be accepted but not ECDSA. Theoretically, since you have the same setup for both, both should fail — but I'm assuming that there is something I'm missing here.

My own DNS is almost exclusively managed by Cloudflare (except for a few DynDNS domains), and one thing that Cloudflare enforces is that only one TLSA rule per port can be active in the zone. That way, you cannot make the 'mistake' of adding multiple entries — their backoffice interface will not allow that, ever, even assuming that there could be a good reason for doing so. Thus, for my own DANE setups, I just have 3 1 1 entries. It seems to work well that way, as far as I can test it...