I found the problem, which might not be relevant to many other people, but it was indeed the issue in my specific case:
The problem
openssl s_client
, apparently by default (?), requests both IPv4 and IPv6 addresses from DNS nameservers and then apparently tries to connect only via IPv6 if it gets a valid response to the IPv6 DNS request.
The problem was that I had not opened the IPv6 HTTP(S) ports, so openssl
was requesting an IPv6 address from DNS servers, getting a valid response and valid IPv6, then attempting to connect to that IPv6 address and getting blocked by the firewall.
Other programs such as nmap
and Firefox were evidently either using IPv4 by default or falling back to IPv4 after noticing that IPv6 was failing, hence why I couldn't detect the issue at first.
How to detect it
If it helps someone else:
I figured out that this was the issue by listening for TCP traffic on port 53 (DNS):
sudo tcpdump -n port 53
where I noticed the following exchange following my execution of openssl s_client
:
18:24:57.649939 IP local-ip.50643 > dns-server.53: 12404+ [1au] A? domain-name.com. (55)
18:24:57.650293 IP local-ip.39731 > dns-server.53: 26725+ [1au] AAAA? domain-name.com. (55)
18:24:57.651843 IP dns-server.53 > local-ip.50643: 12404* 2/0/1 CNAME cname-of-server., A server-ipv4-address (146)
18:24:57.651877 IP dns-server.53 > local-ip.39731: 26725* 2/0/1 CNAME cname-of-server., AAAA server-ipv6-address (158)
indicating that openssl
had requested and subsequently received both IPv4 (A
) and IPv6 (AAAA
) DNS questions and answers.
(Here and in the following, local-ip
, dns-server
, cname-of-server
, server-ipv6-address
and server-ipv4-address
are placeholders and not literal.)
Then I checked for traffic to and from the IPv4 and IPv6 addresses that were given as replies to the DNS query:
sudo tcpdump host server-ipv6-address # in one tab
sudo tcpdump host server-ipv4-address # in another tab
right before running the openssl s_client
command.
I noticed that there were unanswered session establishment (SYN, [S]
in tcpdump
output) packets for the IPv6 host, but nothing on IPv4, indicating that openssl
was ignoring the IPv4 address that it received and trying to connect using only the IPv6 answer.
It works if you fix it
After I changed the firewall settings of the server to allow incoming IPv6 HTTP(S) packets, openssl s_client
was able to successfully connect and tcpdump
showed normal handshakes taking place with the IPv6 host. Side note: Firefox automatically switched to IPv6 after the ports were opened.