Connection reset for HTTPS on ADFS

I have set up a windows server in virtual box with host only network as a labenvironment. I have a problem accessing HTTPS from the host only network. I have set up a domain controller and ADFS on a Windows server 2019 using a self signed certificate for ADFS. Here are some tests I have done to try to isolate the problem. Im testing against the metadata url /FederationMetadata/2007-06/FederationMetadata.xml

• If I open IE in the VM and go to the metadata endpoint on localhost , I get the message "This site is not secure", but can bypass the warning and get the metadata.
• If I go to the host only interface IP, IE just shows a message saying "Can’t connect securely to this page" and it is not possible to bypass.
• If I do curl from the host to the host only IP address with -k flag towards port 443 I get connection reset.
• If I do curl against port 80 on the host only IP I get a 404 page as expected.
• Ping works fine from the host.
• Telnet from host to 443 connects

I have disabled the windows firewall.

the certificate was generated using this powershell

 $selfSignedCert = New-SelfSignedCertificateEx `
     -Subject "CN=adfs.samlsecurity.com" `
     -ProviderName "Microsoft Enhanced RSA and AES Cryptographic Provider" `
     -KeyLength 2048 -FriendlyName 'OAFED SelfSigned' -SignatureAlgorithm sha256 `
     -EKU "Server Authentication", "Client authentication" `
     -KeyUsage "KeyEncipherment, DigitalSignature" `
     -Exportable -StoreLocation "LocalMachine"

So there seem to be connectivity as I can get the HTTP 404 page, but for some reason I get connection refused from 443. I guessing there is something wrong with the TLS setup on windows, but I can't figure out what.

Found it! had a look at the packets in Wireshark. The only difference between packets that got a response from the server and the one that didnt was the use of the SNI extention in the TLS client hello.

This led me to search SNI and connection reset in windows server. Turns out it uses the SNI feature to be able to deliver different certificates based on recester server name in TLS SNI.

ADFS does not by default register any fallback certificate for other server names than localhost and the FQDN for ADFS. When I used the IP address for ADFS, no certificate was applicable and the server closed the connection.

I solved this first by registering a default certificate using

netsh add sslcert ipport=0.0.0.0:442 appid='{<ADFS_GUID>}' certhash=<thumprint without space>

Later i solved it by adding the FQDN and IP to the hosts file.

just wanted to share my experience.

I have a lab ADFS, which I recently assigned a wildcard certificate to. Today went to use it, and Edge was giving me ERR_CONNECTION_RESET. ADFS was up. DNS was correct. Everything looked right so far. I re-assigned the Service Communications Certificate via the AD FS Management console. But was still getting the error.

Your post lead me to my solution, which I think is the same.

The console / ADFS wasn't assigning the certificate I was selecting due to it not being able to match the name exactly (adfs.name.com /= *.name.com).

Running 'netsh http show sslcert' showed all 3 entries with the same cert thumbprint, but did not match my wildcard cert. I then ran Get-AdfsSslCertificate which showed the same. I then ran Set-AdfsSslCertificate -Thumbprint which aligned all 3 ADFS services with the correct certificate, and ADFS was back.