TLS 1.2 Client hello resulting in TCP reset

I've been trying to set up a test environment where I have a windows service hosting an IDP. The IDP is, for legacy reasons, available using both HTTP and HTTPS. On the same host I've also deployed a test program which should call the service and test the IDP. The host is running Windows Server 2016 Standard.

When I call the IDP using HTTPS I get a TCP reset in response to Client Hello:

    Content Type: Handshake (22)
    Version: TLS 1.2 (0x0303)
    Length: 156
    Handshake Protocol: Client Hello
        Handshake Type: Client Hello (1)
        Length: 152
        Version: TLS 1.2 (0x0303)
        Random: 61404bb1f286d56caf2a6f1eb9d47f62ea8e7c8dd7764e6654d56ab4f90fef7a
        Session ID Length: 0
        Cipher Suites Length: 56
        Cipher Suites (28 suites)
            Cipher Suite: TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384 (0xc02c)
            Cipher Suite: TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256 (0xc02b)
            Cipher Suite: TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (0xc030)
            Cipher Suite: TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 (0xc02f)
            Cipher Suite: TLS_DHE_RSA_WITH_AES_256_GCM_SHA384 (0x009f)
            Cipher Suite: TLS_DHE_RSA_WITH_AES_128_GCM_SHA256 (0x009e)
            Cipher Suite: TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384 (0xc024)
            Cipher Suite: TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256 (0xc023)
            Cipher Suite: TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384 (0xc028)
            Cipher Suite: TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256 (0xc027)
            Cipher Suite: TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA (0xc00a)
            Cipher Suite: TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA (0xc009)
            Cipher Suite: TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA (0xc014)
            Cipher Suite: TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA (0xc013)
            Cipher Suite: TLS_DHE_RSA_WITH_AES_256_CBC_SHA (0x0039)
            Cipher Suite: TLS_DHE_RSA_WITH_AES_128_CBC_SHA (0x0033)
            Cipher Suite: TLS_RSA_WITH_AES_256_GCM_SHA384 (0x009d)
            Cipher Suite: TLS_RSA_WITH_AES_128_GCM_SHA256 (0x009c)
            Cipher Suite: TLS_RSA_WITH_AES_256_CBC_SHA256 (0x003d)
            Cipher Suite: TLS_RSA_WITH_AES_128_CBC_SHA256 (0x003c)
            Cipher Suite: TLS_RSA_WITH_AES_256_CBC_SHA (0x0035)
            Cipher Suite: TLS_RSA_WITH_AES_128_CBC_SHA (0x002f)
            Cipher Suite: TLS_RSA_WITH_3DES_EDE_CBC_SHA (0x000a)
            Cipher Suite: TLS_DHE_DSS_WITH_AES_256_CBC_SHA256 (0x006a)
            Cipher Suite: TLS_DHE_DSS_WITH_AES_128_CBC_SHA256 (0x0040)
            Cipher Suite: TLS_DHE_DSS_WITH_AES_256_CBC_SHA (0x0038)
            Cipher Suite: TLS_DHE_DSS_WITH_AES_128_CBC_SHA (0x0032)
            Cipher Suite: TLS_DHE_DSS_WITH_3DES_EDE_CBC_SHA (0x0013)
        Compression Methods Length: 1
        Compression Methods (1 method)
            Compression Method: null (0)
        Extensions Length: 55
        Extension: supported_groups (len=8)
            Type: supported_groups (10)
            Length: 8
            Supported Groups List Length: 6
            Supported Groups (3 groups)
        Extension: ec_point_formats (len=2)
            Type: ec_point_formats (11)
            Length: 2
            EC point formats Length: 1
            Elliptic curves point formats (1)
        Extension: signature_algorithms (len=20)
            Type: signature_algorithms (13)
            Length: 20
            Signature Hash Algorithms Length: 18
            Signature Hash Algorithms (9 algorithms)
        Extension: session_ticket (len=0)
            Type: session_ticket (35)
            Length: 0
            Data (0 bytes)
        Extension: extended_master_secret (len=0)
            Type: extended_master_secret (23)
            Length: 0
        Extension: renegotiation_info (len=1)
            Type: renegotiation_info (65281)
            Length: 1
            Renegotiation Info extension

I've tried a few different methods of generating the certificate. This log was using a cert generated with power shell on the same host running the test:

New-SelfSignedCertificate -CertStoreLocation Cert:\LocalMachine\My -DnsName "host.com" -FriendlyName "host.com"  -NotAfter (Get-Date).AddYears(10)

I've also tried using the certificate which is installed with our application but i get the same result.

When doing the same test on two windows 10 hosts where the certificates has been generated in the same way. One of them also generates a TCP Reset response on Client Hello where as the other works as intended and I am able to authenticate with the IDP.

Calling the IDP using HTTP always works.

Could this issue be because I am using the loopback interface while setting the DNS name to "host.com"? In that case, what should the DNS be? Using 'localhost' and '127.0.0.1' both results in TCP reset. Or could there be some configuration blocking the connection? One would think that since generating the certificate on the same that host that is running the application and service all ciphers would be supported by the server.