Windows Docker slow socket connect

I have two docker containers which are trying to talk to another using web requests, but are experiencing random latency. While debugging, I've broken the issue down to slow socket connection times, by running a simple socketserver in one

import socket
serversocket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
serversocket.bind(('0.0.0.0', 80))
serversocket.listen(5)
while True:
    (clientsocket, address) = serversocket.accept()
    print((clientsocket, address))
    clientsocket.shutdown(socket.SHUT_RDWR)
    clientsocket.close()

and repeatedly binding to it from the other, printing out how long it takes to connect

import socket
a=time.time()
s=socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.connect(("other-container", 80))
print(time.time() - a)

What I'm seeing is connection times like:
0.0009965896606445312
0.0010194778442382812
0.0009961128234863281
0.0009963512420654297
0.0010194778442382812
0.0010199546813964844
3.0038609504699707
0.0005102157592773438

What's interesting, is that if I use 2 clients and repeatedly connect with one until it experiences this mysterious latency, then immediately switch to the other, it is able to connect to the server (multiple times even) while the other is still blocked, trying to connect.

A few specifics about the architecture, both hosts are windows containers running in docker EE using the default docker NAT network. Also worth specifying that I'm seeing these latencies, even when connecting using direct IP addresses (no DNS)

For what it's worth, I'm also seeing this issue using Apache to reverse proxy web requests from one to the other, where most requests complete instantaneously, whereas others hang for seconds (my original issue)

Edit: Further along in my testing, I've also confirmed that the problem also exists if I run my little socket testing client from the host, connecting to the socketserver running in the container, as well as vice versa (server on host and client in container)