Port mapping between machines that used docker containers

I have 2 machines.

• 192.168.0.10
• 192.168.0.20

Which run a few docker services.

On the 2nd machine I'm seeing incoming traffic from a service on the 1st machine on a few ports.

But when I check on that 1st host, I'm not seeing those ports in use.

So for instance I'll be seeing:

 192.168.0.10:48002 ->  192.168.0.20
 192.168.0.10:51364 ->  192.168.0.20
 192.168.0.10:52502 ->  192.168.0.20

But when I go on 192.168.0.10 to check for those ports, they're not shown to be in use.

So this comes up empty:

sudo netstat -tulpan | grep '52502'

What could be causing this and how do I investigate it? I suspect it has to do with docker containers and port mapping, maybe?

EDIT:

Scanning for the service itself shows something like this:

tcp6       0      0 127.0.0.1:48806         127.0.0.1:8088          ESTABLISHED 1065071/java

The internal port going to another port on the docker network... and then nothing. Am I to assume this is some kind of tunnel via which all other traffic goes? How do I map this?

The container traffic is not originating or terminating on the Docker host because it's routed from or to Docker containers. The host itself acts as a router and has no socket in use for this traffic (the exception being docker-proxy to handle NAT hairpinning when not disabled with --userland-proxy=false, in which case NAT hairpinning would be done with iptables and route_localnet).

As Docker uses iptables and NAT, all this is tracked by Netfilter's conntrack.

To see tracked flows, use the conntrack command (install the conntrack package if needed). As all this traffic is NAT-ed, this command will display all the relevant flows:

conntrack -L --any-nat -d 192.168.0.20

It should be run on the system with address 192.168.0.10, displaying the source NAT flows (container as client) seen from the system with address 192.168.0.20.

If the destination application on the system with address 192.168.0.20 is also in a Docker container, the same command when run on that system will display destination NAT flows (container as server). If not, there will be nothing displayed since there's no NAT to display.