http timeout with docker bridge network, but port 80 is opened in firewall

I'm trying to install mailcow-dockerized in my server, but I'm having trouble with Docker network. I tried several ways, but I get lots of connection timeouts in containers.

In order to breakdown the issue, I decided to leave Mailcow behind and installed just Docker to try to identify the source of these connection timeouts.

So, I've installed a fresh Ubuntu 20.04 image from my VPS vendor and setup ufw firewall like this:

sudo ufw default allow outgoing
sudo ufw default deny incoming
sudo ufw limit ssh
sudo ufw allow ssh
sudo ufw allow http
sudo ufw allow https
sudo ufw allow smtp
sudo ufw allow submission
sudo ufw allow submissions
sudo ufw allow pop3
sudo ufw allow pop3s
sudo ufw allow imap2
sudo ufw allow imaps
sudo ufw allow 4190/tcp
sudo ufw allow 8080/tcp
sudo systemctl enable ufw
sudo ufw enable

I installed Docker with the script get-docker.sh from https://get.docker.com

Then I enabled IPV6 in file /etc/docker/daemon.json with:

{
  "ipv6": true,
  "fixed-cidr-v6": "2001:db8:1::/64"
}

Rebooted the server and created a docker-compose.yaml:

version: '2.1'
services:
    S2:
      image: nginx:latest
      ports:
        - 80:80
      restart: always
      networks:
        n1:
          ipv4_address: 172.22.1.254
          aliases:
            - s2
    S3:
      image: nginx:latest
      ports:
        - 8080:80
      restart: always
      networks:
        n1:
          ipv4_address: 172.22.1.248
          aliases:
            - s3
networks:
  n1:
    driver: bridge
    driver_opts:
      com.docker.network.bridge.name: n1
    enable_ipv6: true
    ipam:
      driver: default
      config:
        - subnet: 172.22.1.0/24
        - subnet: fd4d:6169:6c63:6f77::/64

This network configuration I got from docker-compose.yaml in Mailcow git and changed it to adapt to my test.

And I run the containers with docker-compose up -d.

When I do a curl localhost 80 in the host server, it returns the default index.html content from Nginx, but... the connection gets on hold for some minutes and then shell shows the following message at the end:

curl: (28) Failed to connect to 80 port 80: Connection timed out

When I run curl <myservername.com> 80 in my local computer, it returns the index.html content from default Nginx too, but with the message at the end:

curl: (7) Failed to connect to 0.0.0.80 port 80 after 0 ms: Network unreachable

Any clue about why I'm getting these errors?

PS: my ufw status:

# ufw status verbose
Status: active
Logging: on (low)
Default: deny (incoming), allow (outgoing), deny (routed)
New profiles: skip

To                         Action      From
--                         ------      ----
22/tcp                     ALLOW IN    Anywhere
80/tcp                     ALLOW IN    Anywhere
443/tcp                    ALLOW IN    Anywhere
25/tcp                     ALLOW IN    Anywhere
587/tcp                    ALLOW IN    Anywhere
465/tcp                    ALLOW IN    Anywhere
110/tcp                    ALLOW IN    Anywhere
995/tcp                    ALLOW IN    Anywhere
143/tcp                    ALLOW IN    Anywhere
993/tcp                    ALLOW IN    Anywhere
4190/tcp                   ALLOW IN    Anywhere
8080/tcp                   ALLOW IN    Anywhere
22/tcp (v6)                ALLOW IN    Anywhere (v6)
80/tcp (v6)                ALLOW IN    Anywhere (v6)
443/tcp (v6)               ALLOW IN    Anywhere (v6)
25/tcp (v6)                ALLOW IN    Anywhere (v6)
587/tcp (v6)               ALLOW IN    Anywhere (v6)
465/tcp (v6)               ALLOW IN    Anywhere (v6)
110/tcp (v6)               ALLOW IN    Anywhere (v6)
995/tcp (v6)               ALLOW IN    Anywhere (v6)
143/tcp (v6)               ALLOW IN    Anywhere (v6)
993/tcp (v6)               ALLOW IN    Anywhere (v6)
4190/tcp (v6)              ALLOW IN    Anywhere (v6)
8080/tcp (v6)              ALLOW IN    Anywhere (v6)

And lsof results:

# lsof -i -P -n | grep LISTEN
sshd       967 root    3u  IPv4  35459      0t0  TCP *:22 (LISTEN)
sshd       967 root    4u  IPv6  35461      0t0  TCP *:22 (LISTEN)
docker-pr 1290 root    4u  IPv4  39102      0t0  TCP *:80 (LISTEN)
docker-pr 1308 root    4u  IPv6  38124      0t0  TCP *:80 (LISTEN)
docker-pr 1322 root    4u  IPv4  38165      0t0  TCP *:8080 (LISTEN)
docker-pr 1328 root    4u  IPv6  38172      0t0  TCP *:8080 (LISTEN)

Termshark monitoring when running curl localhost 80 in host:

 No. -  Time -   Source -       Destination -  Protocol -   Length - Info -                                      
 1      0.000000 fd4d:6169:6c63 fd4d:6169:6c63 TCP          94       39946 → 80 [SYN] Seq=0 Win=64800 Len=0 MSS= 
 2      0.000047 fd4d:6169:6c63 fd4d:6169:6c63 TCP          94       80 → 39946 [SYN, ACK] Seq=0 Ack=1 Win=64260
 3      0.000088 fd4d:6169:6c63 fd4d:6169:6c63 TCP          86       39946 → 80 [ACK] Seq=1 Ack=1 Win=64896 Len=
 4      0.000516 fd4d:6169:6c63 fd4d:6169:6c63 HTTP         159      GET / HTTP/1.1
 5      0.000544 fd4d:6169:6c63 fd4d:6169:6c63 TCP          86       80 → 39946 [ACK] Seq=1 Ack=74 Win=64256 Len
 6      0.000765 fd4d:6169:6c63 fd4d:6169:6c63 TCP          324      HTTP/1.1 200 OK  [TCP segment of a reassemb
 7      0.000791 fd4d:6169:6c63 fd4d:6169:6c63 TCP          86       39946 → 80 [ACK] Seq=74 Ack=239 Win=64768 L
 8      0.000821 fd4d:6169:6c63 fd4d:6169:6c63 HTTP         701      HTTP/1.1 200 OK  (text/html)
 9      0.000829 fd4d:6169:6c63 fd4d:6169:6c63 TCP          86       39946 → 80 [ACK] Seq=74 Ack=854 Win=64256 L 
 10     65.01291 fd4d:6169:6c63 fd4d:6169:6c63 TCP          86       80 → 39946 [FIN, ACK] Seq=854 Ack=74 Win=64
 11     65.05677 fd4d:6169:6c63 fd4d:6169:6c63 TCP          86       39946 → 80 [ACK] Seq=74 Ack=855 Win=64256 L
 12     130.8576 fd4d:6169:6c63 fd4d:6169:6c63 TCP          86       39946 → 80 [FIN, ACK] Seq=74 Ack=855 Win=64
 13     130.8577 fd4d:6169:6c63 fd4d:6169:6c63 TCP          74       80 → 39946 [RST] Seq=855 Win=0 Len=0
 14     131.0647 fd4d:6169:6c63 fd4d:6169:6c63 TCP          86       [TCP Retransmission] 39946 → 80 [FIN, ACK]
 15     131.0648 fd4d:6169:6c63 fd4d:6169:6c63 TCP          74       80 → 39946 [RST] Seq=855 Win=0 Len=0        
 16     131.2727 fd4d:6169:6c63 fd4d:6169:6c63 TCP          86       [TCP Retransmission] 39946 → 80 [FIN, ACK]
 17     131.2728 fd4d:6169:6c63 fd4d:6169:6c63 TCP          74       80 → 39946 [RST] Seq=855 Win=0 Len=0
 18     131.6888 fd4d:6169:6c63 fd4d:6169:6c63 TCP          86       [TCP Retransmission] 39946 → 80 [FIN, ACK]  
 19     131.6888 fd4d:6169:6c63 fd4d:6169:6c63 TCP          74       80 → 39946 [RST] Seq=855 Win=0 Len=0
 20     132.5208 fd4d:6169:6c63 fd4d:6169:6c63 TCP          86       [TCP Retransmission] 39946 → 80 [FIN, ACK]
 21     132.5209 fd4d:6169:6c63 fd4d:6169:6c63 TCP          74       80 → 39946 [RST] Seq=855 Win=0 Len=0
 22     134.1847 fd4d:6169:6c63 fd4d:6169:6c63 TCP          86       [TCP Retransmission] 39946 → 80 [FIN, ACK]
 23     134.1850 fd4d:6169:6c63 fd4d:6169:6c63 TCP          74       80 → 39946 [RST] Seq=855 Win=0 Len=0
 24     137.5129 fd4d:6169:6c63 fd4d:6169:6c63 TCP          86       [TCP Retransmission] 39946 → 80 [FIN, ACK]  
 25     137.5131 fd4d:6169:6c63 fd4d:6169:6c63 TCP          74       80 → 39946 [RST] Seq=855 Win=0 Len=0        

Termshark results when running curl <myserver.com> 80 in my computer

 No. -  Time -   Source -       Destination -  Protocol -   Length - Info -                                      
 1      0.000000 170.78.36.7    172.22.1.254   TCP          66       62787 → 80 [SYN] Seq=0 Win=64240 Len=0 MSS= 
 2      0.000063 172.22.1.254   170.78.36.7    TCP          66       80 → 62787 [SYN, ACK] Seq=0 Ack=1 Win=64240
 3      0.007119 170.78.36.7    172.22.1.254   TCP          54       62787 → 80 [ACK] Seq=1 Ack=1 Win=131840 Len
 4      0.009563 170.78.36.7    172.22.1.254   HTTP         133      GET / HTTP/1.1
 5      0.009628 172.22.1.254   170.78.36.7    TCP          54       80 → 62787 [ACK] Seq=1 Ack=80 Win=64256 Len
 6      0.009884 172.22.1.254   170.78.36.7    TCP          292      HTTP/1.1 200 OK  [TCP segment of a reassemb
 7      0.010001 172.22.1.254   170.78.36.7    HTTP         669      HTTP/1.1 200 OK  (text/html)
 8      0.019889 170.78.36.7    172.22.1.254   TCP          54       62787 → 80 [ACK] Seq=80 Ack=854 Win=130816
 9      0.039001 170.78.36.7    172.22.1.254   TCP          54       62787 → 80 [FIN, ACK] Seq=80 Ack=854 Win=13 
 10     0.039211 172.22.1.254   170.78.36.7    TCP          54       80 → 62787 [FIN, ACK] Seq=854 Ack=81 Win=64 
 11     0.046453 170.78.36.7    172.22.1.254   TCP          54       62787 → 80 [ACK] Seq=81 Ack=855 Win=130816  

I changed the scope of my test to a simple VPS server with NGinx installation, without Docker, and the problem of http timeout persisted so... I discovered that the problem was caused by the infrastructure of my local VPS vendor...

I changed my server to another vendor and everything worked fine...

Be careful that with some firewall Docker adds specific rules to function properly. I'm not sure if this is related also to UFW, but could be.

When this happened to me wiht Iptables, i had to add some rule to forward the incoming connection on a specific port to the specific port of my service on the Docker Network. So if i'm receiving connection on the port 80, but my dockerized service expose the port 8080, in some case even though you specify the mapping: "80:8080", there is still the need to add some forwarding rule to you firewall.

Another thing you could double check is if the host server can "call" itself.

One command that can help you to troubleshoot is:

curl -Ivvv host port