Score:0

Docker containers can't connect to IP on the Internet but the host can

ki flag

I've been scratching my head about this for a full day now, I'm hoping to get some pointers in the right direction.

From the host I can connect to the Internet just fine, but any container I start does not have connectivity.

When I run this script (172.217.170.174 is what google.com resolved to on that host)

#!/bin/bash

set -x

echo HOST

curl http://172.217.170.174/

echo CONTAINER

docker run --rm -ti --network bridge curlimages/curl curl -v http://172.217.170.174/

I get this result:

+ echo HOST
HOST
+ curl http://172.217.170.174/
<HTML><HEAD><meta http-equiv="content-type" content="text/html;charset=utf-8">
<TITLE>301 Moved</TITLE></HEAD><BODY>
<H1>301 Moved</H1>
The document has moved
<A HREF="http://www.google.com/">here</A>.
</BODY></HTML>
+ echo CONTAINER
CONTAINER
+ docker run --rm -ti --network bridge curlimages/curl curl -v http://172.217.170.174/
*   Trying 172.217.170.174:80...
* connect to 172.217.170.174 port 80 failed: Operation timed out
* Failed to connect to 172.217.170.174 port 80 after 131153 ms: Couldn't connect to server
* Closing connection 0
curl: (28) Failed to connect to 172.217.170.174 port 80 after 131153 ms: Couldn't connect to server

In other words, connecting to an external IP address works from the host but not from the container.

My iptables looks like this:

# Generated by iptables-save v1.8.7 on Tue Jun  6 00:02:11 2023
*filter
:INPUT ACCEPT [0:0]
:FORWARD DROP [0:0]
:OUTPUT ACCEPT [0:0]
:DOCKER - [0:0]
:DOCKER-ISOLATION-STAGE-1 - [0:0]
:DOCKER-ISOLATION-STAGE-2 - [0:0]
:DOCKER-USER - [0:0]
-A FORWARD -j DOCKER-USER
-A FORWARD -j DOCKER-ISOLATION-STAGE-1
-A FORWARD -o docker0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -o docker0 -j DOCKER
-A FORWARD -i docker0 ! -o docker0 -j ACCEPT
-A FORWARD -i docker0 -o docker0 -j ACCEPT
-A FORWARD -o br-a8eda64c0e60 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -o br-a8eda64c0e60 -j DOCKER
-A FORWARD -i br-a8eda64c0e60 ! -o br-a8eda64c0e60 -j ACCEPT
-A FORWARD -i br-a8eda64c0e60 -o br-a8eda64c0e60 -j ACCEPT
-A DOCKER-ISOLATION-STAGE-1 -i docker0 ! -o docker0 -j DOCKER-ISOLATION-STAGE-2
-A DOCKER-ISOLATION-STAGE-1 -i br-a8eda64c0e60 ! -o br-a8eda64c0e60 -j DOCKER-ISOLATION-STAGE-2
-A DOCKER-ISOLATION-STAGE-1 -j RETURN
-A DOCKER-ISOLATION-STAGE-2 -o docker0 -j DROP
-A DOCKER-ISOLATION-STAGE-2 -o br-a8eda64c0e60 -j DROP
-A DOCKER-ISOLATION-STAGE-2 -j RETURN
-A DOCKER-USER -j RETURN
COMMIT
# Completed on Tue Jun  6 00:02:11 2023
# Generated by iptables-save v1.8.7 on Tue Jun  6 00:02:11 2023
*nat
:PREROUTING ACCEPT [0:0]
:INPUT ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
:POSTROUTING ACCEPT [0:0]
:DOCKER - [0:0]
-A PREROUTING -m addrtype --dst-type LOCAL -j DOCKER
-A OUTPUT ! -d 127.0.0.0/8 -m addrtype --dst-type LOCAL -j DOCKER
-A POSTROUTING -s 172.17.0.0/16 ! -o docker0 -j MASQUERADE
-A POSTROUTING -s 172.18.0.0/16 ! -o br-a8eda64c0e60 -j MASQUERADE
-A DOCKER -i docker0 -j RETURN
-A DOCKER -i br-a8eda64c0e60 -j RETURN
COMMIT
# Completed on Tue Jun  6 00:02:11 2023

resolv.conf in the container (shouldn't matter since I'm connecting to an IP):

nameserver 8.8.8.8
nameserver 8.8.4.4
search .

I had just replaced this machine with a new one because the previous one started to do exactly this same thing a few weeks ago. I thought it got broken by an update or something because it was there for seven years, so I had it replaced with a new one. Installed it on my desk in our office network and everything worked, but as soon as they put it in their network this same thing happens again.

I've tried all the google results for "docker container has no internet" and variations and nothing has helped.

Are Docker containers even supposed to have Internet connection through the default bridge network? It always has worked that way without changing any Docker or network settings but now I'm starting to doubt myself?

Any pointers for where I should look next?

Much appreciated.

Things I've tried:

pt flag
Have you made any changes to the firewall rules on your system since Docker started? Does restart Docker (probably `systemctl restart docker`) change the behavior at all? Does the behavior change if change the default policy for the `FORWARD` chain from `DROP` to `ALLOW`? You've only included the rules in your `filter` table; can you replace that with the output from running `iptables-save` (which will also include rules from the `nat` table)?
djdomi avatar
za flag
I clearly missing nat, die else no ip can talk across the network or even without routing
bash avatar
ki flag
@larsks restarting docker didn't change it, neither `iptables -P FORWARD ACCEPT` (is that what you meant with ALLOW?). I've replaced the iptables output, thank you. I did run the command from my first link at the end of my post, that didn't fix it either.
I sit in a Tesla and translated this thread with Ai:

mangohost

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.