Join Log in
Score:1
Boris avatar Ubuntu

No network access from within Docker container

in flag
Boris

I can't access the Internet or even the local network (a network printer) from within a container. For example, pinging google.com's IP:

docker run --rm busybox ping -c 2 142.250.186.142

outputs:

PING 142.250.186.142 (142.250.186.142): 56 data bytes

--- 142.250.186.142 ping statistics ---
2 packets transmitted, 0 packets received, 100% packet loss

With --network=host, it works:

docker run --rm --network=host busybox ping -c 2 142.250.186.142

Host is Ubuntu 22.10. And it used to work until last week. My colleagues using Windows as an host can ping fine.

I tried:

More details:

$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 22.10
Release:        22.10
Codename:       kinetic

$ docker info
Client:
 Context:    default
 Debug Mode: false
 Plugins:
  app: Docker App (Docker Inc., v0.9.1-beta3)
  buildx: Docker Buildx (Docker Inc., v0.9.1-docker)
  compose: Docker Compose (Docker Inc., v2.12.2)
  scan: Docker Scan (Docker Inc., v0.21.0)

Server:
 Containers: 1
  Running: 0
  Paused: 0
  Stopped: 1
 Images: 1
 Server Version: 20.10.21
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: systemd
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: a05d175400b1145e5e6a735a6710579d181e7fb0
 runc version: v1.1.4-0-g5fd4c4d
 init version: de40ad0
 Security Options:
  apparmor
  seccomp
   Profile: default
  cgroupns
 Kernel Version: 5.19.0-26-generic
 Operating System: Ubuntu 22.10
 OSType: linux
 Architecture: x86_64
 CPUs: 8
 Total Memory: 15.4GiB
 Name: boris-ThinkPad-T480
 ID: ZN24:57QW:KT4M:J5H6:BIHZ:BF6I:KDV2:EAM6:CESL:UWNE:43AL:LUAJ
 Docker Root Dir: /var/lib/docker
 Debug Mode: true
  File Descriptors: 24
  Goroutines: 34
  System Time: 2022-12-12T20:08:42.528459605+01:00
  EventsListeners: 0
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

$ docker version
Client: Docker Engine - Community
 Version:           20.10.21
 API version:       1.41
 Go version:        go1.18.7
 Git commit:        baeda1f
 Built:             Tue Oct 25 18:01:58 2022
 OS/Arch:           linux/amd64
 Context:           default
 Experimental:      true

Server: Docker Engine - Community
 Engine:
  Version:          20.10.21
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.18.7
  Git commit:       3056208
  Built:            Tue Oct 25 17:59:49 2022
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.6.12
  GitCommit:        a05d175400b1145e5e6a735a6710579d181e7fb0
 runc:
  Version:          1.1.4
  GitCommit:        v1.1.4-0-g5fd4c4d
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

$ cat /proc/sys/net/ipv4/ip_forward
1

$ iptables -Z && iptables -nvL > before.txt && docker run --rm busybox ping -c 3 142.250.186.142 > ping.txt  &&  iptables -nvL > after.txt && cat before.txt && cat ping.txt && diff before.txt after.txt
Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 LIBVIRT_INP  all  --  *      *       0.0.0.0/0            0.0.0.0/0           

Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 DOCKER-USER  all  --  *      *       0.0.0.0/0            0.0.0.0/0           
    0     0 DOCKER-ISOLATION-STAGE-1  all  --  *      *       0.0.0.0/0            0.0.0.0/0           
    0     0 ACCEPT     all  --  *      docker0  0.0.0.0/0            0.0.0.0/0            ctstate RELATED,ESTABLISHED
    0     0 DOCKER     all  --  *      docker0  0.0.0.0/0            0.0.0.0/0           
    0     0 ACCEPT     all  --  docker0 !docker0  0.0.0.0/0            0.0.0.0/0           
    0     0 ACCEPT     all  --  docker0 docker0  0.0.0.0/0            0.0.0.0/0           
    0     0 LIBVIRT_FWX  all  --  *      *       0.0.0.0/0            0.0.0.0/0           
    0     0 LIBVIRT_FWI  all  --  *      *       0.0.0.0/0            0.0.0.0/0           
    0     0 LIBVIRT_FWO  all  --  *      *       0.0.0.0/0            0.0.0.0/0           

Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 LIBVIRT_OUT  all  --  *      *       0.0.0.0/0            0.0.0.0/0           

Chain DOCKER (1 references)
 pkts bytes target     prot opt in     out     source               destination         

Chain DOCKER-ISOLATION-STAGE-1 (1 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 DOCKER-ISOLATION-STAGE-2  all  --  docker0 !docker0  0.0.0.0/0            0.0.0.0/0           
    0     0 RETURN     all  --  *      *       0.0.0.0/0            0.0.0.0/0           

Chain DOCKER-ISOLATION-STAGE-2 (1 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 DROP       all  --  *      docker0  0.0.0.0/0            0.0.0.0/0           
    0     0 RETURN     all  --  *      *       0.0.0.0/0            0.0.0.0/0           

Chain DOCKER-USER (1 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 RETURN     all  --  *      *       0.0.0.0/0            0.0.0.0/0           

Chain LIBVIRT_FWI (1 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 ACCEPT     all  --  *      virbr0  0.0.0.0/0            192.168.122.0/24     ctstate RELATED,ESTABLISHED
    0     0 REJECT     all  --  *      virbr0  0.0.0.0/0            0.0.0.0/0            reject-with icmp-port-unreachable

Chain LIBVIRT_FWO (1 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 ACCEPT     all  --  virbr0 *       192.168.122.0/24     0.0.0.0/0           
    0     0 REJECT     all  --  virbr0 *       0.0.0.0/0            0.0.0.0/0            reject-with icmp-port-unreachable

Chain LIBVIRT_FWX (1 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 ACCEPT     all  --  virbr0 virbr0  0.0.0.0/0            0.0.0.0/0           

Chain LIBVIRT_INP (1 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 ACCEPT     udp  --  virbr0 *       0.0.0.0/0            0.0.0.0/0            udp dpt:53
    0     0 ACCEPT     tcp  --  virbr0 *       0.0.0.0/0            0.0.0.0/0            tcp dpt:53
    0     0 ACCEPT     udp  --  virbr0 *       0.0.0.0/0            0.0.0.0/0            udp dpt:67
    0     0 ACCEPT     tcp  --  virbr0 *       0.0.0.0/0            0.0.0.0/0            tcp dpt:67

Chain LIBVIRT_OUT (1 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 ACCEPT     udp  --  *      virbr0  0.0.0.0/0            0.0.0.0/0            udp dpt:53
    0     0 ACCEPT     tcp  --  *      virbr0  0.0.0.0/0            0.0.0.0/0            tcp dpt:53
    0     0 ACCEPT     udp  --  *      virbr0  0.0.0.0/0            0.0.0.0/0            udp dpt:68
    0     0 ACCEPT     tcp  --  *      virbr0  0.0.0.0/0            0.0.0.0/0            tcp dpt:68
PING 142.250.186.142 (142.250.186.142): 56 data bytes

--- 142.250.186.142 ping statistics ---
3 packets transmitted, 0 packets received, 100% packet loss
1c1
< Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
---
> Chain INPUT (policy ACCEPT 220 packets, 56220 bytes)
3c3
<     0     0 LIBVIRT_INP  all  --  *      *       0.0.0.0/0            0.0.0.0/0           
---
>   220 56220 LIBVIRT_INP  all  --  *      *       0.0.0.0/0            0.0.0.0/0           
5c5
< Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
---
> Chain FORWARD (policy ACCEPT 3 packets, 252 bytes)
7,8c7,8
<     0     0 DOCKER-USER  all  --  *      *       0.0.0.0/0            0.0.0.0/0           
<     0     0 DOCKER-ISOLATION-STAGE-1  all  --  *      *       0.0.0.0/0            0.0.0.0/0           
---
>     6   504 DOCKER-USER  all  --  *      *       0.0.0.0/0            0.0.0.0/0           
>     6   504 DOCKER-ISOLATION-STAGE-1  all  --  *      *       0.0.0.0/0            0.0.0.0/0           
11c11
<     0     0 ACCEPT     all  --  docker0 !docker0  0.0.0.0/0            0.0.0.0/0           
---
>     3   252 ACCEPT     all  --  docker0 !docker0  0.0.0.0/0            0.0.0.0/0           
13,15c13,15
<     0     0 LIBVIRT_FWX  all  --  *      *       0.0.0.0/0            0.0.0.0/0           
<     0     0 LIBVIRT_FWI  all  --  *      *       0.0.0.0/0            0.0.0.0/0           
<     0     0 LIBVIRT_FWO  all  --  *      *       0.0.0.0/0            0.0.0.0/0           
---
>     3   252 LIBVIRT_FWX  all  --  *      *       0.0.0.0/0            0.0.0.0/0           
>     3   252 LIBVIRT_FWI  all  --  *      *       0.0.0.0/0            0.0.0.0/0           
>     3   252 LIBVIRT_FWO  all  --  *      *       0.0.0.0/0            0.0.0.0/0           
17c17
< Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes)
---
> Chain OUTPUT (policy ACCEPT 267 packets, 34045 bytes)
19c19
<     0     0 LIBVIRT_OUT  all  --  *      *       0.0.0.0/0            0.0.0.0/0           
---
>   267 34045 LIBVIRT_OUT  all  --  *      *       0.0.0.0/0            0.0.0.0/0           
26,27c26,27
<     0     0 DOCKER-ISOLATION-STAGE-2  all  --  docker0 !docker0  0.0.0.0/0            0.0.0.0/0           
<     0     0 RETURN     all  --  *      *       0.0.0.0/0            0.0.0.0/0           
---
>     3   252 DOCKER-ISOLATION-STAGE-2  all  --  docker0 !docker0  0.0.0.0/0            0.0.0.0/0           
>     6   504 RETURN     all  --  *      *       0.0.0.0/0            0.0.0.0/0           
32c32
<     0     0 RETURN     all  --  *      *       0.0.0.0/0            0.0.0.0/0           
---
>     3   252 RETURN     all  --  *      *       0.0.0.0/0            0.0.0.0/0           
36c36
<     0     0 RETURN     all  --  *      *       0.0.0.0/0            0.0.0.0/0           
---
>     6   504 RETURN     all  --  *      *       0.0.0.0/0            0.0.0.0/0

$ sudo iptables -L -v -n -t  nat
Chain PREROUTING (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         
  129 46233 DOCKER     all  --  *      *       0.0.0.0/0            0.0.0.0/0            ADDRTYPE match dst-type LOCAL

Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         

Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 DOCKER     all  --  *      *       0.0.0.0/0           !127.0.0.0/8          ADDRTYPE match dst-type LOCAL

Chain POSTROUTING (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         
    6   504 MASQUERADE  all  --  *      !docker0  172.17.0.0/16        0.0.0.0/0           
18336 2849K LIBVIRT_PRT  all  --  *      *       0.0.0.0/0            0.0.0.0/0           

Chain DOCKER (2 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 RETURN     all  --  docker0 *       0.0.0.0/0            0.0.0.0/0           

Chain LIBVIRT_PRT (1 references)
 pkts bytes target     prot opt in     out     source               destination         
   29  2852 RETURN     all  --  *      *       192.168.122.0/24     224.0.0.0/24        
    0     0 RETURN     all  --  *      *       192.168.122.0/24     255.255.255.255     
    2   120 MASQUERADE  tcp  --  *      *       192.168.122.0/24    !192.168.122.0/24     masq ports: 1024-65535
    6  3810 MASQUERADE  udp  --  *      *       192.168.122.0/24    !192.168.122.0/24     masq ports: 1024-65535
    2    64 MASQUERADE  all  --  *      *       192.168.122.0/24    !192.168.122.0/24
10616
5 + 11
dummyuser avatar
uy flag
dummyuser
Hi Boris, welcome to AskUbuntu. We will need some more info. 1) please ensure your system is fully patched 2) did you follow the instructions at [install Docker](https://docs.docker.com/engine/install/ubuntu/#install-using-the-repository) for docker install ? if not … uninstall and install it according the link.
0
Reply
dummyuser avatar
uy flag
dummyuser
3) If the issue is still present, please edit your question and add the output of the following commands `lsb_release -a`, `docker info`, `docker version`, `cat /proc/sys/net/ipv4/ip_forward` and `iptables -Z && iptables -nvL > before.txt && docker run --rm busybox ping -c 3 142.250.186.142 > ping.txt && iptables -nvL > after.txt && cat before.txt && cat ping.txt && diff before.txt after.txt`
0
Reply
Boris avatar
in flag
Boris
Hi @dummyuser, thanks for helping. 1) system is up to date 2) Docker was installed and reinstalled following the instructions you linked 3) I edited my question with the command outputs
0
Reply
dummyuser avatar
uy flag
dummyuser
Hi Boris, there is a KVM virtualization running on the server too? The data you provided looks good. I do not really like docker an kvm on the same server. Let us do some more stuff. You can get a list of your docker networks with `docker network ls`. The first column is the network ID. Now start the container with a long run `docker run busybox ping -c 20 -i5 142.250.186.142 &` please check to which Network the container is connected. It should be the network named bridge. `docker network inspect <NETWORK ID OF BRIDGE>` you will see a container section at the end of the output
0
Reply
dummyuser avatar
uy flag
dummyuser
Please dump on your LAN interface (Name to be replaced) Do you see the icmp packets to 142.250.186.142? is the source IP your LAN IP? `tcpdump -i <LANINTERFACE> -n -c 30 icmp` please provide the output of `iptables -L -v -n -t nat`
0
Reply
Boris avatar
in flag
Boris
@dummyuser, yes, KVM is installed. I use it for Android Studio, but I can disable/remove it if you think this might be the problem. I installed it well before this connectivity issue started though. Indeed the container is connected to the bridge network. WIth ` tcpdump` I do see packets going from the IP assigned to my laptop to 142.250.186.142, and replies. Adding the output of `iptables -L -v -n -t nat` to the question.
0
Reply
dummyuser avatar
uy flag
dummyuser
Very strange behaviour. I’ve never seen it. The response packet arrives but it is not send to the container. The NAT table looks good, too. I would recommend disabling (at least temp) the KVM stuff for testing. But right now, I’m bit lost.
0
Reply
Boris avatar
in flag
Boris
Bad news: I uninstalled KVM (`sudo apt purge qemu-kvm libvirt-daemon-system libvirt-clients bridge-utils` and rebooted the system), still no network.
0
Reply
dummyuser avatar
uy flag
dummyuser
have a look at [own Bridgeinterface](https://superuser.com/questions/1579858/docker-bridge-network-sporadically-loosing-packets) but is't just a wild guess ... setup your own bidge interface `docker network create --subnet=172.20.0.0/24 --gateway=172.20.0.1 docker20`
0
Reply
Boris avatar
in flag
Boris
A wild but spot-on guess! I followed the steps in the [answer](https://superuser.com/a/1580017/1755714) expect that I don't have to attach the containers to the new network for them to have connectivity. Not using `--net <network>` nor `{ "bridge": "docker1" }` in daemon.json. I inspected the working container and it stills uses the network named "bridge". Thank you so much, my issue is solved. I let you post the answer? Happy to run more diagnostic commands if you want to understand more.
0
Reply
dummyuser avatar
uy flag
dummyuser
That’s the second time today I can confirm IT is crazy. But cool. Have fun with docker. It’s a very cool tool.
0
Reply
Score:1
avatar Ubuntu
in flag
vipulgupta2048

I encountered this issue, it turned out to be the Docker + iptables conflict when the networking was set to host. The host could be using the newer nftables whereas Docker could still be using the iptables-legacy. This GitHub issue issues to explain the problem. Additionally, our setup was Docker-in-Docker where the container inside the container had no internet.

Verify if you have this issue

Run the following commands in your container to verify if this is the case:

ping google.com  # Should fail since your container will have no internet
iptables -L
iptables-legacy -L

If the Docker rules are not present in iptables -L command output, then the conflict exists and we need to solve it.

Solutions

Using update-alternatives to switch to iptables-legacy which would also be available in your distribution: https://wiki.debian.org/iptables#Current_status

Be sure to restart Docker after changing to iptables-legacy for this work,

update-alternatives --set iptables /usr/sbin/iptables-legacy
update-alternatives --set ip6tables /usr/sbin/ip6tables-legacy

Alternative tips

  1. If you are having this problem while building a Docker-in-Docker container then an easy workaround is to build using --network=host.

  2. Bash solution to solve the conflict: https://github.com/garutilorenzo/iptables-docker if nothing works. Haven't tried, let me know if this worked for you.

+ 0
Score:0
dummyuser avatar Ubuntu
uy flag
dummyuser

After some debug commands it turned out the container does send out the icmp request packet, and the response packet arrives at the host.

Right now it is unclear why the response icmp packet is not forwarded to the container. Iptables (filter and nat) looked ok. Something inside docker failed. The system was up to date and docker was installed in the latest (current) version.

Solution:

Setup an own bridged network. Like network create --subnet=172.20.0.0/24 --gateway=172.20.0.1 docker20 read documentation for details.

See answer answer for details.

Disclaimer: I’m not able to reproduce this issue. The root cause is still unclear but will not be investigated any more.

Recommendation: instead of starting containers directly with docker run use docker compose files instead. A docker compose file would look like

version: "3.1"
services:
  alpine-sshd:
    image: busybox 
    container_name: busybox    
    command:   [ping , "-c2", "142.250.186.142"]
# optional own network   
networks:
   default:
     external: true
     name: <NETWORKNAME>
+ 0
Score:0
dpooley avatar Ubuntu
bd flag
dpooley

I had this same issue, and resolved it by disabling IPv6. This can be done via sysctl.

  1. Add the following to /etc/sysctl.conf
net.ipv6.conf.all.disable_ipv6=1
net.ipv6.conf.default.disable_ipv6=1
net.ipv6.conf.lo.disable_ipv6=1
  1. Apply changes with sudo sysctl -p
+ 0
Score:0
David W avatar Ubuntu
li flag
David W

I had the same problem on Ubuntu 22.04 LTS. The containers on my machines spun up via docker compose had no outbound connectivity.

I noticed that docker networks created via "docker compose" did not have any of the options enabled, when compared to the default "bridge" network that containers attach to via "docker run".

For example, running "docker network inspect bridge" shows the following options.

<<snipped>>
        "Options": {
            "com.docker.network.bridge.default_bridge": "true",
            "com.docker.network.bridge.enable_icc": "true",
            "com.docker.network.bridge.enable_ip_masquerade": "true",
            "com.docker.network.bridge.host_binding_ipv4": "0.0.0.0",
            "com.docker.network.bridge.name": "docker0",
            "com.docker.network.driver.mtu": "1500"
        },
<snipped>>

I was able to fix the "docker compose" created network by explicitly creating a network and setting those same options in the docker-compose file.

version: '3'
services:

  busybox:
    hostname: busybox
    image: "busybox:latest"
    restart: always
    command: sh -c "while true; do date; sleep 60; done"
    links:
      - influxdb2
    networks:
      - default

# "docker compose" on my ubuntu 20.04 seems to not declare default options.
#   This is needed... unclear why
networks:
  default:
    driver: bridge
    driver_opts:
      com.docker.network.driver.mtu: "1450"
      com.docker.network.bridge.default_bridge: "true"
      com.docker.network.bridge.enable_icc: "true"
      com.docker.network.bridge.enable_ip_masquerade: "true"
      com.docker.network.bridge.host_binding_ipv4: "0.0.0.0"
      com.docker.network.bridge.name: "docker0"
      com.docker.network.driver.mtu: "1500"

From there, I'm able to do a:

docker compose up -d
docker compose exec -it busybox ping google.com
+ 0
Score:0
Hari Mathur avatar Ubuntu
tn flag
Hari Mathur

If you can access the network using the host's network stack (--network=host) but cannot without it, it would mean that the container network stack is not configured properly (IP address, default gateway..)

There could also be issues with the firewall which can prevent the container from accessing the network.

You could check the network configurations of your container using:

docker inspect <your_container>

You can also check the listings of all your active networks with

docker network ls

Could provide insight if there is any network related issues preventing the container from accessing the internet

+ 0
ธงชาติไทย
Elon Musk
I sit in a Tesla and translated this thread with Ai:

mangohost

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.