Score:7

Cannot apt-get update from inside my docker container if connected to bridge network

gr flag

I am trying to understand, why I am unable to run apt-get update from within my docker containers, either debian:latest or ubuntu:latest.

  • I am able to ping 8.8.8.8 or other address from within the container
  • I am able to ping google.com or other domains from within the container
  • apt-get update works if I start the container with --network host
  • apt-get update does not work if the container is connected to the default bridge network or any other user defined network with adapter set to bridge
  • The host is a virtual machine managed by openstack
  • Everything works as expected on my computer (not openstack, not virtual)

apt-get update will time out with:

root@66230c3e7572:/# apt update
Err:1 http://deb.debian.org/debian buster InRelease                           
  Connection failed [IP: 199.232.138.132 80]
Err:2 http://security.debian.org/debian-security buster/updates InRelease
  Connection failed [IP: 151.101.194.132 80]
Err:3 http://deb.debian.org/debian buster-updates InRelease              
  Connection failed [IP: 199.232.138.132 80]
Reading package lists... Done
Building dependency tree       
Reading state information... Done
All packages are up to date.
W: Failed to fetch http://deb.debian.org/debian/dists/buster/InRelease  Connection failed [IP: 199.232.138.132 80]
W: Failed to fetch http://security.debian.org/debian-security/dists/buster/updates/InRelease  Connection failed [IP: 151.101.194.132 80]
W: Failed to fetch http://deb.debian.org/debian/dists/buster-updates/InRelease  Connection failed [IP: 199.232.138.132 80]
W: Some index files failed to download. They have been ignored, or old ones used instead.

I would like to find out why apt-get udpate is not working while I'm connected to any bridge network on the virtual machine. So any hints on how to debug this problem are very appreciated.

Michael Hampton avatar
cz flag
Did you do this yesterday morning? Try again today. This looks related to the Fastly outage.
gr flag
Fastly was not the problem, the same behavior still persists. I noticed something else: While I can run `wget google.com` and get the `index.html` I am unable to `wget https://file-examples-com.github.io/uploads/2018/04/file_example_AVI_1920_2_3MG.avi` for example.
Score:4
gr flag

After hours and hours I was able to solve the problem

The MTU of the docker's bridge network has to match to the MTU of host's the network adapter

In my case the MTU of eth0 (host) was set to 1450 while MTU of docker0 was set to 1500

You can change the MTU by either

In case you don't have a /etc/docker/daemon.json just create one:

# /etc/docker/daemon.json
# adjust the MTU accordingly to the hosts network adapter

{
    "mtu":1450
}

Don't forget to restart docker.service: systemctl restart docker.service

Further details:

If you wan't to check the settings use ip and compare the mtu value

$ ip a

2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> **mtu 1450** qdisc fq_codel state UP group default qlen 1000
    ...
3: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> **mtu 1500** qdisc noqueue state DOWN group default 
    ...

Note, that docker0 states always 1500 and changes it's value only if a container ist connected to that network

$ ip a
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc fq_codel state UP group default qlen 1000
    ...
3: docker0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP group default 
    ...
17: vethe4b452f@if16: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue master docker0 state UP group default 
    ...

Custom Network

I also tried to just create a custom network with a defined MTU instead of setting the MTU via /etc/docker/daemon.json. This did not work and I don't know why

docker network create --opt com.docker.network.mtu=1450 CustomMTU

iq flag
OH MY GOD thank you for this ...I was getting really tired from these random fails. By the way, I think that almost everybody with WireGuard VPN will have this issue, so yeah if you are reading this and are using WG VPN, just check that you have this configured correctly.
mangohost

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.