Score:0

Docker DIND cannot apt-get update but curl works

ky flag

I've deployed docker-dind on my homelab k3s cluster. And try to build docker images there. However I am getting error while performing apt-get update command. But running curl command on the same network and within the docker:dind works as well as nslookup and ping. I was able to perform similar tasks apt-get update on the host machine fine. So It's probably not a network issue.

My network does not have any proxy setup.

apt-get update on a side car dibian:bullseye and ubuntu:22.04 works fine with no issue. The issue only presented during docker build inside dind

I have tried to deploy a similar k3s setup on DigitalOcean Droplet and Docker:dind pod and everything works fine.

The image that I tried to build is a vanilla debian:bullseye. I can confirm that building on another machine on the same network works. (See sample Dockerfile below)

System Spec:

OS: Ubuntu 22.04.1 LTS (GNU/Linux 5.15.0-52-generic x86_64)
K3S: v1.25.2+k3s1
Containerd: 1.6.8-k3s1

docker info

Client:
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc., v0.9.1)
  compose: Docker Compose (Docker Inc., v2.12.2)

Server:
 Containers: 1
  Running: 0
  Paused: 0
  Stopped: 1
 Images: 2
 Server Version: 20.10.21
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 1c90a442489720eec95342e1789ee8a5e1b9536f
 runc version: v1.1.4-0-g5fd4c4d1
 init version: de40ad0
 Security Options:
  apparmor
  seccomp
   Profile: default
  cgroupns
 Kernel Version: 5.15.0-52-generic
 Operating System: Alpine Linux v3.16 (containerized)
 OSType: linux
 Architecture: x86_64
 CPUs: 48
 Total Memory: 62.67GiB
 Name: docker-dind-test
 ID: 363P:AMIK:KWWZ:KW6T:WMTT:OKJ4:GLN3:NFQW:UCVU:ISRR:CAYJ:42TC
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Registry Mirrors:
  https://mirror.gcr.io/
 Live Restore Enabled: false
 Product License: Community Engine

Dockerfile

FROM debian:bullseye
ENV HOME=/root

RUN apt-get update --allow-releaseinfo-change && \
    apt-get upgrade -y --fix-missing && \
    apt-get install -y --fix-missing --no-install-recommends \
        postgresql-client \
        curl \
        wget \
        ca-certificates \
        openssl \
        libssl-dev

RUN curl -vLI http://deb.debian.org/debian

Docker build logs

Step 1/4 : FROM debian:bullseye
bullseye: Pulling from library/debian
17c9e6141fdb: Pulling fs layer
17c9e6141fdb: Verifying Checksum
17c9e6141fdb: Download complete
17c9e6141fdb: Pull complete
Digest: sha256:bfe6615d017d1eebe19f349669de58cda36c668ef916e618be78071513c690e5
Status: Downloaded newer image for debian:bullseye
 ---> d8cacd17cfdc
Step 2/4 : ENV HOME=/root
 ---> Running in 8e91e6ef9c42
Removing intermediate container 8e91e6ef9c42
 ---> f0df6cfad00b
Step 3/4 : RUN apt-get update --allow-releaseinfo-change &&     apt-get upgrade -y --fix-missing &&     apt-get install -y --fix-missing --no-install-recommends         postgresql-client         curl         wget         ca-certificates         openssl         libssl-dev
 ---> Running in c2fcf3f7da7b
Get:1 http://deb.debian.org/debian bullseye InRelease [116 kB]
Err:1 http://deb.debian.org/debian bullseye InRelease
  Connection timed out [IP: 151.101.78.132 80]
Get:2 http://deb.debian.org/debian-security bullseye-security InRelease [48.4 kB]
Err:2 http://deb.debian.org/debian-security bullseye-security InRelease
  Connection timed out [IP: 151.101.78.132 80]

docker:dind logs

Generating RSA private key, 4096 bit long modulus (2 primes)
.....................++++
......................................................................................................................................++++
e is 65537 (0x010001)
Generating RSA private key, 4096 bit long modulus (2 primes)
..................++++
................++++
e is 65537 (0x010001)
Signature ok
subject=CN = docker:dind server
Getting CA Private Key
/certs/server/cert.pem: OK
Generating RSA private key, 4096 bit long modulus (2 primes)
.................................................................++++
...........................................++++
e is 65537 (0x010001)
Signature ok
subject=CN = docker:dind client
Getting CA Private Key
/certs/client/cert.pem: OK
time="2022-11-14T15:58:50.449742253Z" level=info msg="Starting up"
time="2022-11-14T15:58:50.451348585Z" level=warning msg="could not change group /var/run/docker.sock to docker: group docker not found"
time="2022-11-14T15:58:50.452345732Z" level=info msg="libcontainerd: started new containerd process" pid=675
time="2022-11-14T15:58:50.452384646Z" level=info msg="parsed scheme: \"unix\"" module=grpc
time="2022-11-14T15:58:50.452394281Z" level=info msg="scheme \"unix\" not registered, fallback to default scheme" module=grpc
time="2022-11-14T15:58:50.452412512Z" level=info msg="ccResolverWrapper: sending update to cc: {[{unix:///var/run/docker/containerd/containerd.sock  <nil> 0 <nil>}] <nil> <nil>}" module=grpc
time="2022-11-14T15:58:50.452424664Z" level=info msg="ClientConn switching balancer to \"pick_first\"" module=grpc
time="2022-11-14T15:58:50Z" level=warning msg="containerd config version `1` has been deprecated and will be removed in containerd v2.0, please switch to version `2`, see https://github.com/containerd/containerd/blob/main/docs/PLUGINS.md#version-header"
time="2022-11-14T15:58:50.472503771Z" level=info msg="starting containerd" revision=1c90a442489720eec95342e1789ee8a5e1b9536f version=v1.6.9
time="2022-11-14T15:58:50.489432837Z" level=info msg="loading plugin \"io.containerd.content.v1.content\"..." type=io.containerd.content.v1
time="2022-11-14T15:58:50.489560763Z" level=info msg="loading plugin \"io.containerd.snapshotter.v1.aufs\"..." type=io.containerd.snapshotter.v1
time="2022-11-14T15:58:50.495865825Z" level=info msg="skip loading plugin \"io.containerd.snapshotter.v1.aufs\"..." error="aufs is not supported (modprobe aufs failed: exit status 1 \"ip: can't find device 'aufs'\\nmodprobe: can't change directory to '/lib/modules': No such file or directory\\n\"): skip plugin" type=io.containerd.snapshotter.v1
time="2022-11-14T15:58:50.495898748Z" level=info msg="loading plugin \"io.containerd.snapshotter.v1.btrfs\"..." type=io.containerd.snapshotter.v1
time="2022-11-14T15:58:50.496083409Z" level=info msg="skip loading plugin \"io.containerd.snapshotter.v1.btrfs\"..." error="path /var/lib/docker/containerd/daemon/io.containerd.snapshotter.v1.btrfs (ext4) must be a btrfs filesystem to be used with the btrfs snapshotter: skip plugin" type=io.containerd.snapshotter.v1
time="2022-11-14T15:58:50.496104895Z" level=info msg="loading plugin \"io.containerd.snapshotter.v1.devmapper\"..." type=io.containerd.snapshotter.v1
time="2022-11-14T15:58:50.496118384Z" level=warning msg="failed to load plugin io.containerd.snapshotter.v1.devmapper" error="devmapper not configured"
time="2022-11-14T15:58:50.496148196Z" level=info msg="loading plugin \"io.containerd.snapshotter.v1.native\"..." type=io.containerd.snapshotter.v1
time="2022-11-14T15:58:50.496242442Z" level=info msg="loading plugin \"io.containerd.snapshotter.v1.overlayfs\"..." type=io.containerd.snapshotter.v1
time="2022-11-14T15:58:50.496535560Z" level=info msg="loading plugin \"io.containerd.snapshotter.v1.zfs\"..." type=io.containerd.snapshotter.v1
time="2022-11-14T15:58:50.496656657Z" level=info msg="skip loading plugin \"io.containerd.snapshotter.v1.zfs\"..." error="path /var/lib/docker/containerd/daemon/io.containerd.snapshotter.v1.zfs must be a zfs filesystem to be used with the zfs snapshotter: skip plugin" type=io.containerd.snapshotter.v1
time="2022-11-14T15:58:50.496683278Z" level=info msg="loading plugin \"io.containerd.metadata.v1.bolt\"..." type=io.containerd.metadata.v1
time="2022-11-14T15:58:50.496742065Z" level=warning msg="could not use snapshotter devmapper in metadata plugin" error="devmapper not configured"
time="2022-11-14T15:58:50.496762117Z" level=info msg="metadata content store policy set" policy=shared
time="2022-11-14T15:58:50.500867342Z" level=info msg="loading plugin \"io.containerd.differ.v1.walking\"..." type=io.containerd.differ.v1
time="2022-11-14T15:58:50.500890968Z" level=info msg="loading plugin \"io.containerd.event.v1.exchange\"..." type=io.containerd.event.v1
time="2022-11-14T15:58:50.500911054Z" level=info msg="loading plugin \"io.containerd.gc.v1.scheduler\"..." type=io.containerd.gc.v1
time="2022-11-14T15:58:50.500942937Z" level=info msg="loading plugin \"io.containerd.service.v1.introspection-service\"..." type=io.containerd.service.v1
time="2022-11-14T15:58:50.500959279Z" level=info msg="loading plugin \"io.containerd.service.v1.containers-service\"..." type=io.containerd.service.v1
time="2022-11-14T15:58:50.500980423Z" level=info msg="loading plugin \"io.containerd.service.v1.content-service\"..." type=io.containerd.service.v1
time="2022-11-14T15:58:50.500999583Z" level=info msg="loading plugin \"io.containerd.service.v1.diff-service\"..." type=io.containerd.service.v1
time="2022-11-14T15:58:50.501259889Z" level=info msg="loading plugin \"io.containerd.service.v1.images-service\"..." type=io.containerd.service.v1
time="2022-11-14T15:58:50.501283478Z" level=info msg="loading plugin \"io.containerd.service.v1.leases-service\"..." type=io.containerd.service.v1
time="2022-11-14T15:58:50.501300111Z" level=info msg="loading plugin \"io.containerd.service.v1.namespaces-service\"..." type=io.containerd.service.v1
time="2022-11-14T15:58:50.501314466Z" level=info msg="loading plugin \"io.containerd.service.v1.snapshots-service\"..." type=io.containerd.service.v1
time="2022-11-14T15:58:50.501334903Z" level=info msg="loading plugin \"io.containerd.runtime.v1.linux\"..." type=io.containerd.runtime.v1
time="2022-11-14T15:58:50.501479548Z" level=info msg="loading plugin \"io.containerd.runtime.v2.task\"..." type=io.containerd.runtime.v2
time="2022-11-14T15:58:50.501618443Z" level=info msg="loading plugin \"io.containerd.monitor.v1.cgroups\"..." type=io.containerd.monitor.v1
time="2022-11-14T15:58:50.502676656Z" level=info msg="loading plugin \"io.containerd.service.v1.tasks-service\"..." type=io.containerd.service.v1
time="2022-11-14T15:58:50.502726906Z" level=info msg="loading plugin \"io.containerd.grpc.v1.introspection\"..." type=io.containerd.grpc.v1
time="2022-11-14T15:58:50.502749179Z" level=info msg="loading plugin \"io.containerd.internal.v1.restart\"..." type=io.containerd.internal.v1
time="2022-11-14T15:58:50.502806839Z" level=info msg="loading plugin \"io.containerd.grpc.v1.containers\"..." type=io.containerd.grpc.v1
time="2022-11-14T15:58:50.502829454Z" level=info msg="loading plugin \"io.containerd.grpc.v1.content\"..." type=io.containerd.grpc.v1
time="2022-11-14T15:58:50.502847152Z" level=info msg="loading plugin \"io.containerd.grpc.v1.diff\"..." type=io.containerd.grpc.v1
time="2022-11-14T15:58:50.502862346Z" level=info msg="loading plugin \"io.containerd.grpc.v1.events\"..." type=io.containerd.grpc.v1
time="2022-11-14T15:58:50.502878765Z" level=info msg="loading plugin \"io.containerd.grpc.v1.healthcheck\"..." type=io.containerd.grpc.v1
time="2022-11-14T15:58:50.502895684Z" level=info msg="loading plugin \"io.containerd.grpc.v1.images\"..." type=io.containerd.grpc.v1
time="2022-11-14T15:58:50.502915402Z" level=info msg="loading plugin \"io.containerd.grpc.v1.leases\"..." type=io.containerd.grpc.v1
time="2022-11-14T15:58:50.502931719Z" level=info msg="loading plugin \"io.containerd.grpc.v1.namespaces\"..." type=io.containerd.grpc.v1
time="2022-11-14T15:58:50.502954823Z" level=info msg="loading plugin \"io.containerd.internal.v1.opt\"..." type=io.containerd.internal.v1
time="2022-11-14T15:58:50.503353540Z" level=info msg="loading plugin \"io.containerd.grpc.v1.snapshots\"..." type=io.containerd.grpc.v1
time="2022-11-14T15:58:50.503388785Z" level=info msg="loading plugin \"io.containerd.grpc.v1.tasks\"..." type=io.containerd.grpc.v1
time="2022-11-14T15:58:50.503411978Z" level=info msg="loading plugin \"io.containerd.grpc.v1.version\"..." type=io.containerd.grpc.v1
time="2022-11-14T15:58:50.503433166Z" level=info msg="loading plugin \"io.containerd.tracing.processor.v1.otlp\"..." type=io.containerd.tracing.processor.v1
time="2022-11-14T15:58:50.503457888Z" level=info msg="skip loading plugin \"io.containerd.tracing.processor.v1.otlp\"..." error="no OpenTelemetry endpoint: skip plugin" type=io.containerd.tracing.processor.v1
time="2022-11-14T15:58:50.503490771Z" level=info msg="loading plugin \"io.containerd.internal.v1.tracing\"..." type=io.containerd.internal.v1
time="2022-11-14T15:58:50.503521897Z" level=error msg="failed to initialize a tracing processor \"otlp\"" error="no OpenTelemetry endpoint: skip plugin"
time="2022-11-14T15:58:50.503591536Z" level=info msg="loading plugin \"io.containerd.grpc.v1.cri\"..." type=io.containerd.grpc.v1
time="2022-11-14T15:58:50.503747428Z" level=info msg="Start cri plugin with config {PluginConfig:{ContainerdConfig:{Snapshotter:overlayfs DefaultRuntimeName:runc DefaultRuntime:{Type: Path: Engine: PodAnnotations:[] ContainerAnnotations:[] Root: Options:map[] PrivilegedWithoutHostDevices:false BaseRuntimeSpec: NetworkPluginConfDir: NetworkPluginMaxConfNum:0} UntrustedWorkloadRuntime:{Type: Path: Engine: PodAnnotations:[] ContainerAnnotations:[] Root: Options:map[] PrivilegedWithoutHostDevices:false BaseRuntimeSpec: NetworkPluginConfDir: NetworkPluginMaxConfNum:0} Runtimes:map[runc:{Type:io.containerd.runc.v2 Path: Engine: PodAnnotations:[] ContainerAnnotations:[] Root: Options:map[BinaryName: CriuImagePath: CriuPath: CriuWorkPath: IoGid:0 IoUid:0 NoNewKeyring:false NoPivotRoot:false Root: ShimCgroup: SystemdCgroup:false] PrivilegedWithoutHostDevices:false BaseRuntimeSpec: NetworkPluginConfDir: NetworkPluginMaxConfNum:0}] NoPivot:false DisableSnapshotAnnotations:true DiscardUnpackedLayers:false IgnoreRdtNotEnabledErrors:false} CniConfig:{NetworkPluginBinDir:/opt/cni/bin NetworkPluginConfDir:/etc/cni/net.d NetworkPluginMaxConfNum:1 NetworkPluginConfTemplate: IPPreference:} Registry:{ConfigPath: Mirrors:map[] Configs:map[] Auths:map[] Headers:map[]} ImageDecryption:{KeyModel:node} DisableTCPService:true StreamServerAddress:127.0.0.1 StreamServerPort:0 StreamIdleTimeout:4h0m0s EnableSelinux:false SelinuxCategoryRange:1024 SandboxImage:registry.k8s.io/pause:3.6 StatsCollectPeriod:10 SystemdCgroup:false EnableTLSStreaming:false X509KeyPairStreaming:{TLSCertFile: TLSKeyFile:} MaxContainerLogLineSize:16384 DisableCgroup:false DisableApparmor:false RestrictOOMScoreAdj:false MaxConcurrentDownloads:3 DisableProcMount:false UnsetSeccompProfile: TolerateMissingHugetlbController:true DisableHugetlbController:true DeviceOwnershipFromSecurityContext:false IgnoreImageDefinedVolumes:false NetNSMountsUnderStateDir:false EnableUnprivilegedPorts:false EnableUnprivilegedICMP:false} ContainerdRootDir:/var/lib/docker/containerd/daemon ContainerdEndpoint:/var/run/docker/containerd/containerd.sock RootDir:/var/lib/docker/containerd/daemon/io.containerd.grpc.v1.cri StateDir:/var/run/docker/containerd/daemon/io.containerd.grpc.v1.cri}"
time="2022-11-14T15:58:50.503841080Z" level=info msg="Connect containerd service"
time="2022-11-14T15:58:50.503911541Z" level=info msg="Get image filesystem path \"/var/lib/docker/containerd/daemon/io.containerd.snapshotter.v1.overlayfs\""
time="2022-11-14T15:58:50.504641932Z" level=error msg="failed to load cni during init, please check CRI plugin status before setting up network for pods" error="cni config load failed: no network config found in /etc/cni/net.d: cni plugin not initialized: failed to load cni config"
time="2022-11-14T15:58:50.504786299Z" level=info msg="Start subscribing containerd event"
time="2022-11-14T15:58:50.504883053Z" level=info msg="Start recovering state"
time="2022-11-14T15:58:50.504990905Z" level=info msg="Start event monitor"
time="2022-11-14T15:58:50.505019110Z" level=info msg="Start snapshots syncer"
time="2022-11-14T15:58:50.505036477Z" level=info msg="Start cni network conf syncer for default"
time="2022-11-14T15:58:50.505051308Z" level=info msg="Start streaming server"
time="2022-11-14T15:58:50.505078162Z" level=info msg=serving... address=/var/run/docker/containerd/containerd-debug.sock
time="2022-11-14T15:58:50.505196784Z" level=info msg=serving... address=/var/run/docker/containerd/containerd.sock.ttrpc
time="2022-11-14T15:58:50.505306098Z" level=info msg=serving... address=/var/run/docker/containerd/containerd.sock
time="2022-11-14T15:58:50.505343374Z" level=info msg="containerd successfully booted in 0.033579s"
time="2022-11-14T15:58:50.517625486Z" level=info msg="Setting the storage driver from the $DOCKER_DRIVER environment variable (overlay2)"
time="2022-11-14T15:58:50.517827929Z" level=info msg="parsed scheme: \"unix\"" module=grpc
time="2022-11-14T15:58:50.517846193Z" level=info msg="scheme \"unix\" not registered, fallback to default scheme" module=grpc
time="2022-11-14T15:58:50.517866712Z" level=info msg="ccResolverWrapper: sending update to cc: {[{unix:///var/run/docker/containerd/containerd.sock  <nil> 0 <nil>}] <nil> <nil>}" module=grpc
time="2022-11-14T15:58:50.517880223Z" level=info msg="ClientConn switching balancer to \"pick_first\"" module=grpc
time="2022-11-14T15:58:50.518938275Z" level=info msg="parsed scheme: \"unix\"" module=grpc
time="2022-11-14T15:58:50.518974718Z" level=info msg="scheme \"unix\" not registered, fallback to default scheme" module=grpc
time="2022-11-14T15:58:50.519034405Z" level=info msg="ccResolverWrapper: sending update to cc: {[{unix:///var/run/docker/containerd/containerd.sock  <nil> 0 <nil>}] <nil> <nil>}" module=grpc
time="2022-11-14T15:58:50.519059990Z" level=info msg="ClientConn switching balancer to \"pick_first\"" module=grpc
time="2022-11-14T15:58:50.543863597Z" level=info msg="Loading containers: start."
time="2022-11-14T15:58:50.599392629Z" level=info msg="Default bridge (docker0) is assigned with an IP address 172.17.0.0/16. Daemon option --bip can be used to set a preferred IP address"
time="2022-11-14T15:58:50.631332394Z" level=info msg="Loading containers: done."
time="2022-11-14T15:58:50.641948188Z" level=info msg="Docker daemon" commit=3056208 graphdriver(s)=overlay2 version=20.10.21
time="2022-11-14T15:58:50.642094877Z" level=info msg="Daemon has completed initialization"
time="2022-11-14T15:58:50.667636348Z" level=info msg="API listen on /var/run/docker.sock"
time="2022-11-14T15:58:50.675740617Z" level=info msg="API listen on [::]:2376"
time="2022-11-14T16:00:51.843715103Z" level=info msg="Layer sha256:903e1ef39c915239bdee0efa6d2524604ab820fec073b9e34748c9703040dfa5 cleaned up"
time="2022-11-14T16:00:51.913459982Z" level=info msg="loading plugin \"io.containerd.event.v1.publisher\"..." runtime=io.containerd.runc.v2 type=io.containerd.event.v1
time="2022-11-14T16:00:51.913558196Z" level=info msg="loading plugin \"io.containerd.internal.v1.shutdown\"..." runtime=io.containerd.runc.v2 type=io.containerd.internal.v1
time="2022-11-14T16:00:51.913578733Z" level=info msg="loading plugin \"io.containerd.ttrpc.v1.task\"..." runtime=io.containerd.runc.v2 type=io.containerd.ttrpc.v1
time="2022-11-14T16:00:51.913895838Z" level=info msg="starting signal loop" namespace=moby path=/run/docker/containerd/daemon/io.containerd.runtime.v2.task/moby/f6414ce7601f27bce0762e94ff1cbf32879cb21d25da7dd11fa957ecc1986c37 pid=950 runtime=io.containerd.runc.v2
time="2022-11-14T16:03:28.439293599Z" level=info msg="ignoring event" container=f6414ce7601f27bce0762e94ff1cbf32879cb21d25da7dd11fa957ecc1986c37 module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
time="2022-11-14T16:03:28.439526418Z" level=info msg="shim disconnected" id=f6414ce7601f27bce0762e94ff1cbf32879cb21d25da7dd11fa957ecc1986c37
time="2022-11-14T16:03:28.439623591Z" level=warning msg="cleaning up after shim disconnected" id=f6414ce7601f27bce0762e94ff1cbf32879cb21d25da7dd11fa957ecc1986c37 namespace=moby
time="2022-11-14T16:03:28.439653563Z" level=info msg="cleaning up dead shim"
time="2022-11-14T16:03:28.488341728Z" level=warning msg="cleanup warnings time=\"2022-11-14T16:03:28Z\" level=info msg=\"starting signal loop\" namespace=moby pid=1060 runtime=io.containerd.runc.v2\n"

docker dind k8s pod

apiVersion: v1
kind: Pod
metadata:
  name: docker-dind-test
spec:
  volumes:
    - name: docker-certs-client
      emptyDir:
        medium: Memory
    - name: docker-certs-ca
      emptyDir:
        medium: Memory
    - name: sample-dockerfile
      configMap:
        name: dockerfile
  containers:
    - name: docker-shell
      image: docker:20-cli
      command:
        - sh
        - '-c'
        - "sleep 36000"
      env:
        - name: DOCKER_HOST
          value: tcp://docker:2376
        - name: LC_ALL
          value: en_US.UTF-8
        - name: DOCKER_TLS_CERTDIR
          value: /certs
        - name: DOCKER_TLS_VERIFY
          value: '0'
        - name: DOCKER_CERT_PATH
          value: /certs/client
        - name: DOCKER_DRIVER
          value: overlay2
        - name: IMAGE_NAME
          value: test-docker-image
      volumeMounts:
        - name: docker-certs-client
          mountPath: /certs/client
        - name: docker-certs-ca
          mountPath: /certs/ca
        - name: sample-dockerfile
          mountPath: /code
          readOnly: false
      securityContext:
        capabilities:
          drop:
            - NET_RAW
        privileged: true
    - name: docker-dind
      image: docker:20-dind
      args:
        - '--registry-mirror=https://mirror.gcr.io'
        - '--cri-containerd'
      env:
        - name: DOCKER_HOST
          value: tcp://docker:2376
        - name: LC_ALL
          value: en_US.UTF-8
        - name: DOCKER_TLS_CERTDIR
          value: /certs
        - name: DOCKER_TLS_VERIFY
          value: '1'
        - name: DOCKER_CERT_PATH
          value: /certs/client
        - name: DOCKER_DRIVER
          value: overlay2
      volumeMounts:
        - name: docker-certs-client
          mountPath: /certs/client
        - name: docker-certs-ca
          mountPath: /certs/ca
        - name: sample-dockerfile
          mountPath: /code
          readOnly: false
      securityContext:
        capabilities:
          drop:
            - NET_RAW
        privileged: true
    - name: debian-shell
      image: debian:bullseye
      command:
        - sh
        - '-c'
        - "sleep 36000"
    - name: ubuntu-shell
      image: ubuntu:22.04
      command:
        - sh
        - '-c'
        - "sleep 36000"
  dnsPolicy: ClusterFirst
  hostAliases:
    - ip: 127.0.0.1
      hostnames:
        - docker

curl -IL http://deb.debian.org/debian

HTTP/1.1 302 Found
Connection: keep-alive
Content-Length: 277
Server: Apache
X-Content-Type-Options: nosniff
X-Frame-Options: sameorigin
Referrer-Policy: no-referrer
X-Xss-Protection: 1
Permissions-Policy: interest-cohort=()
Location: http://ftp.debian.org/debian/
Content-Type: text/html; charset=iso-8859-1
Via: 1.1 varnish, 1.1 varnish
Accept-Ranges: bytes
Date: Mon, 14 Nov 2022 17:12:02 GMT
Age: 0
X-Served-By: cache-ams21054-AMS, cache-hkg17920-HKG
X-Cache: HIT, MISS
X-Cache-Hits: 4, 0
X-Timer: S1668445922.866768,VS0,VE175

HTTP/1.1 200 OK
Connection: keep-alive
Content-Length: 6415
Server: Apache
X-Content-Type-Options: nosniff
X-Frame-Options: sameorigin
Referrer-Policy: no-referrer
X-Xss-Protection: 1
Permissions-Policy: interest-cohort=()
X-Clacks-Overhead: GNU Terry Pratchett
Content-Type: text/html;charset=UTF-8
Via: 1.1 varnish, 1.1 varnish
Accept-Ranges: bytes
Date: Mon, 14 Nov 2022 17:12:02 GMT
Age: 0
X-Served-By: cache-ams12782-AMS, cache-hkg17925-HKG
X-Cache: HIT, MISS
X-Cache-Hits: 7, 0
X-Timer: S1668445922.339227,VS0,VE174
Vary: Accept-Encoding
kupson avatar
cn flag
This could be a MTU related problem. It can happen if e.g. your homelab network environment Path MTU discovery is not working correctly. Please try to lower the MTU on the network interface inside the DinD.
XPLOT1ON avatar
ky flag
@kupson I see, I tried to configured MTU value (1450) at docker dind and it works. the MTU on the host interface is set to 1500, but flannel and cni interface mtu is set to 1450. To my understanding they should match right?
kupson avatar
cn flag
The MTU on the interface inside DinD should be lower than the outside one. I would try 1280 just to see if it helps.
Score:0
na flag

Faced this specific issue Docker DIND cannot apt-get update but curl works, and found that docker:dind is based on Alpine Linux and not Ubuntu/Debian Linux.

So I had to replace apt-get install with apk add to make my install scripts work.

Score:0
ky flag

Update on current issue, mentioned by @kupson

the MTU settings does indeed solved this problem

- name: docker-dind
  image: docker:20-dind
  args:
    - '--registry-mirror=https://mirror.gcr.io'
    - '--cri-containerd'
    - '--mtu=1450'
Score:-1
cg flag

It seems that your Linux server is running out of memory.

This can be noticed by the statement "RestrictOOMScoreAdj" which refers to a file is present in /proc/$pid/oom_score_adj.

When your Linux machine runs out of memory, Out of Memory (OOM) killer is a process called by the Linux kernel to free some memory. It is often encountered on servers which have a number of memory intensive processes running.

You can fix this by adding more memory to your machine, either by manually terminating expandable services or configuring a SAWP partition on your system.

I sit in a Tesla and translated this thread with Ai:

mangohost

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.