Score:0

docker-compose `restart: always` randomly not working

in flag

I administer a few Debian servers with Docker CE installed and docker-compose orchestrating several services (roughly 20 containers per machine)..

Every single service is configured with restart: always in docker-compose. However, there are random services (usually 1-2 per machine) that shut down correct but do not restart during a host reboot. This behaviour is completely random. Sometimes all services start, sometimes one single service from a docker-compose.yml file does not restart.

Following is an example where Traefik did shut down correctly but did not come up:

  1. Service configured to restart automatically:
$ cat docker-compose.yml
version: '3'

services:
  reverse-proxy:
    image: traefik:1.7
    restart: always
    command: --web
    ports:
      - "80:80"
      - "443:443"
      - "8080:8080"
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
      - ./traefik.toml:/traefik.toml
    networks:
      - web

  1. Exit code is 128
$ docker-compose ps
         Name                Command        State     Ports
---------------------------------------------
traefik_reverse-proxy_1   /traefik --web   Exit 128
  1. Logs show that the service has shut down correctly but there are no signs of activity since then:
$ docker-compose logs --tail 6 -t
Attaching to traefik_reverse-proxy_1
reverse-proxy_1  | 2022-01-21T14:05:28.042399112Z time="2022-01-21T14:05:28Z" level=info msg="Stopping server gracefully"
reverse-proxy_1  | 2022-01-21T14:05:28.042450915Z time="2022-01-21T14:05:28Z" level=debug msg="Waiting 10s seconds before killing connections on entrypoint http..."
reverse-proxy_1  | 2022-01-21T14:05:28.042463326Z time="2022-01-21T14:05:28Z" level=debug msg="Waiting 10s seconds before killing connections on entrypoint api..."
reverse-proxy_1  | 2022-01-21T14:05:28.053256515Z time="2022-01-21T14:05:28Z" level=debug msg="Entrypoint api closed"
reverse-proxy_1  | 2022-01-21T14:05:28.053283046Z time="2022-01-21T14:05:28Z" level=debug msg="Entrypoint http closed"
reverse-proxy_1  | 2022-01-21T14:05:28.059721498Z time="2022-01-21T14:05:28Z" level=info msg="Shutting down"
  1. Server uptime corresponds with the shutdown message:
$ uptime
 11:21:31 up 29 days, 20:15,  1 user,  load average: 0.46, 0.43, 0.44
  1. My docker-version is the following:
$ docker --version
Docker version 19.03.12, build 48a66213fe

Let's not focus on Traefik alone because it is completely random which container does not start and when.

mx flag
I'm having the same issue....debian 11, exit code 128 on containers. Did you ever get to the bottom of this? Worked fine on CentOS
adamsfamily avatar
in flag
Unfortunately not :( I reinstalled the particular VPS a little later (probably a different version of all of the components: OS, Docker, docker-compose) and currently it works. Albeit, it does bother me that I don't know what caused the problem.
mangohost

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.