Daemonized influxdb container won't start on Ubuntu 22

I'm trying to upgrade a server to Ubuntu 22. Among other things it's running a docker image of InfluxDB 2.7. The whole setup is puppetised, so I'm not "upgrading" the OS as much as I'm setting up the whole thing on a new instance that is running Ubuntu 22, while the old one is running Ubuntu 20.

The influxdb container is supposed to run as a service, and I'm running into the problem that the service just won't start. Or rather, seems to exit immediately again.

So here's what I'm doing: The container is started by a unit file, but actually that unit file is only calling a start script that launches the container. The unit file looks like this:

[Unit]
Description=Daemon for bmetry_influx_dev
After=docker.service
Wants=
Requires=docker.service

[Service]
Restart=on-failure
StartLimitInterval=20
StartLimitBurst=5
TimeoutStartSec=5
RestartSec=5
Environment="HOME=/root"
SyslogIdentifier=docker-bmetry_influx_dev
ExecStart=/usr/local/bin/docker-run-bmetry_influx_dev-start.sh
ExecStop=/usr/local/bin/docker-run-bmetry_influx_dev-stop.sh

[Install]
WantedBy=multi-user.target

The script that launches the container, looks like this:

#!/usr/bin/env bash

/usr/bin/docker rm  bmetry_influx_dev >/dev/null 2>&1

/usr/bin/docker create \
--net bridge \
-m 0b \
-p 8086:8086 \
 \
-v /opt/influx/data:/var/lib/influxdb2:rw \
 \
--restart=always \
--name bmetry_influx_dev \
influxdb:2.7.0 \
 
/usr/bin/docker start  bmetry_influx_dev

Nothing spectacular, no rocket science here. In fact, if I just call that startup script, the container starts and is running fine. But the service just won't run. Whenever I ask for its status, this is what it shows:

docker-bmetry_influx_dev.service - Daemon for bmetry_influx_dev
     Loaded: loaded (/etc/systemd/system/docker-bmetry_influx_dev.service; enabled; vendor preset: enabled)
     Active: inactive (dead) since Tue 2023-06-20 08:09:28 UTC; 16min ago
    Process: 3880 ExecStart=/usr/local/bin/docker-run-bmetry_influx_dev-start.sh (code=exited, status=0/SUCCESS)
    Process: 3989 ExecStop=/usr/local/bin/docker-run-bmetry_influx_dev-stop.sh (code=exited, status=0/SUCCESS)
   Main PID: 3880 (code=exited, status=0/SUCCESS)
        CPU: 134ms

Jun 20 08:09:28 bmetry-influxdb.sitetest.ch systemd[1]: Started Daemon for bmetry_influx_dev.
Jun 20 08:09:28 bmetry-influxdb.sitetest.ch docker-bmetry_influx_dev[3886]: 1aef7d1501d9715736d2a2ffe0382a92a7eb26bce790d8c60de5c48daec915fb
Jun 20 08:09:28 bmetry-influxdb.sitetest.ch docker-bmetry_influx_dev[3891]: bmetry_influx_dev
Jun 20 08:09:28 bmetry-influxdb.sitetest.ch docker-bmetry_influx_dev[3991]: bmetry_influx_dev
Jun 20 08:09:28 bmetry-influxdb.sitetest.ch docker-bmetry_influx_dev[4074]: bmetry_influx_dev
Jun 20 08:09:28 bmetry-influxdb.sitetest.ch systemd[1]: docker-bmetry_influx_dev.service: Deactivated successfully.

There's an awful lot of talk about success in here for something that is not running, and no useful error message.

Hunting through syslog, this weird message is what comes up:

bmetry-influxdb dockerd[688]: time="2023-06-20T08:09:28.784772033Z" level=info msg="Container failed to exit within 0s of signal 15 - using the force" container=1aef7d1501d9715736d2a2ffe0382a92a7eb26bce790d8c60de5c48daec915fb

I mean, using the force is nice and all, but I'd rather it would use it to jedi-mind-trick a more understandable error message into my head.

Again, I want to emphasize that the container runs without issue if I call the startup script manually, so the issue seems unlikely to be docker related. The whole setup also works without issue on Ubuntu 20, so most probably something changed with the unit files, and I assume I have to change something in there, but I can't figure out what. Any help is highly appreciated!

You've configured your container to run in detached mode -- effectively, "in the background". The problem is that by default systemd assumes you are using a Type=simple service, which means that it assumes that service has stopped if the main process returns -- which is exactly what happens with your docker-run-bmetry_influx_dev-start.sh; the script exits, so systemd thinks the process has stopped and runs the ExecStop code.

There are a few ways to fix this, but arguably the correct one is to stop using systemd. There are complications involved in trying to manage Docker containers with systemd that boil down to the fact that you are monitoring the Docker client, not the container that you start.

If you've set a restart policy of always (which you have), then Docker itself will take care of starting the container when your system boots. You don't need systemd for that.

The general solution for running a program that backgrounds automatically is to configure the systemd service with Type=forking, but that won't work here: the container isn't forked from your script; your script is simply talking to the Docker daemon via its API (and the Docker daemon is responsible for actually spawning the container), so the container isn't visible to systemd.

You can try running the container in non-detached mode (probably the easiest way to do this is to replace docker create with docker run and get rid of the docker start call) and set Type=exec, but in this case you're not actually monitoring the container -- you're simply monitoring the Docker client. It's possible for the client to exit without impacting the container, which may result in unexpected behavior.