systemd terminates etcd service started with podman - reception only permitted for main PID

Jonas

12/21/22, 3:54 PM

I try to start etcd as a systemd service run in a container with podman.

After start I get this error log from systemd:

systemd[1]: etcd.service: Got notification message from PID 4696, but reception only permitted for main PID 4868

But etcd seem to be able to start an try to notify the container daemon:

21T15:31:08.817Z","caller":"etcdserver/server.go:2500","msg":"cluster version>
Aug 21 15:31:08 ip-10-0-0-71 podman[4696]: {"level":"info","ts":"2021-08-21T15:31:08.817Z","caller":"etcdmain/main.go:47","msg":"notifying init daemon>
Aug 21 15:31:08 ip-10-0-0-71 podman[4696]: {"level":"info","ts":"2021-08-21T15:31:08.818Z","caller":"etcdmain/main.go:53","msg":"successfully notified>

But systemd seem to not be aware of that and terminates the etcd service:

Aug 21 15:32:34 ip-10-0-0-71 systemd[1]: etcd.service: start operation timed out. Terminating.
Aug 21 15:32:35 ip-10-0-0-71 podman[4696]: {"level":"info","ts":"2021-08-21T15:32:35.000Z","caller":"osutil/interrupt_unix.go:64","msg":"received sign>
Aug 21 15:32:35 ip-10-0-0-71 podman[4696]: {"level":"info","ts":"2021-08-21T15:32:35.000Z","caller":"embed/etcd.go:367","msg":"closing etcd server","n>

This is the systemd service status:

$ sudo systemctl status etcd.service
● etcd.service - etcd
     Loaded: loaded (/etc/systemd/system/etcd.service; enabled; vendor preset: enabled)
     Active: failed (Result: timeout) since Sat 2021-08-21 15:32:35 UTC; 8min ago
    Process: 4868 ExecStart=/usr/bin/podman run -p 2380:2380 -p 2379:2379 --volume=/var/lib/etcd:/etcd-data:z --name etcd 842445240665.dkr.ecr.eu-nort>
   Main PID: 4868 (code=exited, status=0/SUCCESS)
        CPU: 3.729s

This is my systemd unit service file for etcd started with podman:

cat <<EOF | sudo tee /etc/systemd/system/etcd.service
[Unit]
Description=etcd
After=podman_ecr_login.service mk_etcd_data_dir.service

[Service]
Type=notify
ExecStart=/usr/bin/podman run -p 2380:2380 -p 2379:2379 --volume=/var/lib/etcd:/etcd-data:z \
 --name etcd <my-aws-account>.dkr.ecr.eu-north-1.amazonaws.com/etcd:v3.5.0 \
 /usr/local/bin/etcd --data-dir=/etcd-data \
 --name etcd0 \
 --advertise-client-urls http://127.0.0.1:2379 \
 --listen-client-urls http://0.0.0.0:2379 \
 --initial-advertise-peer-urls http://127.0.0.1:2380 \
 --listen-peer-urls http://0.0.0.0:2380 \
 --initial-cluster etcd0=http://127.0.0.1:2380

[Install]
WantedBy=multi-user.target
EOF

sudo systemctl enable etcd
sudo systemctl start etcd

I suspect that this might be related to Type=notify or perhaps the way I use podman or etcd. I start etcd in a similar way as described in etcd documentation: Run etcd clusters inside containers - Running a single node etcd. I run this on Debian 11 with Podman 3.0.1.

Any suggestions on how I can start etcd with podman as a systemd service?

150

1 + 0

linux

debian

systemd

etcd

podman

Score:0

Server

Jonas

12/21/22, 4:04 PM

According to this issue comment, these services should be run as Type=simple since they don't signal back to systemd. This PR to podman set it to Type=exec which also seem to work good.

After change to Type=exec in my service unit file, it now works:

$ sudo systemctl status etcd.service
● etcd.service - etcd
     Loaded: loaded (/etc/systemd/system/etcd.service; enabled; vendor preset: enabled)
     Active: active (running) since Sat 2021-08-21 15:59:23 UTC; 1min 28s ago
   Main PID: 4662 (podman)
      Tasks: 11 (limit: 442)
     Memory: 137.9M
        CPU: 3.576s
     CGroup: /system.slice/etcd.service
             ├─4662 /usr/bin/podman run -p 2380:2380 -p 2379:2379 --volume=/var/lib/etcd:/etcd-data:z --name etcd <my-aws-account>.dkr.ecr.eu-north-1.amaz>
             └─4846 /usr/bin/conmon --api-version 1 -c 616b317dc255ca86b308857dc6a180510fc166975a8a28437f3434111f03e7ad -u 616b317dc255ca86b308857dc6a>

0 + 2

Erik Sjölund

12/22/22, 6:26 AM

Maybe things have changed with Podman 3.3.0? Quote from the [release notes of Podman v3.3.0](https://github.com/containers/podman/releases/tag/v3.3.0) (released August 2021) _Systemd unit files generated by podman generate systemd now use Type=notify by default, instead of using PID files._ . Support for `Type=notify` in _etcd_ seems to be present: [main.go](https://github.com/etcd-io/etcd/blob/6a32bbad756b656da23af007ac4a0256b3dab7b5/server/etcdmain/main.go#L48) [etcd.service](https://github.com/etcd-io/etcd/blob/main/contrib/systemd/etcd.service)

Jonas

12/22/22, 6:59 AM

@ErikSjölund Interesting! Thank you!

Elon Musk

I sit in a Tesla and translated this thread with Ai:

EN: systemd terminates etcd service started with podman - reception only permitted for main PID

TH: systemd ยุติบริการ etcd ที่เริ่มต้นด้วย podman - การรับอนุญาตสำหรับ PID หลักเท่านั้น

RO: systemd termină serviciul etcd început cu podman - recepția este permisă numai pentru PID principal

RU: systemd завершает службу etcd, запущенную с помощью podman — прием разрешен только для основного PID

VI: systemd chấm dứt dịch vụ etcd bắt đầu với podman - chỉ được phép tiếp nhận đối với PID chính

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.