systemd ExecStartPost is executed too fast

We have a unit file which:

uses ExecStartPre which generates a config file ExecStart uses the config file to start a service ExecStartPost which deletes the file created by ExecStartPre. (the file has hardcoded passwords and we do not want to keep it on disk. so it must only exist while the service is being started or restarted.

If the service is started successfully it should delete the config file, if the service fails to start it should also delete the file.

this is the systemd file:

[Unit]
Description=Prometheus Server
Documentation=https://prometheus.io/docs/introduction/overview/
After=network-online.target

[Service]
User=prommgr
Group=secapm
Restart=on-failure
ExecStartPre=/usr/bin/python2 /prom/config/anon_yml.py
ExecStart=/bin/sh -c "/prom/appl/prometheus/prometheus --config.file=/prom/config/prometheus.yml --storage.tsdb.path=/prom/data --web.listen-address=127.0.0.1:9090  --storage.tsdb.retention.time=1825d &>>/prom/logs/prometheus.log"
ExecStartPost=/usr/bin/python2 anon_yml.py --delete 

[Install]
WantedBy=multi-user.target

the config file in question is prometheus.yml the script anon_yml.py will create the file from a jinja template and populate it with the needed passwords. ExecStart will then start the service

in theory ExecStartPost would then remove the prometheus.yml file from disk after ExecStart has finished, however when the service is being restarted i have errors of config file is missing. it only means ExecStartPost did not wait until ExecStart finished.

How to make sure the config file is only deleted after the process has already been restarted?

The reason why ExecStartPost= executes too fast is because your service is of Type=simple (the default when ExecStart= is specified but Type= is omitted). In this case, ExecStartPost= will execute right after the process in ExecStart= is started.

If you want ExecStartPost= to execute only after the last ExecStart= process exited successfully, then you need to change your service to Type=oneshot:

[Service]
Type=oneshot
...

use like a flag to check if your Post action can be executed or not.

• append to the ExecStart cmd:

&& touch /tmp/flag

• replace ExecStartPost with:

timeout 10 bash -c -- 'while test ! -f /tmp/flag; do /usr/bin/python2 anon_yml.py --delete && rm -f /tmp/flag ; sleep 1; done'

it looks a bit dirty but should work..