Why does my systemd timer only trigger once when the unit is a target?

I have a couple of services (static site generators) that I want to trigger on a regular basis from the same systemd timer. I found this question/answer, which covers exactly what I want to do, and describes a setup whereby a .target file that Wants= multiple services is triggered by a corresponding timer. This sounds great, but I'm finding that when I actually set this up, it only ever triggers once, then disables itself!

I've prepared a minimal working example (this doesn't trigger multiple services, but demonstrates the same problem):

test-timer.timer:

[Unit]
Description=A test timer

[Timer]
OnCalendar=*-*-* *:*:30
Unit=test-timer.target

[Install]
WantedBy=timers.target

test-timer.target:

[Unit]
Description=Target unit
Wants=test-timer.service
After=test-timer.service

[Install]
WantedBy=timers.target

test-timer.service:

[Unit]
Description=Run test

[Service]
ExecStart=/usr/bin/bash -c "date --rfc-3339='seconds' >> /tmp/test-timer-output"

[Install]
Also=test-timer.target

Enable the timer:

$ sudo cp test-timer.* /etc/systemd/system/
$ sudo systemctl enable --now test-timer.timer
Created symlink /etc/systemd/system/timers.target.wants/test-timer.timer → /etc/systemd/system/test-timer.timer.

Then, when I look at the output of systemctl list-timers --all, prior to the first run I get (ignoring other timers):

NEXT                        LEFT       LAST     PASSED       UNIT                ACTIVATES
Fri 2021-10-08 10:38:30 EDT 21s left   n/a      n/a          test-timer.timer    test-timer.target

After the first run, NEXT and LEFT have been replaced with n/a:

NEXT     LEFT    LAST                        PASSED        UNIT                  ACTIVATES
n/a      n/a     Fri 2021-10-08 10:38:32 EDT 1min 5s ago   test-timer.timer      test-timer.target

I've also tried adding Persistent=true to the test-timer.target and explicitly enabling test-timer.target, but neither of these work. Any time I do systemctl restart test-timer.timer, it restarts, but only triggers one run, then never has another go.

If I remove the layer of indirection by changing the Unit= line of test-timer.timer to Unit=test-timer.service, the service happily triggers itself every minute as expected.

Am I missing some configuration or installation step?

Using BindsTo has some disadvantages:

• A minor disadvantage is that you're defying the purpose of a target unit: Target units serve as well-known synchronization points and therefore must not contain any logic on their own (i.e. target units must not statically depend on any other units besides other target units). By statically adding dependencies to a target unit (BindsTo is a very strong dependency) you move part of a service unit's logic to said target unit, making it impossible to understand what a service unit does by just looking at the service unit. That's why service units must "register" themselves as dependency of target units using the [Install] section. However, due to this inversion of logic, you can no longer use BindsTo - because its inversion, BoundBy, can't be specified directly.

• Anyway, a more major disadvantage is that BindsTo enforces concurrency. Even though you mention only a single service unit, I assume that you're actually trying to start multiple service units by a single timer - otherwise the target unit would serve no purpose. If you'd attempt to bind multiple service unit to your target unit, all service units are equally required to run for the target unit to run. This means that all service units must start at the same time (i.e. enforced concurrency), and also means that if any of the specified service units stops, your target unit will stop, too. This might cause unexpected behaviour when the target is started again (either manually, or due to a timer) while some of the service units are still running.

To avoid these disadvantages I recommend the following:

1. Create the timer unit test-timer.timer that starts test-timer.target. Don't forget to enable the timer unit: systemctl enable test-timer.timer

   test-timer.timer:

   [Unit]
   Description=A test timer
   
   [Timer]
   OnCalendar=*-*-* *:*:30
   Unit=test-timer.target
   
   [Install]
   WantedBy=timers.target
   

2. Create the mentioned test-timer.target. You won't need to specify the required service units here, the service units will rather "register" themselves as dependencies using WantedBy. Since we don't want the target unit to run indefinitely, we add StopWhenUnneeded=true. Systemd will then stop the target unit as soon as it served its purpose, i.e. when Systemd queued the service units to start.

   test-timer.target:

   [Unit]
   Description=Target unit
   StopWhenUnneeded=true
   

3. Create your service units just the way you'd anyway. You can add Wants, Requires, Before, After, and all other (e.g. Conflicts) directives just as usual. Also add WantedBy=test-timer.target, allowing the service unit to "register" itself as requirement of the target unit when enabled. After enabling the service unit Systemd will start the service unit as soon as the target unit is started, i.e. when the timer triggers.

   However, due to StopWhenUnneeded=true, Systemd would stop the target unit as soon as the service unit was queued to start (i.e. immediately). This wouldn't hinder anyone (either manually, or due to a timer) to start the target unit again. Since the service units might still run, you might experience unexpected behaviour, just as with BindsTo. Thus we also add Upholds=test-timer.target and After=test-timer.target. These directives cause Systemd to ensure that the target unit runs while the service unit runs. It won't stop the target unit though; that's StopWhenUnneeded=true's job. Together they effectively cause Systemd to keep the target unit running until all service units with Upholds=test-timer.target stop (BindsTo in contrast stops the target unit as soon as any service unit stops).

   In general I recommend using Type=oneshot services instead of the default Type=simple in the context of service units triggered by timers. That's because Systemd will consider a Type=simple service unit started immediately and dead when the command finished. With Type=oneshot Systemd will rather consider the service unit "starting" while the command runs and switch directly to dead after it has finished. This is a major difference for concurrency and After/Before directives: With Type=simple the command of a service unit won't wait for the command of another service unit that was declared to run After to finish. With Type=oneshot Systemd does wait.

   • Okay, now let's assume that we want to start two service units, test-timer-1.service and test-timer-2.service. They can run in parallel, but aren't required to. Systemd shall therefore trigger the timer unit, then start the target unit, and then both service units in parallel. The target unit shouldn't stop before both service units stopped, and so should the timer unit. To achieve this create the following two service units. Don't forget to enable the service units (systemctl enable test-timer-1.service test-timer-2.service).

     test-timer-1.service:

     [Unit]
     Description=Run test 1
     After=test-timer.target
     Upholds=test-timer.target
     
     [Service]
     Type=oneshot
     ExecStart=/usr/bin/bash -c "sleep 3 ; date --rfc-3339='seconds' >> /tmp/test-timer-output-1"
     
     [Install]
     WantedBy=test-timer.target
     

     test-timer-2.service:

     [Unit]
     Description=Run test 2
     After=test-timer.target
     Upholds=test-timer.target
     
     [Service]
     Type=oneshot
     ExecStart=/usr/bin/bash -c "sleep 3 ; date --rfc-3339='seconds' >> /tmp/test-timer-output-2"
     
     [Install]
     WantedBy=test-timer.target
     

   • If you rather want test-timer-2.service to start not before test-timer-1.service has finished, you simply add After=test-timer-1.service to test-timer-2.service. Systemd shall then trigger the timer unit, then start the target unit, and then just the first service unit. When the first service unit has finished, Systemd shall start the second service unit. Systemd shall stop the target unit as soon as all service units have finished (here: just the second service unit), also causing the timer unit to stop. This would be impossible with BindsTo. Again, don't forget to enable the service units.

     test-timer-1.service:

     [Unit]
     Description=Run test 1
     After=test-timer.target
     Upholds=test-timer.target
     
     [Service]
     Type=oneshot
     ExecStart=/usr/bin/bash -c "sleep 3 ; date --rfc-3339='seconds' >> /tmp/test-timer-output-1"
     
     [Install]
     WantedBy=test-timer.target
     

     test-timer-2.service:

     [Unit]
     Description=Run test 2
     After=test-timer.target test-timer-1.service
     Upholds=test-timer.target
     
     [Service]
     Type=oneshot
     ExecStart=/usr/bin/bash -c "sleep 3 ; date --rfc-3339='seconds' >> /tmp/test-timer-output-2"
     
     [Install]
     WantedBy=test-timer.target
     

I'm using this exact setup to orchestrate backups: I first run some service units to prepare the system for backup (e.g. create snapshots), then run the backups (some can run in parallel, some must run after another) and finally clean up.