Score:1

Server

systemctl list-units for type target not behaving as expected

Kevin Powell

2/15/24, 12:48 AM

Summary: 'systemctl list-units --type=target --all' does not always list targets that I've added

Detail:

I have a project that consists of several related daemons; let's call this project 'Foo.' The daemons are managed by systemd service units, and the services are grouped together for easier management in a target. I have 'production' and 'testing' deployments of projcet Foo. So, I've created foo_testing.target and foo_production.target under /etc/systemd/system. In fact, I have this arrangement on a variety of physical hosts and VMs -- some have both production and testing, some have only one or the other. I therefore need to be able to interrogate a given host as to what is installed and what is running.

Having reviewed the docs for systemd and systemctl I hit upon

systemctl list-units --type=target --all

This works fine for targets that are running. If production and/or testing is running it will reliably show up in the output of list-units.

The problem is stopped targets. If I 'systemctl stop foo_testing.target' and immediately ask for the list of units, then foo_testing.target will be in the list (marked inactive). But, if I leave the testing deployment stopped for a day or two, and then come back and do 'systemctl list-units --type=target --all' then foo_testing won't be in the list at all.

So, why does systemd 'forget' about my targets if they haven't been active for a while?

Here's the system info for the most problematic host:

$ lsb_release -a
Distributor ID: Ubuntu
Description:    Ubuntu 21.04
Release:    21.04
Codename:   hirsute
$ uname -a
Linux <redacted> 5.11.0-49-generic #55-Ubuntu SMP Wed Jan 12 17:36:34 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
$systemctl --version
systemd 247 (247.3-3ubuntu3.7)
+PAM +AUDIT +SELINUX +IMA +APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +ZSTD +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=hybrid

[Edit] As requested, here's the output of 'systemctl cat'

$ systemctl cat "foo*.target"
# /etc/systemd/system/foo_testing.target
[Unit]
Description=Foo Testing Environment
AllowIsolate=no
Wants=serv1_testing.service serv2_testing.service serv3_testing.service

# /etc/systemd/system/foo_production.target
[Unit]
Description=Foo Production Environment
AllowIsolate=no
Wants=serv1_prod.service serv2_prod.service serv3_prod.service

Note, though, that this output does not change. It is the behavior of 'systemctl list-units' that changes.

[Edit 2] Q: Any non-target systemd units with matching base name installed on the system?

A: such units do exist -- owing to my naming scheme. Is this potentially part of the explanation?

171

1 + 3

systemd

systemctl

anx

2/15/24, 1:05 AM

Please upgrade to a **[supported](https://wiki.ubuntu.com/Releases) OS release**, Ubuntu 21.04 is dangerous after [lacking security upgrades since 2022-01-20](https://lists.ubuntu.com/archives/ubuntu-announce/2022-January/000276.html). Any non-target systemd units with matching base name installed on the system? (If in doubt, just show the output of `find /etc/systemd /run/systemd -iname "*foo*"`)

Kevin Powell

2/15/24, 1:09 AM

well -- 17 whole minutes before somebody unhelpfully suggested that I reinstall the OS. Longer than expected, actually. Does it never occur that I might not have approval from (mis)management to go reinstalling the entire OS at the drop of a hat?

anx

2/15/24, 1:43 AM

While not relevant in this case because a still-supported release is also affected, [this site](https://serverfault.com/help/on-topic) generally focuses on system management in a business environment. The sort of environment where not today, but two years ago a decision was made to move all software support deadline further out than the expected time to get upgrades budgeted, planned approved & done, possibly by switching to LTS releases.

Score:0

Server

anx

2/15/24, 1:28 AM

list-units cannot be expected to report consistently on inactive units:

list-units [PATTERN…]

List units that systemd currently has in memory. This includes units that are either referenced directly or through a dependency, units that are pinned by applications programmatically, or units that were active in the past and have failed.

from man systemctl, emphasis mine.

The command seems more geared towards letting a human see the unit state at a glance, and must be expected to forget about units no longer needed, e.g. after some unspecified time, or on daemon reload or reexec.

If in a script you need the running status of a unit, use systemctl is-active (for state is-enabled, for file existence: list-unit-files, ..) which produce consistent results, no matter the in-memory status.

+ 4

Kevin Powell

2/15/24, 1:46 AM

OK -- that does explain the behavior. I'll accept your answer on that basis. However, I still don't have something that will let me use systemctl to answer the question "which targets are on this host?" For now, I can use 'systemctl is-active' and ask specifically about production and testing. But, in the future, maybe I want 3 environments (experimental, testing, production) or 4, or more... Is it really the case that the best way is 'ls /etc/systemd/system | grep target' ?

anx

2/15/24, 1:52 AM

@KevinPowell see edit, there are additional commands depending on what you need (running status, enablement state, unit file existence, ..)

Kevin Powell

2/15/24, 2:02 AM

Thanks for the additional info. Turns out what I wanted was probably "systemctl list-unit-files --type=target" all along. The way list-units is currently documented (particularly when combined with the docs for --all : which says "When listing units with list-units, also show inactive units...") seems to leave a lot of room for misinterpretation.

user1686

2/18/24, 7:27 PM

From what I remember, it used to be that systemd kept unit files in memory for quite a bit longer after they went inactive (though still not loading all of them proactively), so the documentation might be a leftover from that time.

Elon Musk

I sit in a Tesla and translated this thread with Ai:

EN: systemctl list-units for type target not behaving as expected

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.