systemd: Configure correct dependencies with LVM devices

I have a device setup that I don't know how to model correctly with systemd:

The setup

I have the following setup:

                            ---------
                            | mount |  /mnt/*
                            ---------
                              |   A
           4. umount /mnt/*   |   |  3. mount /dev/data/* /mnt/*
                              v   |
                            ---------
                            |  LVM  |  /dev/mapper/data-*, /dev/data/*
                            ---------
                              |   A
           5. vgchange -a n   |   |  2. automatic or `vgchange -a y`
                              v   |
                            ---------
                            | LUKS  |  /dev/mapper/decr_device
                            ---------
                              |   A
 6. cryptsetup luksClose      |   |  1. cryptsetup luksOpen /dev/sdb decr_device ...
    /dev/mapper/decr_device   v   |
                            ---------
                            |  HDD  |  /dev/sdb
                            ---------

So there are several stages of (artificial) devices. Step 1, 2 and 3 are necessary to mount the partitions. Step 4, 5 and 6 are necessary to encrypt/luksClose the HDD again.

The problem

I want to perform all steps within systemd on a per step base (so every steps becomes an own unit).

Systemd does step 1 and 6 more less automatically when I provide a correct /etc/crypttab. It also performs the step 3 and 4 correctly when I provide a correct /etc/fstab. I have, however, not found a possibility to get step 2 and 5 into systemd.

I have tried:

systemctl add-requires dev-data-stuff.device systemd-cryptsetup@decr_device.service

which errors with:

Failed to add dependency: Unit file dev-data-stuff.device does not exist.

I considered to (over)write the dev-data-stuff.device with some manual script that calls vgchange -a ... at start or exit but have not found any documentation to do that with a "device"-file. Do you know a way?

Fun fact: The system mounts this already automatically. I guess, because systemd decrypts decr_device, LVM then automatically creates /dev/data/*, which trigger the mount script with the help of udev. However, I want to have the dependency chain modeled in systemd on its whole to be able to execute them by hand, too. Currently, there is now way (except manually) to disassemble the whole stuff, i.e. executing steps 4-6.

Method 1: "Bottom Up" mounting upon insertion

I have covered the simpler case without the LVM layer because this was my own use case. The post in connection with consultation of the different systemd unit related man files should be enough to understand the underlying machinery.

As you have correctly noted, the LVM layer needs additional action, for this purpose I created my_vgchange@<lvmvolgrp>.service in /etc/systemd/system:

[Unit]
DefaultDependencies=no
IgnoreOnIsolate=true

BindsTo=systemd-cryptsetup@<luksvol>.service
After=systemd-cryptsetup@<luksvol>.service

[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=/sbin/vgchange -ay <lvmvolgrp>
ExecStop=/sbin/vgchange -an <lvmvolgrp>

[Install]
WantedBy=systemd-cryptsetup@<luksvol>.service

DefaultDependencies=no and IgnoreOnIsolate=true are strictly speaking not necessary, however, these settings mirror those in the generated [email protected] units. In both cases, this makes sense because of the, for .service units, atypical invocation/timing/use-case of these units.

Next, we make sure that the LVM layer stops whenever the underlying LUKS layer stops via the BindsTo= attribute. We also need to ensure the correct temporal order between these two units, After= accomplishes this. (note that making an After= statement in one unit is equivalent to making a Before= statement in another, either one will do).

Type=oneshot + RemainAfterExit=yes are both necessary in order to have the service behave as expected, fire off the ExecStart= statement(s) when starting and ExecStop= when stopping (oneshot can have multiple ExecStart= and ExecStop= statements, unlike other service types). RemainAfterExit=yes means that the unit is considered "active" after startup, otherwise it would immediately be considered "dead" after activation, which would preclude us from correctly triggering the unit in the greater startup/shutdown chain. These settings mirror the ones of [email protected] (at least in my distro) and for exactly the same reasons, to keep track of ExecStart/ExecStop invocations over the lifetime of the mount unit.

In theory, ExecStart= may be omitted because udev comes with default rules in /lib/udev/rules.d/ to automatically start a lvm2-pvscan@<device_no>.service whenever a physical volume appears in the system, which ultimately leads to the creation of a .device unit representing the logical volume in systemd. However, calling vgchange -ay on an already activated device seems to do no harm and allows us to ignore the udev layer/rules (in regards to LVM) and treat [email protected] as a fully start and stoppable unit.

Finally, WantedBy= in the install section ensures that the auto generated [email protected] unit will automatically try to start [email protected], ensuring a "bottom up" chain. As usual, settings in the install section are only realized by calling systemctl enable (or reenable, or disable -> enable).

[All enabling/disabling unit files really does is (re)creating <unit>.wants <unit>.requires directories and placing symlinks in them, it is otherwise completely orthogonal from start/stopping.]

At this point, a custom udev rule should automatically start systemd-cryptsetup@<luksvol>.service whenever the device is attached, and the unit then "wants" to start my_vgchange@<lvmvolgrp>.service. Furthermore, systemd will try to stop the LVM service before it attempts to stop the LUKS service because of the defined temporal ordering.

At this point, all we have to do is connect the mount point in /etc/fstab with my_vgchange@<lvmvolgrp>.service. An entry would look like this:

/dev/lvmvg/lv    /mnt/lv    ext4    defaults,nofail,x-systemd.wanted-by=my_vgchange@<lvmvolgrp>.service,x-systemd.requires=my_vgchange@<lvmvolgrp>.service,x-systemd.after=my_vgchange@<lvmvolgrp>.service    0 2

The mount point still implicitly BindsTo= the .device unit representing the LVM logical volume, thereby ensuring that it is stopped whenever the underlying device stops/vanishes, for example due to vgchange -an.

Now a newly inserted LUKS-LVM drive should automatically mount by itself and simply stopping systemd-cryptsetup@<luksvol>.service is enough to unmount/stop everything, while starting the mnt-lv.mount unit also pulls in the underlying units.

Downsides: classic mount/umount invocations may not work. If the drive is not removed from the system, systemd timers seem to somehow trigger udev and cause everything to be remounted.

Method 2: Using a systemd .automount unit with LUKS+LVM

This approach depends on the automount functionality provided by systemd. Volumes get only mounted when a user/process tries to access a file under the mount point. Provided an "idle timeout" (deactivated by default) value is set (x-systemd.idle-timeout= in /etc/fstab or TimeoutIdleSec= in a .automount unit, the default unit is seconds) unmounting also happens automatically.

The /etc/fstab entry would look like this:

/dev/lvmvg/lv   /mnt/lv    ext4    nofail,x-systemd.automount,x-systemd.idle-timeout=600    0 2

systemd-fstab-generator is now generating two units from this line, <mnt-lv>.automount and <mnt-lv>.mount, both can be found inside /run/systemd/generator/. Unlike normally, the .mount unit is not "wanted/required" by the local-fs.target, instead it stands in a TriggeredBy relationship with its corresponding .automount unit (see systemd.unit).

Importantly, the mount options with relevance to systemd units are now only affecting the .automount unit. For example, fail/nofail will decide if <mnt-lv>.automount is WantedBy= or RequiredBy= local-fs.target. The auto/noauto option has no effect, while x-systemd.requires= and similar x-systemd.* settings also only affect the automount unit.

Summing up, the <mnt-lv>.automount is started by local-fs.target on startup and is supposed to be constantly "active" throughout operation of the system. It watches over the mount point and starts/stops the <mnt-lv>.mount unit when necessary. Therefore all we need to do is make sure that:

1. the underlying LUKS and LVM units are required by the mount unit
2. LUKS -> LVM -> mount units are correctly ordered
3. the underlying units shut down when the mount unit gets stopped, by binding to it

Since we can no longer edit .mount unit attributes in the fstab via x-systemd.*, we can use the "override" mechanism to add configuration to the generated unit via systemctl edit <mnt-lv>.mount:

[Unit]
Requires=systemd-cryptsetup@<luksvol>.service my_vgchange@<lvmvolgrp>.service
After=systemd-cryptsetup@<luksvol>.service my_vgchange@<lvmvolgrp>.service

The LVM my_vgchange@<lvmvolgrp>.service unit is naturally quite similar to the one of method 1:

[Unit]
DefaultDependencies=no
IngoreOnIsolate=true

BindsTo=<mnt-lv>.mount
Before=<mnt-lv>.mount
After=systemd-cryptsetup@<luksvol>.service

[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=/sbin/vgchange -ay <lvmvolgrp>
ExecStop=/sbin/vgchange -an <lvmvolgrp>

The important difference is that this time, its lifetime is bound to the "upper level" .mount unit. Before= is not really necessary because it's equivalent to the After= directive in the mount unit. After= places the LVM service after LUKS unit.

Finally, we append to the generated LUKS service unit by calling systemctl edit systemd-cryptsetup@<luksvol>.service

[Unit]
BindsTo=<mnt-lv>.mount
Before=<mnt-lv>.mount

Like the LVM service, both are now stopped whenever the mount unit is stopped, while both are required by the mount unit whenever it is activated. All three units are activated and deactivated in the correct order.