Score:0

Server

systemD user manager service keeps failing

Pauchu

5/2/24, 9:03 AM

I am running a bunch of Ubuntu 20.04 vServers for my company. They have a regularly appearing problem, being that the user manager service for the SSH User ([email protected]), sometimes fails when I log in using SSH.

The only common denominator these servers have is that they are all running Apache servers, an Icinga monitoring service and a daily backup to s3, and that they run with the same hosting company. What gives me the most headache however is, that this doesn't happen consistently, you might log in 10 times without the service failing, also it is not happening on all of the servers.

In the syslog I found the following:

Failed to create /user.slice/user-UID.slice/[email protected]/init.scope control group: Permission denied

I found a post suggesting to give rwx to the group on the directory the file is created in.

sudo chmod o+rwx -R /sys/fs/cgroup/systemd/

However this is only a temporary solution and does not survive reboots.

My best guess as to why it happens so inconsistently would be, that some of the servers are compromised, however I haven't found any indicators to this conclusion.

Maybe some of you have ever experienced something similar?

266

1 + 10

linux

ubuntu

systemd

user1686

5/7/24, 6:41 AM

What kind of "vServer" are these – KVM? OpenVZ? Hyper-V? Docker? LXC? Such cgroup-related issues hint at the server either being run under some container framework than being a full VM, _or_ running some additional cgroup management software that fights with systemd.

Pauchu

5/8/24, 7:57 AM

@user1686 according to the provider, they use a solution called Virtuozzo, of which I had never heard before I just looked it up. Apart from the company website and the fact that the Linux version is supposedly based on OpenVZ, I really couldnt find any information on it.

user1686

5/8/24, 8:08 AM

It's an old container framework, from around 15 years before Docker. I'm not *sure* whether it is actually the source of your problems, but simply by its nature of being a _container_ system (i.e. not a VM) which needs to virtualize cgroups as part of its job, it's very high on my suspect list. Can you re-check the permissions of /sys/fs/cgroup/systemd immediately after a reboot?

user1686

5/8/24, 8:09 AM

(You could actually disable the "user manager" feature entirely, if you're not using it for anything specific; it's not necessary for SSH in general.)

Pauchu

5/8/24, 8:32 AM

Permissions after reboot are rwx for owner and r-x for group and all, just as before, leading to the service failing as it can't create the file. I am not using any features of the user manager, that I am aware of, so disabling it might be the best option. I just wasn't sure if anything critical is tied to it.

user1686

5/8/24, 9:11 AM

Hmm, on a second thought (after coffee), the permissions _are_ supposed to be like that for /sys/fs/cgroup/systemd in general – however, systemd (the pid 1 system manager) is supposed to pre-create and `chown` /sys/fs/cgroup/systemd/user.slice/user-XXXX.slice/[email protected] to your own UID (due to the Delegate= option in [email protected]). Does that happen?

Pauchu

5/8/24, 9:22 AM

No, that doesn't happen, however I just had a look through the man pages and config files and as it turns out there's a file overriding defaults in both `/etc/systemd/user.conf.d/` and `/etc/systemd/system/[email protected]/`, probably put there by the provider on setup. Now one of there files has the option `[Service] Delegate=no` Might that be the reason?

user1686

5/8/24, 9:27 AM

Yeah, that sounds like it would be the reason – it's exactly the option that normally asks systemd to switch cgroup ownership for that particular service.

Pauchu

5/8/24, 9:41 AM

I just tested changing the option to yes and it resolved the issue. I'll only change it on my testing server for now, as there might be a reason the provider changed it, though I cannot think of one. I will answer this question myself, if that is alright for you. Thanks a lot, this was driving me crazy for months.

user1686

5/8/24, 10:20 AM

I would assume it used to cause some problems with their Virtuozzo/OpenVZ setup (might be a leftover from an older version, e.g. if VZ did not support cgroup delegation properly). The whole [email protected] _could_ be masked more or less safely (on headless servers it runs "empty" by default, though you can take a look via `systemctl --user` just to be sure).

Score:1

Server

Pauchu

5/8/24, 9:45 AM

As it turns out the provider left a config file in /etc/systemd/system/[email protected]/ setting the Delegate option to no, changing it to yes resolved the problem. Because of this option, systemd did not change the ownership of the directory the user manager tried to create files in to the user, hence the permission to create files didn't exist.

+ 0

Elon Musk

I sit in a Tesla and translated this thread with Ai:

EN: systemD user manager service keeps failing

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.