Score:0

Ubuntu

Unable to run systemctl commands on the host from the container which was working till Ubuntu 16.04

Aravindhan Krishnan

10/28/22, 4:30 AM

When the host OS is Ubuntu 16.04 or RHEL 7.x, the following command is working, helping us to run systemctl commands on the host from within the docker container:

# nsenter --mount=/hostroot/proc/1/ns/mnt -- systemctl start dummy.service

But in the newer host OS-es, Ubuntu 20.04 and RHEL 8.x, this doesn't work and we get the following error:

# nsenter --mount=/hostroot/proc/1/ns/mnt -- systemctl start dummy.service
Failed to connect to bus: No data available

I have attached a minimalistic example and commands to run and reproduce the issue:

Sample service which I wanted to start on the host, from the container:

# cat /etc/systemd/system/dummy.service
[Unit]
Description=dummy service
[Service]
ExecStart=/usr/bin/sleep infinity

Dockerfile of my container:

# cat Dockerfile
FROM ubuntu:20.04
ENV TZ=UTC
RUN ln -snf /usr/share/zoneinfo/$TZ /etc/localtime && echo $TZ > /etc/timezone
RUN apt update -y --fix-missing
RUN apt install -y util-linux
STOPSIGNAL SIGRTMIN+3
CMD [ "/bin/bash" ]

Build the image:

# docker build -t trial .

Delete any stale containers:

# docker rm -f trial

Run the image:

# docker run -it -d --net=host --privileged -v /:/hostroot -v /sys/fs/cgroup:/sys/fs/cgroup:ro --name trial trial

Reproduce the issue:

# docker exec -it trial bash
# nsenter --mount=/hostroot/proc/1/ns/mnt -- systemctl start dummy.service
Failed to connect to bus: No data available

I would like to know if there are any additional options or any docker run command that needs to be altered to get this working.

234

0 + 0

cgroup

systemd

docker

16.04

20.04

Score:0

Ubuntu

Gakio

5/11/23, 4:37 PM

I looked into this for a good several hours, it seems the method sd_bus_start was changed to include additional checks. I wasn't able to narrow down what else it is looking for, however I was able to come up with a more elegant solution to accomplish the same task using remote systemctl commands instead of mounting all the directories from the host.

Remote systemctl

systemctl supports remote commands via the --host / -H flag. It is using ssh to connect to the remote host, so an ssh key pair will be needed. Since we are controlling the host we are on, this is pretty straightforward to setup.

Docker command (or Kubernetes arg)

Here is the full command that can be used, I will break down each part below. The assumptions of the container are that it has systemctl and ssh installed, the container is running on the host network, and that the root account's home directory is mounted (you can use another use if you want).

(ls ~/.ssh/id_rsa || ssh-keygen -b 2048 -t rsa -f ~/.ssh/id_rsa -q -N "") 
  && (grep -qxF $(cat ~/.ssh/id_rsa.pub) ~/.ssh/authorized_keys || echo $(cat ~/.ssh/id_rsa.pub) > ~/.ssh/authorized_keys)
  && (grep -qxF "StrictHostKeyChecking no" ~/.ssh/config || echo "StrictHostKeyChecking no" >> ~/.ssh/config)
  && (grep -qxF "UserKnownHostsFile /dev/null" ~/.ssh/config || echo "UserKnownHostsFile /dev/null" >> ~/.ssh/config)
  && systemctl -H [email protected] start nfs-server.service

This command is seeing if the ~/.ssh/id_rsa file exists, otherwise create one.

(ls ~/.ssh/id_rsa || ssh-keygen -b 2048 -t rsa -f ~/.ssh/id_rsa -q -N "")

Now we add our public key to our authorized keys if it does not exist in the file already.

(grep -qxF "$(cat ~/.ssh/id_rsa.pub)" ~/.ssh/authorized_keys || echo "$(cat ~/.ssh/id_rsa.pub)" > ~/.ssh/authorized_keys)

This can probably be made more secure by putting it in a section of the ssh config only for 127.0.0.1, but we need

(grep -qxF "StrictHostKeyChecking no" ~/.ssh/config || echo "StrictHostKeyChecking no" >> ~/.ssh/config) 
  && (grep -qxF "UserKnownHostsFile /dev/null" ~/.ssh/config || echo "UserKnownHostsFile /dev/null" >> ~/.ssh/config)

Finally we have the actual systemctl command. Notice the -H [email protected].

systemctl -H [email protected] start nfs-server.service

For maximum security, it would be best to setup the keys and users outside of the containers first (via Ansible or similar) and only allow the systemctl -H command inside the container.

0 + 0

Aravindhan Krishnan

5/15/23, 6:48 AM

Thanks for spending your valuable time on this. We also internally arrived at a very similar inference (though via observation-only & not code reading) that some additional new checks are causing the change in behavior. We also noted that in earlier OS-es, such command execution resulted in a crash. Likely systemd-dev fixed this issue which is causing behavioral anomalies to the end-user. The suggested solution by you is the same one we got recommended from canonical as well. Thanks

Elon Musk

I sit in a Tesla and translated this thread with Ai:

EN: Unable to run systemctl commands on the host from the container which was working till Ubuntu 16.04

TH: ไม่สามารถเรียกใช้คำสั่ง systemctl บนโฮสต์จากคอนเทนเนอร์ซึ่งทำงานจนถึง Ubuntu 16.04

RO: Nu se pot rula comenzi systemctl pe gazdă din containerul care funcționa până la Ubuntu 16.04

RU: Не удалось запустить команды systemctl на хосте из контейнера, который работал до Ubuntu 16.04.

VI: Không thể chạy các lệnh systemctl trên Máy chủ từ vùng chứa đang hoạt động cho đến Ubuntu 16.04

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.