Score:1

Network manager is occasionally not running after startup

cn flag

The issue is that sometimes after boot NetworkManager service is not running. It works fine to start it manually sudo service network-manager restart or reboot the computer but since I want to access these computers remotely that's not always an option. Most times booting works fine (NetworkManager is running) so its hard to reproduce, say it only happens once every 20th boot.

I have seen this issue on two desktops running Ubuntu 20.04.2 LTS and kernel 5.10.25-051025

After it has happened I have tried to look through the journal but the only thing that stick out is this line gnome-shell3166: JS ERROR: error initializing the NetworkManager Agent: NM.SecretAgentError: NetworkManager is not running

The computers are also running as NFS servers (if that could be related) and eth/network/interfaces looks like this

auto lo
iface lo inet loopback

Any ideas how I can determine the reason why NetworkManager is sometimes not running and how to ensure it is always running after boot (so I can access the computer remotely)?

Updates

Running status --full --no-pager network-manager returns only this:

NetworkManager.service - Network Manager
     Loaded: loaded (/lib/systemd/system/NetworkManager.service; enabled; vendor preset: enabled)
     Active: inactive (dead)
       Docs: man:NetworkManager(8)

Found these lines in journalctl. can they be relevant?

dec 13 13:34:56 dataPC2 systemd[1]: network.target: Found ordering cycle on NetworkManager.service/start
dec 13 13:34:56 dataPC2 systemd[1]: network.target: Found dependency on basic.target/start
dec 13 13:34:56 dataPC2 systemd[1]: network.target: Found dependency on sockets.target/start
dec 13 13:34:56 dataPC2 systemd[1]: network.target: Found dependency on avahi-daemon.socket/start
dec 13 13:34:56 dataPC2 systemd[1]: network.target: Found dependency on sysinit.target/start
dec 13 13:34:56 dataPC2 systemd[1]: network.target: Found dependency on local-fs.target/start
dec 13 13:34:56 dataPC2 systemd[1]: network.target: Found dependency on mnt-NAS_data.mount/start
dec 13 13:34:56 dataPC2 systemd[1]: network.target: Found dependency on network.target/start
dec 13 13:34:56 dataPC2 systemd[1]: network.target: Job NetworkManager.service/start deleted to break ordering cycle starting with network.target/start
dec 13 13:34:56 dataPC2 systemd[1]: network.target: Found ordering cycle on wpa_supplicant.service/start
dec 13 13:34:56 dataPC2 systemd[1]: network.target: Found dependency on basic.target/start
dec 13 13:34:56 dataPC2 systemd[1]: network.target: Found dependency on sockets.target/start
dec 13 13:34:56 dataPC2 systemd[1]: network.target: Found dependency on avahi-daemon.socket/start
dec 13 13:34:56 dataPC2 systemd[1]: network.target: Found dependency on sysinit.target/start
dec 13 13:34:56 dataPC2 systemd[1]: network.target: Found dependency on local-fs.target/start
dec 13 13:34:56 dataPC2 systemd[1]: network.target: Found dependency on mnt-NAS_data.mount/start
dec 13 13:34:56 dataPC2 systemd[1]: network.target: Found dependency on network.target/start
dec 13 13:34:56 dataPC2 systemd[1]: network.target: Job wpa_supplicant.service/start deleted to break ordering cycle starting with network.target/start
dec 13 13:34:56 dataPC2 systemd[1]: network.target: Found ordering cycle on systemd-resolved.service/start
dec 13 13:34:56 dataPC2 systemd[1]: network.target: Found dependency on systemd-tmpfiles-setup.service/start
dec 13 13:34:56 dataPC2 systemd[1]: network.target: Found dependency on local-fs.target/start
dec 13 13:34:56 dataPC2 systemd[1]: network.target: Found dependency on mnt-NAS_data.mount/start
dec 13 13:34:56 dataPC2 systemd[1]: network.target: Found dependency on network.target/start
dec 13 13:34:56 dataPC2 systemd[1]: network.target: Job systemd-resolved.service/start deleted to break ordering cycle starting with network.target/start

The mnt-NAS_data.mount is something we have created, could it disturb the network manager perhaps?

[Unit]
Description=NAS_2
After=network.target

[Mount]
What=/dev/disk/by-label/storage
Where=/mnt/NAS_data
Type=ext4
TimeoutSec=20

[Install]
WantedBy=multi-user.target
user10489 avatar
in flag
When it fails to start, please cut and paste the output of `systemctl status --full --no-pager network-manager` and add it to the question.
user10489 avatar
in flag
inactive/dead is interesting...next step is to dig through the logs and find out why it is dead.
Emil avatar
cn flag
Thank you for your support and help. I tried to add journalctl -b output to my answer but it was too long. Any tip on a good way to filter it or if there are any other logs that might be of interest? I also updated the question with some other lines that I found from journalctl that might be relevant.
user10489 avatar
in flag
The dependency loop is the problem.
Score:0
in flag

The additional logs and config files you added to your question indicate there is a dependency loop in your systemd config. Probably network manager starts sometimes because the loop isn't always broken in the same place.

Physical disks have an implicit dependency to mount before the network is up. Because you have listed ext4 as the type and listed what looks to be a physical device, systemd thinks this is a physical device that must be mounted before the network is up.

By adding an After=network dependency on this target, you have created a loop.

The systemd.mount man page suggets that if this is a network block device, you need to add _netdev to the mount option string. Normally this would go in /etc/fstab with the rest of the mount options, but you may be able add it in a systemd config file with Options=_netdev in the mount section.

If this is not a network block device or otherwise reliant on the network, you should remove the network dependency instead.

Emil avatar
cn flag
Awesome! Thanks a lot for good explanation and solution. Indeed its a physical disk so removing After=network from the config file solved it. No issue after 200 reboots :)
mangohost

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.