How to get Ubuntu 22 LXC containers on the same host to be able to resolve each other’s names?

Environment

Host is Ubuntu 22.04.2 LTS I have 2 sets of containers

1. Ubuntu 16.04.7 LTS containers which work fine
2. Ubuntu 22.04.2 LTS which I’m testing currently as I want to upgrade all containers eventually

$ apt list -i | grep lxc
liblxc-common/jammy,now 1:5.0.0~git2209-g5a7b9ce67-0ubuntu1 amd64 [installed,automatic]
liblxc1/jammy,now 1:5.0.0~git2209-g5a7b9ce67-0ubuntu1 amd64 [installed,automatic]
lxc-templates/jammy,now 3.0.4-5 amd64 [installed]
lxc-utils/jammy,now 1:5.0.0~git2209-g5a7b9ce67-0ubuntu1 amd64 [installed,automatic]
lxc/jammy,jammy,now 1:5.0.0~git2209-g5a7b9ce67-0ubuntu1 all [installed]
lxcfs/jammy,now 5.0.0-0ubuntu2 amd64 [installed,automatic]

$ apt list -i | grep dnsmasq
dnsmasq-base/jammy-updates,jammy-security,now 2.86-1.1ubuntu0.1 amd64 [installed,automatic]
Container creation
Created both the 16 and 22 containers in the same way, replacing the release name
lxc-create --template download --name {name} --logfile {log} -l trace -- --dist ubuntu --release {release} --arch amd64

Container creation and setup

1. Created both the 16 and 22 containers in the same way, replacing the release name

lxc-create --template download --name {name} --logfile {log} -l trace -- --dist ubuntu --release {release} --arch amd64

2. Also, I manually add each container’s IP in the host’s /etc/hosts file

3. “resetting dnsmasq cache” - sudo pkill -1 -u lxc-dnsmasq

The only difference is that in Ubuntu 16, I did ifup eth0, which isn't available in Ubuntu 22, so I did ip link set eth0 up instead. Is this related to my issue?

Issue

The issue is my /etc/resolv.conf file.

In my Ubuntu 16 containers, this had one simple line

nameserver 10.0.3.1

In my Ubuntu 22 containers, this contains

nameserver 127.0.0.53
options edns0 trust-ad
search .

My Ubuntu 16 containers can ping each other just fine. My Ubuntu 22 containers cannot. They also can’t ping the host by its name, which the Ubuntu 16 containers can

myuname@mycontainer-local:~$ sudo ping -v rabbitmq-local
ping: rabbitmq-local: Temporary failure in name resolution

(yes, I strangely need sudo on the 22 containers, otherwise it complains it’s not allowed to open a socket)

The thing is, I never configured /etc/resolv.conf or anything that would generate it on my Ubuntu 16 containers. So I have no clue how it knew that it should use 10.0.3.1, but on Ubuntu 22 it clearly doesn’t. Editing the /etc/resolv.conf on Ubuntu 22 to match the old file from Ubuntu 16, fixes the issue. But obviously this file is automatically generated, so this is only a temporary workaround. What is a more permanent solution?

I looked into potentially editing /etc/netplan/10-lxc.yaml but

1. I’m not 100% clear on what to edit
2. I’m not sure if it’s possible to get it working without manually adding 10.0.3.1 to the file. Something was configuring it automatically on Ubuntu 16 and I’d like to keep doing it automatically instead of just hard coding this IP in a conf file somewhere (though if there’s no other option, I guess I’ll go with that)

The only clue I have is that there might be something different between Ubuntu 16's ifup eth0 and Ubuntu 22's ip link set eth0 up

In later versions (possibly since 18.04), /etc/resolve.conf is automatically generated by systemd. Editing it, as you discovered, is temporary. For permanent change, you must edit one of the resolved.conf files under /etc/systemd.

I define the DNS servers and search domains by editing /etc/systemd/resolved.conf adding the following:

DNS=x.x.x.x y.y.y.y
FallbackDNS=z.z.z.z
Domains=mydomain.org corp.mydomain.org

(Note that nameserver will always be 127.0.0.53 in /etc/resolv.conf but lookups use those you defined)

Restart the service with systemctl restart systemd-resolved.service

Watch what resolved is up to with journalctl -u systemd-resolved -f

resolvectl is a very interesting tool if you have time to play with it (I haven't). It allows some configuration changes but I don't know if they are persistent.

See man 5 resolved.conf and man resolvectl for details.

/etc/resolv.conf in later Ubuntu releases always points to 127.0.0.53 which is the IP that systemd-resolved binds to for use as a local resolver. resolvectl will show you the configuration of where the local resolver forwards requests upstream. If resolvectl does not show your 10.0.3.1 nameserver as the configured upstream resolver, that is where you would want to investigate and fix the configuration.

If using lxd the general expectation is that the DNS will be provided via DHCP to the containers. In this case you are using lxc directly via lxc-create so I can't speak to that.