Score:3

How can I force Ubuntu 22.04 SERVER to use ONLY the DNS servers listed in /etc/systemd/resolved.conf (NOT my router's DNS) AND use DNS-over-TLS?

fk flag

I have added these lines to /etc/systemd/resolved.conf in a Ubuntu 22.04 SERVER install with NO DESKTOP - I emphasize that because I can't use the normal desktop method of setting network options in Ubuntu:

DNS=9.9.9.9 149.112.112.112 1.1.1.1 1.0.0.1
DNSOverTLS=yes

This works fine EXCEPT that all connections are supposed to be going through a VPN tunnel which is identified as tun0. So with the above configuration in place, if I do

sudo tcpdump -ni tun0 -p port 53 or port 853

in one terminal window and try to ping google.com from the other, I see several requests but they are all going out on port 853. Great, so DNS-over-TLS is working. The problem is that if instead I run

sudo tcpdump -ni enp0s10 -p port 53 or port 853

I see lines showing that it is also trying to go to my router at 192.168.1.1, still trying to use port 853 but it looks like it makes four attempts to go there. This is NOT great. I don't want it trying to use my router's DNS, all that traffic should be going through the tunnel.

I tried an alternate approach which was to temporarily rename /etc/resolv.conf (which is a symlink to somewhere else) and replaced it with a version that is an actual text file (not a symlink) in which I put these lines:

nameserver 9.9.9.9
nameserver 149.112.112.112
nameserver 1.1.1.1
options use-vc edns0 trust-ad
search localdomain

Other than the nameserver links the only thing I changed was adding "use-vc" in the options line because I read that was SUPPOSED to enable DNS-over-TLS. When I do that I get no traffic going to the local network interface when I do a DNS request, BUT the requests go out tun0 using port 53, not port 853, so then it is NOT using DNS-over-TLS. So the use-vc option is not being honored.

I have been reading pages for the last five hours and I can't find a solution to this, primarily because most pages on the subject either assume you are running a desktop version of Ubuntu or they appear to be for different versions of Ubuntu where things are done a little differently. How can I make it so DNS requests only go out tun0, but still use DNS-over-TLS?

EDIT to answer questions asked - I went back to using the static resolv.conf because it was the only way I could be certain traffic would go out the VPN but as noted above DNS requests are going out on port 53. not port 853. With that in mind, this is the output of resolvectl status:

Global
       Protocols: -LLMNR -mDNS +DNSOverTLS DNSSEC=no/unsupported
resolv.conf mode: foreign
     DNS Servers: 9.9.9.9 149.112.112.112 1.1.1.1 1.0.0.1

Link 2 (enp0s10)
Current Scopes: DNS
     Protocols: +DefaultRoute +LLMNR -mDNS +DNSOverTLS DNSSEC=no/unsupported
   DNS Servers: 192.168.1.1
    DNS Domain: localdomain

Link 3 (tun0)
Current Scopes: none
     Protocols: -DefaultRoute +LLMNR -mDNS +DNSOverTLS DNSSEC=no/unsupported

Link 4 (wlp5s0)
Current Scopes: none
     Protocols: -DefaultRoute +LLMNR -mDNS +DNSOverTLS DNSSEC=no/unsupported

This is the output of ip a (with MAC addresses redacted):

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
2: enp0s10: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether xx:xx:xx:xx:xx:xx brd ff:ff:ff:ff:ff:ff
    inet 192.168.1.120/24 metric 100 brd 192.168.1.255 scope global dynamic enp0s10
       valid_lft 6592sec preferred_lft 6592sec
3: tun0: <POINTOPOINT,MULTICAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UNKNOWN group default qlen 500
    link/none
    inet 10.12.216.2/24 scope global tun0
       valid_lft forever preferred_lft forever
4: wlp5s0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether xx:xx:xx:xx:xx:xx brd ff:ff:ff:ff:ff:ff

I am not certain about netplan or systemd, but I ASSUME netplan because there is a file /etc/netplan/00-installer-config.yaml which contains the following:

# This is the network config written by 'subiquity'
network:
  ethernets:
    enp0s10:
      dhcp4: true
  version: 2

However please remember what I said about resolv.conf being made into a static file.

The VPN is OpenVPN, it connects to an OpenVPN server at a distant location. To make it work I have to run this at every startup:

/usr/sbin/openvpn --client --config /etc/openvpn/client/xBuQqzYnLLTezXRvrOofMBGgYAEgcY.ovpn

For security reasons I don't really want to post that entire .ovpn file and I am not sure it would be helpful anyway, but here is the top part with some redactions:

client
dev tun
proto udp
remote (redacted)
nobind
remote-cert-tls server
tls-version-min 1.2
verify-x509-name (redacted)
cipher AES-256-CBC
auth SHA256
auth-nocache
askpass (redacted)
verb 3
script-security 2
up /etc/openvpn/update-resolv-conf
down /etc/openvpn/update-resolv-conf
redirect-gateway
dhcp-option DNS 10.12.216.2

After that are the various keys and certificates.

I really don't think the issue is in the .ovpn configuration, I think it is that if you use the default symlinked resolv.conf it tries to go to the router first (192.168.1.1). In case you are wondering what that version of resolv.conf looks like, it is

nameserver 127.0.0.53
options edns0 trust-ad
search localdomain

Which I believe is the default on all Ubuntu versions. Now what exactly 127.0.0.53 is I am not sure, I gather it is some kind of local DNS server or maybe a proxy or something, but all I do know is that it tries the router's DNS server first and that is exactly what I don't want happening.

Regarding that use-vc option, for some reason when I read the details on that I saw TCP and read TLS, so that explains why it didn't work. TCP and TLS aren't the same thing but when you have researched something long enough, with no results, small distinctions like that start to blur!

mpboden avatar
do flag
According to [man resolv.conf](https://man7.org/linux/man-pages/man5/resolv.conf.5.html), the option, `use-vc`, states: `This option forces the use of TCP for DNS resolutions.` I’m guessing that this isn’t what you want. So it’s not the solution to the problem. Meanwhile, could you update your post to include the output of `resolvectl status` and `ip a`? Also, are you using Netplan or systemd-networkd to setup your network? Could you provide more info such as your .conf files? Finally, any info about your VPN setup might help as well. Static or DHCP?
UbuntuUser avatar
fk flag
mpboden, edited the post to try to answer your questions.
mpboden avatar
do flag
127.0.0.53 is the local stub resolver. This might help explain: [askubuntu post](https://askubuntu.com/questions/1462604/what-is-dns-resolver-associated-with-a-network-interface/1462613#1462613) Also, see the man page for systemd-resolved: [systemd-resolved](https://man7.org/linux/man-pages/man8/systemd-resolved.service.8.html)
Score:3
do flag

It seems that you've found a solution to your problem by adding the Global Domain Route of ~. to the /etc/systemd/resolved.conf configuration file. While this appears to work, I feel this approach is limiting in the sense that you're only able to affect changes for global DNS settings within this file and not specific links. But there's a way to get around this roadblock via the D-Bus interface.

D-Bus is an interprocess communication system that allows messages to be sent to various services. In this case, you want to communicate with systemd-resolved to change DNS settings on individual links instead of just globally with the /etc/systemd/resolved.conf file. There are several ways to communicate with the D-Bus, from a low-level D-Bus API to higher-level bindings. But this can get a bit complicated, so we'll take advantage of a helper script for your openvpn setup to keep things as easy as possible. Within this script, the command, busctl, is used to communicate with the D-Bus and change settings on individual links. This is more informational than useful at this point, because you won't be interacting with busctl directly. That's not to say that you can't or won't in the future, though.

Suffice it to say, the Global Domain Route of ~. is key to controlling the default route of DNS queries. It basically says that DNS queries for all domains not specifically listed on any other link will be routed to the DNS servers associated with the link that the Global Domain Route is set to. That's a mouthful, but hopefully this will make more sense as we proceed.

I'd also like to mention that an advantage of editing individual links instead of defining global DNS settings is that it'll allow more flexibility in the future for situations like Split DNS. I won't go into full detail here, but an explanation of Split DNS as well as routing domains and search domains can be found at these links:

Anyway, I feel that the approach I'm outlining below is probably more in line with the intended use of systemd-resolved than my previous answer. Hence, this is my preferred answer and the way that I would configure my system.

Let's get started...


Setup /etc/resolv.conf

Make sure /etc/resolv.conf is a symlink to /run/systemd/resolve/stub-resolv.conf. This is the default setup in Ubuntu 22.04. Even though the local resolver at 127.0.0.53 is used, it still queries any upstream DNS servers for records not in it's cache. It should look like the following:

$ ls -l /etc/resolv.conf
lrwxrwxrwx 1 root root 39 Apr 22 23:47 /etc/resolv.conf -> ../run/systemd/resolve/stub-resolv.conf

If /etc/resolv.conf is a static file or a symlink to anything other than /run/systemd/resolve/stub-resolv.conf, do the following:

$ sudo rm /etc/resolv.conf
$ sudo ln -s /run/systemd/resolve/stub-resolv.conf /etc/resolv.conf

Setup /etc/systemd/resolved.conf

Next, don't set any global DNS servers in /etc/systemd/resolved.conf. We'll set specific DNS servers later for your VPN tunnel elsewhere.

Regarding settings for DNS-over-TLS, you can either set them globally or for any specific link. For this setup, we'll set them globally in /etc/systemd/resolved.conf so that it affects all servers on all links. Additionally, let's set it to opportunistic instead of yes so that it'll succeed whether the server supports DNS-over-TLS or not. If you want to enforce DNS-over-TLS and fail if it isn't supported, then change this to yes.

Edit /etc/systemd/resolved.conf as follows:

#DNS=
#FallbackDNS=
#Domains=
#DNSSEC=no
DNSOverTLS=opportunistic
#MulticastDNS=no
#LLMNR=no
#Cache=no-negative
#CacheFromLocalhost=no
#DNSStubListener=yes
#DNSStubListenerExtra=
#ReadEtcHosts=yes
#ResolveUnicastSingleLabel=no  

                       

Setup openvpn

With openvpn, it's typical that DNS servers are pushed to the client, yet the system isn't automatically configured to use them. This is because the process is slightly different on various operating systems. With Ubuntu, an openvpn installation typically provides a helper script to implement the DNS servers to redirect queries through the tunnel, thus preventing DNS leaks. This is the helper script I mentioned in the beginning that we are going to use.

Here is the current configuration of your .ovpn file:

client
dev tun
proto udp
remote (redacted)
nobind
remote-cert-tls server
tls-version-min 1.2
verify-x509-name (redacted)
cipher AES-256-CBC
auth SHA256
auth-nocache
askpass (redacted)
verb 3
script-security 2
up /etc/openvpn/update-resolv-conf
down /etc/openvpn/update-resolv-conf
redirect-gateway
dhcp-option DNS 10.12.216.2

Look at the following lines from your .ovpn file:

script-security 2
up /etc/openvpn/update-resolv-conf
down /etc/openvpn/update-resolv-conf

This section defines a helper script called, update-resolv-conf, that runs when the VPN goes up and down. Within the script, the application calls /sbin/resolvconf. But this will fail in the default installation of Ubuntu 22.04, because /sbin/resolvconf isn't installed. This is because your system is running systemd and uses systemd-resolved.

/sbin/resolvconf is a framework for managing /etc/resolv.conf and modifies the config file directly, but this file is no longer used in the same fashion as before. (As you're now aware, it's a symlink to /run/systemd/resolve/stub-resolv.conf.) Needless to say, you need a different helper script called, update-systemd-resolved which uses busctl to access the D-Bus interface.

Install the helper script with the following:

sudo apt install openvpn-systemd-resolved

This places the script in /etc/openvpn.

After this script is installed, modify your .ovpn file and replace the previous lines with the following:

script-security 2
up /etc/openvpn/update-systemd-resolved
down /etc/openvpn/update-systemd-resolved
down-pre
dhcp-option DOMAIN-ROUTE .

You'll need to reboot the system for this to take effect, but first look at the current output of resolvectl:

Before any changes to .ovpn:

$ resolvectl
Global
       Protocols: -LLMNR -mDNS +DNSOverTLS DNSSEC=no/unsupported
resolv.conf mode: stub
     DNS Servers:

Link 2 (enp0s10)
Current Scopes: DNS
     Protocols: +DefaultRoute +LLMNR -mDNS +DNSOverTLS DNSSEC=no/unsupported
   DNS Servers: 192.168.1.1
    DNS Domain: localdomain

Link 3 (tun0)
Current Scopes: none
     Protocols: -DefaultRoute +LLMNR -mDNS +DNSOverTLS DNSSEC=no/unsupported

Link 4 (wlp5s0)
Current Scopes: none
     Protocols: -DefaultRoute +LLMNR -mDNS +DNSOverTLS DNSSEC=no/unsupported

Reboot your system and run resolvectl again. Take note of the following changes:

  • With the addition of dhcp-option DOMAIN-ROUTE . to your .ovpn file, the Global Routing Domain ~. has been added to Link 3 (tun0).
  • The -DefaultRoute Protocol has changed to +DefaultRoute on Link 3 (tun0).
  • A DNS server 10.12.216.2 has been added to Link 3 (tun0). This is because in your .ovpn file, the line dhcp-option DNS 10.12.216.2 was included. (There might also be additional DNS servers listed for Link 3 (tun0) that were pushed to you from the VPN server.)

After changes to .ovpn:

$ resolvectl
Global
       Protocols: -LLMNR -mDNS +DNSOverTLS DNSSEC=no/unsupported
resolv.conf mode: stub
     DNS Servers:

Link 2 (enp0s10)
Current Scopes: DNS
     Protocols: +DefaultRoute +LLMNR -mDNS +DNSOverTLS DNSSEC=no/unsupported
   DNS Servers: 192.168.1.1
    DNS Domain: localdomain

Link 3 (tun0)
Current Scopes: none
     Protocols: +DefaultRoute +LLMNR -mDNS +DNSOverTLS DNSSEC=no/unsupported
Current DNS Server: 10.12.216.2
       DNS Servers: 10.12.216.2
        DNS Domain: ~. 
        
Link 4 (wlp5s0)
Current Scopes: none
     Protocols: -DefaultRoute +LLMNR -mDNS +DNSOverTLS DNSSEC=no/unsupported    

At this point, the tunnel will be using 10.12.216.2 as your default DNS server for all domain queries. But if you want to use the DNS servers you originally listed -- and only them --, 9.9.9.9 and 149.112.112.112, then you need to do the following:

  • Ignore any DNS servers pushed to you from your VPN server.
  • Specifically list the DNS servers in your .ovpn file that you want to use.
  • Omit the current DNS server listed in your .ovpn file

All of this can be done by adding the following to your .ovpn file:

pull-filter ignore "dhcp-option DNS"
dhcp-option DNS 9.9.9.9
dhcp-option DNS 149.112.112.112

And don't forget to delete or comment out dhcp-option DNS 10.12.216.2 that's currently in your .ovpn file.

After the changes, your .ovpn file should look like the following:

client
dev tun
proto udp
remote (redacted)
nobind
remote-cert-tls server
tls-version-min 1.2
verify-x509-name (redacted)
cipher AES-256-CBC
auth SHA256
auth-nocache
askpass (redacted)
verb 3
script-security 2
up /etc/openvpn/update-systemd-resolved
down /etc/openvpn/update-systemd-resolved
down-pre
dhcp-option DOMAIN-ROUTE .
redirect-gateway
pull-filter ignore "dhcp-option DNS"
dhcp-option DNS 9.9.9.9
dhcp-option DNS 149.112.112.112

Reboot and run resolvectl. Your DNS servers have been added to Link 3 (tun0). Any DNS servers previously listed in your .ovpn file or pushed to you from your VPN server are now gone.

$ resolvectl
Global
       Protocols: -LLMNR -mDNS +DNSOverTLS DNSSEC=no/unsupported
resolv.conf mode: stub
     DNS Servers:

Link 2 (enp0s10)
Current Scopes: DNS
     Protocols: +DefaultRoute +LLMNR -mDNS +DNSOverTLS DNSSEC=no/unsupported
   DNS Servers: 192.168.1.1
    DNS Domain: localdomain

Link 3 (tun0)
Current Scopes: none
     Protocols: +DefaultRoute +LLMNR -mDNS +DNSOverTLS DNSSEC=no/unsupported
Current DNS Server: 9.9.9.9
       DNS Servers: 9.9.9.9 149.112.112.112
        DNS Domain: ~.
        
Link 4 (wlp5s0)
Current Scopes: none
     Protocols: -DefaultRoute +LLMNR -mDNS +DNSOverTLS DNSSEC=no/unsupported

Conclusion

With your system setup like this, the Global Routing Domain of ~. that is set on Link 3 (tun0) states that all DNS queries for domains not specifically listed on a different link will be routed to the DNS servers listed for Link 3 (tun0). As an example, if you had ~example.com set as a routing domain on Link 2 (enp0s10), then any DNS queries for domains ending in example.com would be routed to your local DNS server at 192.168.1.1. All others would be sent to the DNS servers listed for Link 3 (tun0).

As you can see, this is highly customizable. If in the future you want to also configure another VPN (either corporate or private) with specific DNS queries going through that route, then you can do so.


Extra

As a final note, I mentioned earlier that the helper script that was installed makes changes to the system via the D-Bus, specifically with the command busctl. Feel free to inspect the script and checkout the man page for busctl here.

An easier interface to use, however, and a command that you're familiar with, is resolvectl, which is a thin wrapper around the D-Bus interface. Check out the following examples:

Set the Global Routing domain for tun0:

sudo resolvectl domain tun0 ~.

Clear all domains for tun0:

sudo resolvectl domain tun0 ""

Set tun0 as a default route:

sudo resolvectl default-route tun0 yes

Define DNS server for tun0:

sudo resolvectl dns tun0 9.9.9.9

Clear DNS Servers for tun0:

sudo resolvectl dns tun0 ""

Set DNS-over-TLS for tun0:

sudo resolvectl dnsovertls tun0 yes

With these commands and many more not shown, you can change settings for individual links. And what's cool is that the change is immediate. There's no need to reboot or restart a service, so it's a fun way to play and experiment. Unfortunately, these changes won't survive a reboot, so that's why the helper script used above is beneficial. Without it, you'd have to write your own.

To learn more about changing DNS settings per-link using resolvectl, check out the man page here. Meanwhile, here's the relevant section:

dns [LINK [SERVER...]], domain [LINK [DOMAIN...]], default-route
[LINK [BOOL...]], llmnr [LINK [MODE]], mdns [LINK [MODE]], dnssec
[LINK [MODE]], dnsovertls [LINK [MODE]], nta [LINK [DOMAIN...]]
    Get/set per-interface DNS configuration. These commands may
    be used to configure various DNS settings for network
    interfaces. These commands may be used to inform
    systemd-resolved or systemd-networkd about per-interface DNS
    configuration determined through external means. The dns
    command expects IPv4 or IPv6 address specifications of DNS
    servers to use. Each address can optionally take a port
    number separated with ":", a network interface name or index
    separated with "%", and a Server Name Indication (SNI)
    separated with "#". When IPv6 address is specified with a
    port number, then the address must be in the square brackets.
    That is, the acceptable full formats are
    "111.222.333.444:9953%ifname#example.com" for IPv4 and
    "[1111:2222::3333]:9953%ifname#example.com" for IPv6. The
    domain command expects valid DNS domains, possibly prefixed
    with "~", and configures a per-interface search or route-only
    domain. The default-route command expects a boolean
    parameter, and configures whether the link may be used as
    default route for DNS lookups, i.e. if it is suitable for
    lookups on domains no other link explicitly is configured
    for. The llmnr, mdns, dnssec and dnsovertls commands may be
    used to configure the per-interface LLMNR, MulticastDNS,
    DNSSEC and DNSOverTLS settings. Finally, nta command may be
    used to configure additional per-interface DNSSEC NTA
    domains.

    Commands dns, domain and nta can take a single empty string
    argument to clear their respective value lists.

    For details about these settings, their possible values and
    their effect, see the corresponding settings in
    systemd.network(5).
Score:1
fk flag

Well after trying a bunch of stuff including the suggestions made in other answers, what actually tentatively appears to be working is this:

Lines added to /etc/systemd/resolved.conf:

Domains=~.
DNS=9.9.9.9 149.112.112.112
FallbackDNS=1.1.1.1 1.0.0.1
DNSOverTLS=yes
DNSSEC=allow-downgrade

Use the default resolv.conf which is by default symlinked to

../run/systemd/resolve/stub-resolv.conf

Use the default netplan (no dhcp4-overrides additions!)

Now if I run sudo tcpdump -ni enp0s10 -p port 53 or port 853 I see no output, whereas if I run sudo tcpdump -ni tun0 -p port 53 or port 853 I see DNS requests going out on port 853. Though, strangely, if I stop tcpdump and then restart it, it sometimes won't show anything, but the DNS requests appear to still work. Rebooting fixes that but I am not sure why that happens.

If I run resolvectl status I see this:

Global
           Protocols: -LLMNR -mDNS +DNSOverTLS DNSSEC=allow-downgrade/supported
    resolv.conf mode: stub
  Current DNS Server: 9.9.9.9
         DNS Servers: 9.9.9.9 149.112.112.112
Fallback DNS Servers: 1.1.1.1 1.0.0.1
          DNS Domain: ~.

Link 2 (enp0s10)
    Current Scopes: DNS
         Protocols: +DefaultRoute +LLMNR -mDNS +DNSOverTLS DNSSEC=allow-downgrade/supported
Current DNS Server: 192.168.1.1
       DNS Servers: 192.168.1.1
        DNS Domain: localdomain

Link 3 (tun0)
Current Scopes: none
     Protocols: -DefaultRoute +LLMNR -mDNS +DNSOverTLS DNSSEC=allow-downgrade/supported

Link 4 (wlp5s0)
Current Scopes: none
     Protocols: -DefaultRoute +LLMNR -mDNS +DNSOverTLS DNSSEC=allow-downgrade/supported

My suspicion is that it is the Domains=~. in /etc/systemd/resolved.conf that makes the difference. The page at https://www.internetsociety.org/blog/2018/12/dns-privacy-in-linux-systemd/ is where I found that, and it explains as follows:

Finally, setting ‘Domains’ to ‘~.’ instructs ‘systemd-resolved’ to prefer the specified nameserver over any per-link DNS server that may be available. This is an important setting as otherwise a non-DoT per-link DNS resolver could take precedence over the DoT resolver.

Now honestly I don't completely understand that but in practice it seems to prevent DNS-over-TLS requests from being sent first to my router. It was a case of grasping at straws, I saw this and thought it was worth a try and it APPEARS to be working.

That said, getting this to work really should not have been this hard! I have spent many hours trying to make this work and just because it worked for me tonight, I would not guarantee it will work for anyone else, particularly if you are running anything other than Ubuntu 22.04 server!

Thank you to everyone who made suggestions! In particularly I thought the approach that modified /etc/netplan/00-installer-config.yaml should have worked, but the minute I added those "dhcp4-overrides:" and "use-dns: false" line I had no DNS at all, and had to reboot to get it back!

Score:0
do flag

UPDATE

This is no longer my preferred answer. I've added another answer here.


There are several steps needed to get this to work.

  1. Change symlink for /etc/resolv.conf
  2. Configure /etc/systemd/resolved.conf with new DNS servers and enable DNS-over-TLS
  3. Configure Netplan to not use DNS servers pushed via DHCP

1. Change symlink for /etc/resolv.conf

The default setup of Ubuntu 22.04 is to use the local stub DNS caching server located at 127.0.0.53. In this default state, /etc/resolv.conf is a symlink pointing to /run/systemd/resolve/stub-resolv.conf. But there are four different modes of handling /etc/resolv.conf. So there's flexibility here in configuration.

  • Mode 1 (stub): use stub resolver with search domains using symlink /etc/resolv.conf pointing to /run/systemd/resolve/stub-resolv.conf

  • Mode 2 (static): use stub resolver without search domains using symlink /etc/resolv.conf pointing to /usr/lib/systemd/resolv.conf

  • Mode 3 (uplink): use uplink DNS nameservers using symlink /etc/resolv.conf pointing to /run/systemd/resolve/resolv.conf

  • Mode 4 (foreign): use static DNS nameservers by editing /etc/resolv.conf directly

See the manpage for systemd-resolved:

/ETC/RESOLV.CONF         top

       Four modes of handling /etc/resolv.conf (see resolv.conf(5)) are
       supported:

       •   systemd-resolved maintains the
           /run/systemd/resolve/stub-resolv.conf file for compatibility
           with traditional Linux programs. This file lists the
           127.0.0.53 DNS stub (see above) as the only DNS server. It
           also contains a list of search domains that are in use by
           systemd-resolved. The list of search domains is always kept
           up-to-date. Note that /run/systemd/resolve/stub-resolv.conf
           should not be used directly by applications, but only through
           a symlink from /etc/resolv.conf. This file may be symlinked
           from /etc/resolv.conf in order to connect all local clients
           that bypass local DNS APIs to systemd-resolved with correct
           search domains settings. This mode of operation is
           recommended.

       •   A static file /usr/lib/systemd/resolv.conf is provided that
           lists the 127.0.0.53 DNS stub (see above) as only DNS server.
           This file may be symlinked from /etc/resolv.conf in order to
           connect all local clients that bypass local DNS APIs to
           systemd-resolved. This file does not contain any search
           domains.

       •   systemd-resolved maintains the
           /run/systemd/resolve/resolv.conf file for compatibility with
           traditional Linux programs. This file may be symlinked from
           /etc/resolv.conf and is always kept up-to-date, containing
           information about all known DNS servers. Note the file
           format's limitations: it does not know a concept of
           per-interface DNS servers and hence only contains system-wide
           DNS server definitions. Note that
           /run/systemd/resolve/resolv.conf should not be used directly
           by applications, but only through a symlink from
           /etc/resolv.conf. If this mode of operation is used local
           clients that bypass any local DNS API will also bypass
           systemd-resolved and will talk directly to the known DNS
           servers.

       •   Alternatively, /etc/resolv.conf may be managed by other
           packages, in which case systemd-resolved will read it for DNS
           configuration data. In this mode of operation
           systemd-resolved is consumer rather than provider of this
           configuration file.

       Note that the selected mode of operation for this file is
       detected fully automatically, depending on whether
       /etc/resolv.conf is a symlink to /run/systemd/resolve/resolv.conf
       or lists 127.0.0.53 as DNS server.

For your particular situation, we are going to use Mode 3 (uplink) by creating a symlink that points to: /run/systemd/resolve/resolv.conf. This is because this file is automatically updated with DNS servers defined in both Netplan and from within /etc/systemd/resolv.conf, which we are going to edit next.

But before editing /etc/systemd/resolv.conf, if you run cat /etc/resolv.conf, you'll see that it will currently define the DNS server pushed to you via DHCP:

$ cat /run/systemd/resolve/resolv.conf
# This is /run/systemd/resolve/resolv.conf managed by man:systemd-resolved(8).
# Do not edit.
#
# This file might be symlinked as /etc/resolv.conf. If you're looking at
# /etc/resolv.conf and seeing this text, you have followed the symlink.
#
# This is a dynamic resolv.conf file for connecting local clients directly to
# all known uplink DNS servers. This file lists all configured search domains.
#
# Third party programs should typically not access this file directly, but only
# through the symlink at /etc/resolv.conf. To manage man:resolv.conf(5) in a
# different way, replace this symlink by a static file or a different symlink.
#
# See man:systemd-resolved.service(8) for details about the supported modes of
# operation for /etc/resolv.conf.

nameserver 192.168.1.1
search .

2. Configure resolved.conf

Next, /etc/systemd/resolved.conf needs to be configured with new DNS servers of your choosing. You also want to allow DNS-over-TLS. So edit /etc/systemd/resolved.conf to include the following:

DNS=9.9.9.9 149.112.112.112 1.1.1.1 1.0.0.1
DNSOverTLS=yes

After a sudo systemctl restart systemd-resolved, take another look at /etc/resolv.conf, which you now know points to /run/systemd/resolv/resolv.conf (comments removed):

    $ cat /etc/resolv.conf
    
    nameserver 9.9.9.9
    nameserver 149.112.112.112
    nameserver 1.1.1.1
    # Too many DNS servers configured, the following entries may be ignored.
    nameserver 1.0.0.1
    nameserver 192.168.1.1

    search .

You'll notice that it now includes the DNS servers defined in both /etc/systemd/resolved.conf as well as what's pushed to you via DHCP.

Take note, according the the manpage of resolved.conf:

DNS requests are sent to one of the listed DNS servers
in parallel to suitable per-link DNS servers acquired
from systemd-networkd.service(8) or set at runtime by
external applications.

Therefore, you now need to disable the DNS nameservers for enp0s10. That's next with Netplan.

3. Configure Netplan

According to the manpage for Netplan, and since you're using Ubuntu Server which uses systemd-networkd and not NetworkManager, you can override the default DHCP behavior:

dhcp4-overrides (mapping)
              (networkd backend only) Overrides default DHCP behavior;  see  the  DHCP  Overrides
              section below.

Taking a look at the DHCP Overrides section, you don't have to use the DNS servers pushed to you by setting use-dns: false:

       ``use-dns`` (bool)
       :    Default: ``true``. When ``true``, the DNS servers received from the
            DHCP server will be used and take precedence over any statically
            configured ones. Currently only has an effect on the ``networkd``
            backend.

Therefore, edit your yaml config file, /etc/netplan/00-installer-config.yaml, and add the following:

# This is the network config written by 'subiquity'
network:
  ethernets:
    enp0s10:
      dhcp4: true
      dhcp4-overrides:
        use-dns: false
  version: 2

Restart Netplan: sudo netplan try.

Restart systemd-resolved: sudo systemctl restart systemd-resolved

Then look at /etc/resolv.conf (comments removed). Your local uplink nameserver is no longer present.

    $ cat /etc/resolv.conf

    nameserver 9.9.9.9
    nameserver 149.112.112.112
    nameserver 1.1.1.1
    # Too many DNS servers configured, the following entries may be ignored.
    nameserver 1.0.0.1

    search .

If you run resolvectl status, you should see your DNS servers under Global, but none under Link 2 (enp0s10):

Global
       Protocols: -LLMNR -mDNS +DNSOverTLS DNSSEC=no/unsupported
resolv.conf mode: uplink
     DNS Servers: 9.9.9.9 149.112.112.112 1.1.1.1 1.0.0.1

Link 2 (enp0s10)
Current Scopes: DNS
     Protocols: +DefaultRoute +LLMNR -mDNS +DNSOverTLS DNSSEC=no/unsupported
    DNS Domain: localdomain

Link 3 (tun0)
Current Scopes: none
     Protocols: -DefaultRoute +LLMNR -mDNS +DNSOverTLS DNSSEC=no/unsupported

Link 4 (wlp5s0)
Current Scopes: none
     Protocols: -DefaultRoute +LLMNR -mDNS +DNSOverTLS DNSSEC=no/unsupported

Take note: The resolv.conf mode is uplink, because /etc/resolve.conf is a symlink to /run/systemd/resolv/resolv.conf. This mode definition is automatically changed depending on what file /etc/resolv.conf points to or if its a standard file.

Finally, if you do a DNS query with dig +tls google.com, you should see one of your defined nameservers being used and not the stub resolver or your local uplink server. And if it all goes well, it should be using DNS-over-TLS through your VPN.

UbuntuUser avatar
fk flag
This answer looks like it should be correct but the problem is this: When I change /etc/netplan/00-installer-config.yaml as shown it fails, first because it has to be dhcp4-overrides not dhcp-overrides, but the big problem is once I complete that step DNS doesn't work at all. Even after I change everything back to the way it was DNS will not work until I reboot. Restarting Netplan and systemd-resolved won't even do it, it has to be a full reboot. Everything else prior to that step appears to work, but not that for some reason.
mpboden avatar
do flag
@UbuntuUser That was a typo on my part regarding `dhcp4-overrides`. My answer has been updated. Now then, you're saying that after all this, DNS doesn't work at all? It works on my end. The only difference is that I don't have a VPN setup. Are you willing to try it first without starting the VPN? That'll at least rule that out.
UbuntuUser avatar
fk flag
Yes, I am saying that when I add those lines to /etc/netplan/00-installer-config.yaml it doesn't work at all. Unfortunately I didn't set up the VPN tunnel so I can't turn it off but it would not surprise me if your instructions work when there is no VPN in the mix. Everything you wrote makes perfect sense and I appreciate the effort that you put into that answer and hope it helps others who may have a similar issue, just without the VPN. I did find a way that appears to work and posted that as another answer but I am not 100% confident in it, but it does seem to work, any thoughts?
mpboden avatar
do flag
@UbuntuUser I've been doing a lot of research about your situation and added another answer. I also wanted to let you know I found some info [here](https://github.com/systemd/systemd/pull/11050#issuecomment-485891527) where someone said that they couldn't get `UseDNS=false` to work in conjuction with a symlink from `/etc/resolv.conf` to `/run/resolvconf/resolv.conf`. This was from 2019, and I haven't seen anything as to whether it was ever resolved or not. Perhaps this is why it wasn't working for you.
Score:0
mk flag
rfm

This is a suggestion, not an answer, because I don't have a setup where I could test it, but I think it's going to be too long to fit in a comment.

  1. Back out the replacement of resolv.conf with a static file. This will route all DNS queries through the stub resolver provided by systemd-resolved.
  2. Configure systemd-resolved to use DNS-over-TLS. [https://medium.com/@jawadalkassim/enable-dns-over-tls-in-linux-using-systemd-b03e44448c1c] has some instructions that look OK to me, and I see you started out this way to begin with.
  3. Stop dhcp from adding the DNS server that your router is supplying when it configures the connections. Do this in the netplan file and run netplan try to enable it.
network:
  version: 2
  ethernets:
    enp0s10:
      dhcp4: true
      dhcp4-overrides:
        use-dns: false

I think after all that, DNS requests on the local machine will be sent to the local stub resolver in systemd-resolved. systemd-resolved will forward to the configured DNS servers using DNS over TLS. Assuming the VPN setup changed the default route to be on tun0, it shouldn't go over enp0s10.

UbuntuUser avatar
fk flag
As I noted in another comment, when I change /etc/netplan/00-installer-config.yaml as shown DNS doesn't work at all. Even after I change everything back to the way it was DNS will not work until I reboot. Restarting systemd-resolved won't even do it, it has to be a full reboot.
mangohost

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.