It depends
Applications might do their own thing, independent of what the OS is configured to do.
For example after setting a custom secure DNS provider in the “Settings > Privacy & Security > Security” in the Chrome webbrowser the system resolver is no longer used.
System resolver
When the application does not do it's own thing, the application typically calls on the system resolver by using OS/kernel functions such as the legacy gethostbyname()
and/or the more modern getaddrinfo()
to translate the hostname or fully qualified domain name (FQDN) to an IPv4/IPv6 address.
The Name Service Switch (NSS) configuration file, /etc/nsswitch.conf
, is used to configure the sources from which to obtain name-service information and in what order.
That /etc/nsswitch.conf
is for example where the usual default order of /etc/hosts
(the "files" keyword) getting consulted before DNS is configured with:
# /etc/nsswitch.conf
#
# Example configuration of GNU Name Service Switch functionality.
# If you have the `glibc-doc-reference' and `info' packages installed, try:
# `info libc "Name Service Switch"' for information about this file.
hosts: files dns
When the hostname can't be found in the hosts files, then fallback to the DNS resolver is used.
The DNS resolver is configured with /etc/resolv.conf
.
Classically that file contains a list of name servers (with the first nameserver
getting used as the default nameserver and any additional ones only getting used when the ones before do not respond):
# /etc/resolv.conf file
domain example.com
search int.example.com ad.example.com
nameserver 8.8.8.8
nameserver 1.1.1.1
but in modern Linux distributions you'll typically see that the systemd resolver gets configured there. That has some advanced and interesting features which I won't explain here:
# This is /run/systemd/resolve/stub-resolv.conf managed by man:systemd-resolved(8).
# Do not edit.
#
# This file might be symlinked as /etc/resolv.conf. If you're looking at
# /etc/resolv.conf and seeing this text, you have followed the symlink.
#
# This is a dynamic resolv.conf file for connecting local clients to the
# internal DNS stub resolver of systemd-resolved. This file lists all
# configured search domains.
#
# Run "resolvectl status" to see details about the uplink DNS servers
# currently in use.
#
# Third party programs should typically not access this file directly, but only
# through the symlink at /etc/resolv.conf. To manage man:resolv.conf(5) in a
# different way, replace this symlink by a static file or a different symlink.
#
# See man:systemd-resolved.service(8) for details about the supported modes of
# operation for /etc/resolv.conf.
nameserver 127.0.0.53
VPN
AFAIK usually VPN software has a feature that changes the contents of /etc/resolv.conf
for the duration of the connection to the VPN server, when configured to do so.
See for example: https://community.openvpn.net/openvpn/wiki/Pushing-DNS-to-clients
Nothing fancy needed with iptables, changing resolv.conf immediately adjusts how your whole system does DNS resolution.
DNS protocols
DNS queries always use UDP and port 53, except when they don't.
See How does the DNS protocol switch from UDP to TCP?
I think that using something other than classical DNS queries over UDP requires either using the more advanced systemd resolvd, (which supports for example DNS over TLS,) as a resolver rather than relying the classic C libraries.
Alternatively, similar to how PAM is designed, Name Service Switch (NSS) is designed to add new modules without changing any of the programs and code that call on the system resolver. Add an additional resolver library to support a different DNS protocol and adjust nsswitch.conf
to use that library, see for example https://github.com/dimkr/nss-tls