The Problem

I have a VPC in which I need to access the servers using private FQDNs. The VPC is accessible through a wireguard VPN. The VPN server also serves as a DNS server running BIND9. I have set the DNS server with a private zone according to this tutorial. From inside the VPC, the DNS works as expected and I am able to reach the servers by the FQDNs defined in the DNS zone.

When connecting to the VPC through the VPN tunnel, I am unable to resolve those FQDNs although I have setup my VPN client to use my private DNS server. I know the VPN client uses my private DNS server because when I run nslookup google.com I see my DNS's IP address as you can see the result below:

Server:     10.118.0.2
Address:    10.118.0.2#53
...

If I run the nslookup myprivate.domain.com from a machine connected to the VPC through the VPN tunnel, I receive a NXDOMAIN, the same goes for the ping, it fails with the error Name or service not known. However, if I run ping on the private IP address from the VPC, it works. So if myprivate.domain.com is hosted on on the server at10.118.0.3, the ping succeeds on the IP address but fails on the FQDN.

Additionally, see the dig results when inside the VPC vs when from a machine connected through the VPN: dig dev.myprivatedomain.com ns: FROM VPC:

;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 51703
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
; COOKIE: 9dccc02158dee7f70100000061a7e0a1ce2597e377b9c301 (good)
;; QUESTION SECTION:
;dev.myprivatedomain.com.       IN  NS

;; AUTHORITY SECTION:
nabuinternal.com.   604800  IN  SOA ns1.myprivatedomain.com. ...

;; Query time: 0 msec
;; SERVER: 10.118.0.2#53(10.118.0.2)
;; WHEN: Wed Dec 01 20:52:49 UTC 2021
;; MSG SIZE  rcvd: 93

FROM VPN:

;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 57158
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
;; QUESTION SECTION:
;dev.myprivatedomain.com.       IN  NS

;; AUTHORITY SECTION:
com.            900 IN  SOA a.gtld-servers.net. nstld.verisign-grs.com. 1638392201 1800 900 604800 86400

;; Query time: 44 msec
;; SERVER: 10.118.0.2#53(10.118.0.2)
;; WHEN: Wed Dec 01 15:57:05 EST 2021
;; MSG SIZE  rcvd: 122

I notice that both use the same DNS server, but when requested from the VPN, the authority for my private domain is not returned.

In conclusion, after several hours of research, I'm falling short to find out what is missing for clients connecting to the VPC through the VPN to be able to resolve the FQDNs defined by my private DNS.

Additional Information

• Server is ubuntu 20.04 LTS
• Bind 9: BIND 9.16.1-Ubuntu (Stable Release)
• wireguard: wireguard-tools v1.0.20200513 installed through wirespeed
• UFW enabled

The VPN and DNS server ip in the VPC is 10.118.0.2.

The VPN address pool is 10.99.0.0/16 and I have set the BIND9 configurations in the following manner:

acl "trusted" {
    10.118.0.2;    # the vpn and dns server
    ...
    10.99.0.0/16;  # vpn address pool
    
};
options {
    directory "/var/cache/bind";
    listen-on-v6 { any; };
    recursion yes;
    allow-recursion { trusted; };
    listen-on { 10.118.0.0/20; 10.99.0.0/16; };  
    allow-transfer { none; };
    forwarders {
            8.8.8.8;
            8.8.4.4;
    };
};

The zone is configured this way:

$TTL    604800
@       IN      SOA     ns1.myprivatedomain.com. admin.myprivatedomain.com. (
                              9         ; Serial
                         604800         ; Refresh
                          86400         ; Retry
                        2419200         ; Expire
                         604800 )       ; Negative Cache TTL
;                         
; name servers - NS records
    IN      NS      ns1.myprivatedomain.com.

; name servers - A records
ns1.myprivatedomain.com.          IN      A       10.118.0.2

; 10.118.0.0/20 - A records
dev.myprivatedomain.com.            IN      A      10.118.0.4
staging.myprivatedomain.com.         IN      A      10.118.0.3

and the reverse zone file:

$TTL    604800
@       IN      SOA     ns1.myprivatedomain.com. admin.myprivatedomain.com. (
                              7         ; Serial
                         604800         ; Refresh
                          86400         ; Retry
                        2419200         ; Expire
                         604800 )       ; Negative Cache TTL
;
; name servers - NS records
    IN      NS      ns1.myprivatedomain.com.

; PTR Records
2.0   IN      PTR     ns1.myprivatedomain.com.         ; 10.118.0.2
4.0   IN      PTR     dev.myprivatedomain.com.         ; 10.118.0.4
3.0   IN      PTR     staging.myprivatedomain.com.      ; 10.118.0.3

UFW is set to allow port 53 for both TCP and UDP.

Also, UFW has before rules to allow traffic from the VPN:

*nat
:POSTROUTING ACCEPT [0:0]
-A POSTROUTING -s 10.99.0.0/16 -o eth1 -j MASQUERADE

The previous rule was setup in order to allow a client to connect to the VPN tunnel and use the private DNS server. Without this rule, I am unable to access the internet unless I set the DNS address to a public one like google's one. I have found this rule during my research, however I am not yet very familiar with firewall configurations and I don't fully understand yet the implication of it. It has helped me get closer to my goal, but I need to do further reading on it.

Below is the wireguard VPN client config:

[Interface]
...
DNS = 10.118.0.2
Address = 10.99.0.2/16

[Peer]
...
AllowedIPs = 10.99.0.0/16, 10.118.0.0/20