Nginx reverse proxy gives 404 error, but only for a specific domain name

Sorry to repeat a question that has been asked a dozen times before, but I have read each and every answer to those questions without being able to solve my problem.

I am trying to setup an nginx docker container as a reverse proxy to be able to access resources on a local network through an easy to remember domain name, as well as provide SSL encryption.

The domain name is amars.no, a domain that I own, but where the official DNS entry (ie. what Google/Cloudflare DNS would report) does not match what I will use on the local network to reach the reverse proxy. Instead I have setup a dnsmasq server with a custom entry for multiple subdomains of this domain name, which seems to be working as expected (dig hub.amars.no returns the IP of the nginx server).

I have then added sites to the nginx config, one for each local service that should have its own url, with a subdomain of amars.no for each service. The problem is that attempting to access any of these domains in a browser returns a 404 not found error. The extremely weird thing is that I can change amars.no to literaly anything else, even something like google.com, and it will work the way I want it to as long as the entry in the dnsmasq hosts file and the nginx config matches up. It is just when I use the domain I own, amars.no, that a 404 is returned. I have to use amars.no to be able to get a valid SSL certificate from let's encrypt.

I have tried to simplify the setup for troubleshooting, and for now left HTTPS out of the setup to avoid any problems caused by that.

Here's a summary of the network setup:

• 192.168.0.22: dnsmasq server (port 53) and Jenkins web interface (port 8081)
• 192.168.0.23: nginx reverse proxy server (in docker) (port 80 and 443), also serving a static html file with links to the different services

For now I am trying to get two different urls working:

• jenkins.amars.no (should point (proxy pass) to 192.168.0.22:8081)
• hub.amars.no (should serve the static html file at the nginx server

This is the contents of the hosts file used by dnsmasq (lone-gatekeeper is the computer name):

127.0.0.1 localhost
127.0.1.1 lone-gatekeeper

192.168.0.23 hub.amars.no
192.168.0.23 jenkins.amars.no

# The following lines are desirable for IPv6 capable hosts
::1     ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters

the nginx container is configured and run by this docker-compose file:

version: '3'

services:
  reverse:
    container_name: reverse
    hostname: reverse
    image: nginx
    ports:
      - 80:80
      - 443:443
    volumes:
      - /opt/nginx/nginx.conf:/etc/nginx/nginx.conf
      - /opt/nginx/conf.d:/etc/nginx/conf.d
      - /opt/ssl:/etc/ssl/private
      - /var/www/html:/usr/share/nginx/html

The nginx config (the result of using docker exec reverse nginx -T):

nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
# configuration file /etc/nginx/nginx.conf:

user  nginx;
worker_processes  auto;

error_log  /var/log/nginx/error.log notice;
pid        /var/run/nginx.pid;


events {
    worker_connections  1024;
}


http {
    include       /etc/nginx/mime.types;
    default_type  application/octet-stream;

    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';

    access_log  /var/log/nginx/access.log  main;

    sendfile        on;
    #tcp_nopush     on;

    keepalive_timeout  65;

    #gzip  on;

    include /etc/nginx/conf.d/sites-enabled/*.conf;
}

# configuration file /etc/nginx/mime.types:

types {
    text/html                                        html htm shtml;
    text/css                                         css;
    text/xml                                         xml;
    image/gif                                        gif;
    image/jpeg                                       jpeg jpg;
    application/javascript                           js;
    application/atom+xml                             atom;
    application/rss+xml                              rss;

    text/mathml                                      mml;
    text/plain                                       txt;
    text/vnd.sun.j2me.app-descriptor                 jad;
    text/vnd.wap.wml                                 wml;
    text/x-component                                 htc;

    image/avif                                       avif;
    image/png                                        png;
    image/svg+xml                                    svg svgz;
    image/tiff                                       tif tiff;
    image/vnd.wap.wbmp                               wbmp;
    image/webp                                       webp;
    image/x-icon                                     ico;
    image/x-jng                                      jng;
    image/x-ms-bmp                                   bmp;

    font/woff                                        woff;
    font/woff2                                       woff2;

    application/java-archive                         jar war ear;
    application/json                                 json;
    application/mac-binhex40                         hqx;
    application/msword                               doc;
    application/pdf                                  pdf;
    application/postscript                           ps eps ai;
    application/rtf                                  rtf;
    application/vnd.apple.mpegurl                    m3u8;
    application/vnd.google-earth.kml+xml             kml;
    application/vnd.google-earth.kmz                 kmz;
    application/vnd.ms-excel                         xls;
    application/vnd.ms-fontobject                    eot;
    application/vnd.ms-powerpoint                    ppt;
    application/vnd.oasis.opendocument.graphics      odg;
    application/vnd.oasis.opendocument.presentation  odp;
    application/vnd.oasis.opendocument.spreadsheet   ods;
    application/vnd.oasis.opendocument.text          odt;
    application/vnd.openxmlformats-officedocument.presentationml.presentation
                                                     pptx;
    application/vnd.openxmlformats-officedocument.spreadsheetml.sheet
                                                     xlsx;
    application/vnd.openxmlformats-officedocument.wordprocessingml.document
                                                     docx;
    application/vnd.wap.wmlc                         wmlc;
    application/wasm                                 wasm;
    application/x-7z-compressed                      7z;
    application/x-cocoa                              cco;
    application/x-java-archive-diff                  jardiff;
    application/x-java-jnlp-file                     jnlp;
    application/x-makeself                           run;
    application/x-perl                               pl pm;
    application/x-pilot                              prc pdb;
    application/x-rar-compressed                     rar;
    application/x-redhat-package-manager             rpm;
    application/x-sea                                sea;
    application/x-shockwave-flash                    swf;
    application/x-stuffit                            sit;
    application/x-tcl                                tcl tk;
    application/x-x509-ca-cert                       der pem crt;
    application/x-xpinstall                          xpi;
    application/xhtml+xml                            xhtml;
    application/xspf+xml                             xspf;
    application/zip                                  zip;

    application/octet-stream                         bin exe dll;
    application/octet-stream                         deb;
    application/octet-stream                         dmg;
    application/octet-stream                         iso img;
    application/octet-stream                         msi msp msm;

    audio/midi                                       mid midi kar;
    audio/mpeg                                       mp3;
    audio/ogg                                        ogg;
    audio/x-m4a                                      m4a;
    audio/x-realaudio                                ra;

    video/3gpp                                       3gpp 3gp;
    video/mp2t                                       ts;
    video/mp4                                        mp4;
    video/mpeg                                       mpeg mpg;
    video/quicktime                                  mov;
    video/webm                                       webm;
    video/x-flv                                      flv;
    video/x-m4v                                      m4v;
    video/x-mng                                      mng;
    video/x-ms-asf                                   asx asf;
    video/x-ms-wmv                                   wmv;
    video/x-msvideo                                  avi;
}

# configuration file /etc/nginx/conf.d/sites-enabled/hub.conf:
server {
  listen      80;
  server_name hub.amars.no;

  location / {
    root /usr/share/nginx/html;
    index index.html index.htm;
  }

}

# configuration file /etc/nginx/conf.d/sites-enabled/jenkins.conf:
upstream jenkins {
  server 192.168.0.22:8081;
}

server {
  listen      80;
  server_name jenkins.amars.no;

  location / {
    proxy_pass http://jenkins;
  }

}

the result of dig jenkins.amars.no (on a computer where 192.168.0.22 is set as the primary DNS server) (hub.amars.no gives same result):

; <<>> DiG 9.10.6 <<>> jenkins.amars.no
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 18497
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
;; QUESTION SECTION:
;jenkins.amars.no.      IN  A

;; ANSWER SECTION:
jenkins.amars.no.   0   IN  A   192.168.0.23

;; Query time: 18 msec
;; SERVER: 192.168.0.22#53(10.0.0.52)
;; WHEN: Mon Aug 07 01:01:42 CEST 2023
;; MSG SIZE  rcvd: 60

Ok, after months of ripping my hair out over this issue (before asking here), I think I finally found the root cause of the problem, and why none of the other solutions worked.

This seems to be an ipv6 issue. The computer I used for accessing the services is configured to prefer/prioritize ipv6 connections whenever available (I guess most modern systems do). When my dnsmasq instance didn't have an ipv6 address for the domain name in question, it queried its upstream DNS server (Google DNS) instead, and got the official ipv6 address on record for this domain. Running dig jenkins.amars.no AAAA confirms this. I haven't created any website/service for this domain, so it naturally returns a 404.

I didn't have any luck adding the ipv6 address of the nginx proxy server to the /etc/hosts file dnsmasq is using for local DNS records, so instead I killed the dnsmasq instance and instead spun up a pihole docker instance where I added both ipv4 and ipv6 addresses to the nginx proxy server. With this in place everything now works and configuring SSL support on top of this works as expected as well.

Also, this would not have caused a problem if I didn't use a domain name that already exists on the internet (and exist in public DNS records), because the local DNS server would not get a response from its upstream DNS. However, I had to use a valid, existing domain name that I own to get a valid SSL certificate.

What I don't fully understand is why changing to another (existing) domain name didn't cause trouble. I tried changing the url to jenkins.vg.no (vg.no is a well-known existing domain that have AAAA records in public DNS servers), which worked flawlessly (obviously only with plain HTTP, no SSL).

tl;dr: in 2023, make sure to support both ipv4 and ipv6 when creating custom DNS records on a local DNS server.