nginx default_server breaks mTLS clients when 404 is encountered before other statuses

There's an nginx reverse proxy with several server blocks like this one:

upstream nodes_app3-ui {
        server 192.168.1.3:40000;
        server 192.168.1.4:40000;
        server 192.168.1.5:40000;
}

server {
        listen 443 ssl;
        server_name app3.myserver.net;

        ssl_certificate /etc/letsencrypt/live/myserver.net/fullchain.pem;
        ssl_certificate_key /etc/letsencrypt/live/myserver.net/privkey.pem;

        ssl_client_certificate /root/my-ca.pem;
        ssl_verify_client      on;

        location / {
                proxy_pass http://nodes_app3-ui;
        }
}

(The proxy serves my app with LetsEncrypt certification while enforcing clients to present certification from a private CA.) This works fine.

Now *.myserver.net also exists as wildcard record in DNS and therefore I need to serve a 404 response to requests that don't match any explicit server definition, as otherwise nginx would just happily present the (I guess) first server it knows to any unknown request.

I tried adding this:

server {
        listen 80 default_server;

        location / {
                return 404;
        }
}

server {
        listen 443 ssl default_server;

        ssl_certificate /etc/letsencrypt/live/myserver.net/fullchain.pem;
        ssl_certificate_key /etc/letsencrypt/live/myserver.net/privkey.pem;

        location / {
                return 404;
        }
}

This also works fine, as long as a client (browser) has visited one of the existing servers (and presented a client certificate) before hitting the 404 default_server. If I start a fresh browser (tried Firefox, Chrome) and call https://nonexistentfoobar.myserver.net first (receiving 404), then any subsequent calls to existing servers will yield 400 Bad Request - No required SSL certificate was sent.

Why not? I thought this might just be a firefox bug, but Chrome behaves just the same. When closing and restarting the browser, and not visiting a 404 URL first, everything stays fine.

Is there any obvious explanation for the behaviour or is some quirk in my nginx conf the culprit?

Thanks!

// Update 1: Clearing browser certificate caches solves the issue. So it seems that both Chrome and Firefox get permanently confused when they see client certificate request from a domain after seeing anything (even an error) without such a request.

// Update 2: There's no problem when I also verify client certifcates in the 404-server. But that's not really what I want.