Certbot failed to authenticate some domains (authenticator: nginx)

Im running nginx as reverse proxy and for some reason im unable to renew letencrypt certificate, i have tried to upgrade nginx and certbot to latest version. It has worked before and im not sure if some server configuration have been changed.

When running certbot renew i get following error from certbot:

certbot renew --nginx --dry-run

Certbot failed to authenticate some domains (authenticator: nginx). The Certificate Authority reported these problems: Domain: myurl Type: unauthorized Detail: : Invalid response from http://myurl/.well-known/acme-challenge/asdasd: 404

Hint: The Certificate Authority failed to verify the temporary nginx configuration changes made by Certbot. Ensure the listed domains point to this nginx server and that it is accessible from the internet.

Example Nginx error log:

2023/08/03 08:22:36 [error] 13286#13286: *32 open() "/usr/share/nginx/html/.well-known/acme-challenge/asdasd" failed (2: No such file or directory), client: ipadress, server: ipadress, request: "GET /.well-known/acme-challenge/asdasd HTTP/1.1", host: "url"

For what i understand nginx is unable to serve the acme-challange or certbot is writing the challange to another location than /usr/share/nginx/html ....

Can someone point me in right direction?

thanks in advance

Update Output from @symcbean commands gives "No such file or directory" ls -l /usr/share/nginx/html/.well-known/acme-challenge/ ls: cannot access '/usr/share/nginx/html/.well-known/acme-challenge/': No such file or directory

How do i know that cerbot is trying to write to /usr/share/... and not another folder?

Example of nginx site config.

upstream beta-api {
    server 10.166.0.4:8384;
}

server {
    server_name betaapi.ancytree.com;
    listen 80;
    # set client body size to 8M #
    client_max_body_size 8M;
        
    location / {
        proxy_pass http://beta-api;
        include proxy_params;
    }

}

You've looked at your logs - which is good start! Since it appears that LetsEncrypt is accessing your site, you can rule out a lot of issues related to DNS and HTTP.

The next thing you should try is to run through the HTTP-01 challenge yourself and see what happens.

Certbot renewals will be run as a cron job - so this needs to be run using the same account:

echo "test" >/usr/share/nginx/html/.well-known/acme-challenge/asdasd
ls -l /usr/share/nginx/html/.well-known/acme-challenge/asdasd
cat /usr/share/nginx/html/.well-known/acme-challenge/asdasd
curl -I http://myurl/.well-known/acme-challenge/asdasd

If that does not provide an answer for you, update your post with the output of these commands.

(likely candidates are file permissions including MAC, missing directories or a missing mapping in your webserver between the filesystem and URL path)