Only allow local addresses using nginx reverse proxy when accessed from external web URL address

I have set up a vaultwarden server locally, nginx and have linked it to the website xxx.ddnsfree.com. I can access it fine using xxx.ddnsfree.com.

I am trying to make it so that I can only access the server at xxx.ddnsfree.com from a local ip address, but from the link and not the ip address of the server. My server sits at 192.168.1.66.

This is what my /etc/nginx/sites-dietpi/vaultwarden.conf file looks like.

I added allow 192.168.1.0/24 and deny all.

    location / {
      proxy_http_version 1.1;
      proxy_set_header "Connection" "";

      proxy_set_header Host $host;
      proxy_set_header X-Real-IP $remote_addr;
      proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
      proxy_set_header X-Forwarded-Proto $scheme;

      proxy_pass https://127.0.0.1:8001;
      allow 192.168.1.0/24;
      deny all;
    }

    location /notifications/hub/negotiate {
      proxy_http_version 1.1;
      proxy_set_header "Connection" "";

      proxy_set_header Host $host;
      proxy_set_header X-Real-IP $remote_addr;
      proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
      proxy_set_header X-Forwarded-Proto $scheme;

      proxy_pass https://127.0.0.1:8001;
    }

    location /notifications/hub {
      proxy_http_version 1.1;
      proxy_set_header Upgrade $http_upgrade;
      proxy_set_header Connection "upgrade";

      proxy_set_header Host $host;
      proxy_set_header X-Real-IP $remote_addr;
      proxy_set_header Forwarded $remote_addr;
      proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
      proxy_set_header X-Forwarded-Proto $scheme;

      proxy_pass https://127.0.0.1:3012;
    }

All my devices are 192.168.1.xx.

It works fine if I remove those lines, but when I add them, I get a 403 Forbidden error.

However, it works if I go to 192.168.1.66 (local ip of my server) from the web browser.

How can I fix this? What am I doing wrong? I need to be able to access it from the url address.