How to redirect/block IP access to Nginx

I am hosting a Website on some EC2 Instances behind an Elastic Load Balancer. The Website can be reached via the public IP of the Load Balancer. This public IP is changing frequently so I have to redirect/block it.

For redirecting any access from the public DNS of any AWS services I'm using this Virtual Host with a wildcard. This is working perfectly:

server {
    listen 443;
    server_name *.amazonaws.com;
    ssl_certificate /etc/nginx/ssl/bundle.crt;
    ssl_certificate_key /etc/nginx/ssl/cert.key;
    return 301 https://www.domain.de$request_uri;
}

I've tried the same for IP Addresses but I can't get it working.

server {
    listen 80;
    server_name *.*.*.*;
    return      444;
}

server {
        listen 80 default_server;
        server_name _;
        return      444;
    } 

How can I redirect or block any access via an IP Address on my server?

You need to use a default virtual host to block access directed to any non-defined virtual host.

server {
    listen 443 ssl default_server;
    ssl_certificate /path/to/certificate;
    ssl_certificate_key /path/to/key;
    return 444;
}