block nginx connection after multiple 404 s on same tcp connection

I'd like to protect two servers against attackers who blindly probe for urls.

my idea is to block requests after a certain amount of 404 status codes.

I could do this with fail2ban.

However one of the servers is behind a reverse proxy that I do not control and that does not communicate the originating IP address (and it will need a very long time before the request of forwarding the originating IP address will be handled), meaning that from nginx's point of few all requests originate from the revere proxy's IP and obviously I don't want to block all incoming traffic if somebody provokes too many 404s.

Is there any way (without fail2ban, with fail2ban or with any other tool) to detect at least 404 errors from the same https connection and to terminate this connection or to let it 404 unconditionally for any subsequent request?

Please tell me if my question is not clear and I'll try to rephrase it, to give more information.

You can setup Nginx to cache responses, including responses with an error such as a 404. However, as far as I know, you cannot count the number of errors (but that shouldn't matter).

The only issue here is that if you create a new page and earlier it was cached as a 404, then it won't work. At least, not immediately (depending on how long your cache takes to timeout).

That will definitely help greatly, though. All the hits that would otherwise go to your backend and generate a 404 will be stopped at the Nginx instance and return the exact same 404 error over and over again.

Another way, is to define a location and run a command (which could then run ipset to add the IP address). So something like:

location /phpmyadmin.php {
  content_by_lua_block {
    os.execute("/usr/bin/block-ip.sh")
  } 
}

I'm not too sure how you could handle many paths, however.

My own experience with fail2ban is that it's rather slow and "backward" (not proactive). If you really want to completely block an IP address, though, the easiest is to have your app. send a message to the front end where you can run iptables to block the IP. For that you need to forward the original IP which Nginx doesn't do by default, but it's easy to add an X-Forwarded-For header that your app. can then send back to the small tool you're using to add IPs to your iptables. Note also you should not directly add it to iptables. Instead you should use a list. For that look at the ipset.

If you can't change your apps for such and want to use fail2ban, you'd have to check the Nginx logs and detect those paths you do not like.

Nginx has a limit_req module, so you can use it as pre-filtering.

See https://stackoverflow.com/questions/37877242/nginx-limit-req-based-on-http-header for similar question.

For instance by counting and limiting the 404 attempts (e. g. in related error location) and/or filter them by fail2ban hereafter if you'd log only limited requests. This way you could decrease maxretry and findtime for the jail in fail2ban and can avoid too high load by lot of messages in log (see fail2ban :: wiki / Best-practice).

For the filter example (basically you don't need the filter you can simply set failregex etc directly in jail), see filter.d/nginx-limit-req.conf. You can also configure nginx to log X-Forwarded-For header instead or additionally to the real IP. And even ban both of them using different maxretry/findtime combined with different limits and bursts in limit_req module.