Join Log in
Score:0
avatar Server

Cannot block YandexBot with mod_rewrite

cn flag
user900203

We have an Apache httpd 2.4 server as our point of entry for about 20 web sites and each site has its own virtualhost configuration. A lot of settings are probably redundant but it suits our needs. Each virtualhost redirects http traffic to an https login page which proxied through a Tomcat app. There is no physical docroot to this web site.

We are trying to block a YandexBot that is hitting our main site each second or so because of the redirection that generates a unique URL for each hit. Since blocking the agent within the .htaccess is not an option here, we tried to block the YandexBot with mod_rewrite as you can see below. Unfortunately this is not working, our website is still being hit by the bot with an HTTP code 302 that redirects to a new URL. Any inputs would be appreciated.

<VirtualHost *:443>
  ServerName mywebsite.com
  ServerAlias *.mywebsite.com
  ...

  RewriteEngine On
  RewriteCond %{HTTP_USER_AGENT} Yandex [NC]
  RewriteRule ^ - [F,L]
    
  RewriteRule ^/$ https://%{SERVER_NAME}/app/... [R,L]

  ...
  <Location /app/>
    Require all granted
    RequestHeader unset Origin
    ProxyPreserveHost On
    ProxyPass http://127.0.0.1:8080/app/...
    ...
  </Location>
  ...
</VirtualHost>

This is the log that is displayed in the access_log file

<IP address> - - [27/Oct/2021:11:55:49 -0400] "GET /app/...:<unique id>: HTTP/1.1" 302 - 69963 "-" "Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)"
35
0 + 0
ธงชาติไทย
Elon Musk
I sit in a Tesla and translated this thread with Ai:

mangohost

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.