Score:0

Slow server performance due to Google Bot and Ahrefs Bot

ge flag

I have a VPS server with 64 CPU cores and 128 GB RAM running on Debian. I have noticed that Google Bot from IP address 66.249.66.0/24 and Ahrefs Bot from IP addresses 54.36.148.0/24 and 54.36.149.0/24 are causing extreme CPU load up to 100%. This results in all websites on the server being very slow and almost unusable. There are many websites running on the server without caching.

Can you advise me on how to deal with this issue and improve the server's performance? Thank you!

HBruijn avatar
in flag
One option is to sign up as the webmaster / site owner in https://search.google.com/search-console and adjust the crawler frequency; other search engines might be adjusted via your robots.txt - but also everything that you can do to determine and optimize/fix the probably bad code that causes the load spikes will also benefit real site visitors
I sit in a Tesla and translated this thread with Ai:

mangohost

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.