Score:0

Nginx caching not working for crawlers

br flag

I have been using Nginx caching for years but recently noticed an issue which I suspect has been happening for years.

If I access a page that's never been visited before I of course get a cache miss, refresh the page and now its a cache hit.

However I noticed when crawlers such as Bingbot or Googlebot crawl the site, the pages they crawl never get cached, they always remain cache misses until I manually visit them.

Is there a setting somewhere in Nginx that prevents bot traffic from caching pages?

FYI I am not treating bots any differently to users in my Nginx config.

Here's a sample of my config.

    proxy_socket_keepalive on;
    proxy_redirect off;
    proxy_cache Google-Cache;
    proxy_cache_revalidate on;
    proxy_ignore_headers  Expires Cache-Control;
    proxy_cache_use_stale error timeout updating http_500 http_502 http_503 http_504;   
    proxy_cache_background_update on; 
    proxy_cache_lock on;


   proxy_cache_valid 200 206 304 306 301 302 404 410 300s;
Michael Hampton avatar
cz flag
What is your `proxy_cache_key`?
mangohost

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.