Join Log in
Score:1
avatar Server

NGINX - Send request to backend if the request is coming from a crawler/bot

us flag
dvdmn

I have a Single Page Application without server-side rendering. So whenever someone shares a URL from the site on Facebook or Twitter it does not show the dynamic title of the page.

I would like to send crawler/bot requests to a special backend that serves a page with OpenGraph meta tags.

Front-end is hosted on NGINX and I have this configuration so far:

server {
    listen       80;
    listen  [::]:80;

    location / {
    
        if ($http_user_agent ~* "linkedinbot|googlebot|yahoo|bingbot|baiduspider|yandex|yeti|yodaobot|gigabot|ia_archiver|facebookexternalhit|twitterbot|developers\.google\.com") {
            return 307 https://renderer_service_domain$request_uri;
    }

    root   /usr/share/nginx/html;
    index  index.html;
}

This works, but Angular's URL has the # before the route url, I need to remove it before sending the request to the special backend.

So, how can I remove the hash from the $request_uri ?

I tried couple rewrite examples, but it sends the request to the same domain, I need to send the request to another server/domain name.

122
0 + 0
anx avatar
fr flag
anx
Search engines explicitly discourage setting up special treatment to trick their bots - doing so is essentially lying about how your page really looks/works, even if its just pretending to be more accessible.
0
Reply
Michael Hampton avatar
cz flag
Michael Hampton
They also explicitly discourage abusing the hash mark as a routing indicator, as it becomes too difficult or impossible for them to index your content, and also too difficult or impossible for you to solve this problem. It was in vogue briefly about half a decade ago but fell out of favor very quickly. You should rather fix your application.
1
Reply
Score:2
avatar Server
us flag
Tero Kilkanen

The hash # and contents after it are not sent to the HTTP server, so they are not a part of $request_uri variable.

With your example configuration, what is the exact request your backend receives?

I don't have experience with single page apps and Google, but I don't think it is a good approach to send different content to crawler bots than to end users.

0 + 0
us flag
dvdmn
I ended up removing the # from the frontend URL. the rest worked just fine.
0
Reply
ธงชาติไทย
Elon Musk
I sit in a Tesla and translated this thread with Ai:

mangohost

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.