Score:2

Drupal

robots.txt file per environment

lazarcelic

6/25/23, 2:03 PM

We are using Drupal 8.9 and we have an EC2 stage site and our main site, which use the same code base.

Is it possible to have a different robots.txt file for each site (environment)?

We don't want the stage site to appears on search results.

0 + 0

Kevin

6/25/23, 2:25 PM

Use the RobotsTxt module.

leymannx

6/25/23, 4:40 PM

If it's about disallowing robots on a stage site, I'd simply put server-side basic authentication like a .htpasswd up front. Password can be simple, memorisable if it's just for keeping robots out.

Score:2

Drupal

leymannx

6/25/23, 4:26 PM

There's the RobotsTxt module that in the end after you removed the robots.txt file takes config to provide the contents of a /robots.txt page.

Together with the Configuration Split module you can then have it play different contents per environment.

To exclude Drupal's default robots.txt file from being recreated during scaffolding, adjust the file-mapping in composer.json like that:

"drupal-scaffold": {
    "locations": {
        "web-root": "web/"
    },
    "file-mapping": {
        "[web-root]/robots.txt": false
    }
},

0 + 0

leymannx

6/25/23, 4:38 PM

RobotsTxt also nice for multi-sites.

Score:1

Drupal

Darvanen

7/1/23, 10:42 PM

I patch the root .htaccess file to redirect to a deny-robots.txt file (which will appear as robots.txt in the reponse) based on the domain, but you could easily use the same directives in your apache configuration:

RewriteEngine on

# Deny robots everywhere except the production site
RewriteCond %{HTTP_HOST} !^www\.mysite\.com\.au$
RewriteRule ^robots.txt(\??.*)$ deny-robots.txt$1

Contents of deny-robots.txt:

User-agent: *
Disallow: *

0 + 0

Darvanen

7/1/23, 10:44 PM

The reason I patch the drupal `.htaccess` file instead of putting this in apache config is I use the same apache config between a few different sites.

Elon Musk

I sit in a Tesla and translated this thread with Ai:

EN: robots.txt file per environment

TH: ไฟล์ robots.txt ต่อสภาพแวดล้อม

RO: fișier robots.txt per mediu

RU: файл robots.txt для каждой среды

VI: tệp robots.txt trên mỗi môi trường

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.