Manage multi robots.txt for multi subdomains

Wednesday 30 June 2010 2:21:35 am

Hi,

I try to set a robots.txt per subdomain. But I'm unable to find how to do this. I know there's the robots.txt in the root directory, and I use it for the moment with the main domain (www).

I'm doing a mobile version and I need to set a robots.txt for this version. Does anyone know the way (.htaccess rule ?).

Cheers,

Fabien.

Wednesday 30 June 2010 5:43:29 am

Hi Jérôme,

Thanks for your help. This is certainly the best way to customize robots.txt

Good job.

Cheers,

Fabien.

Monday 16 August 2010 3:22:28 pm

Hi Luca

As I said, you'll need to create a dedicated content class :

• Create a new "Robots.txt" content class, with at least one field dedicated to store the content of your robots.txt file
• Instantiate your new content class in your content tree, using translations to alter the content of your robots.txt (yes, you will have to override the pagelayout for this content as it must be a blank layout, not HTML)
• Add an URL wildcard rule to map robots.txt to your content node

To add a URL wildcard, go to Setup / URL Wildcards :

• New URL wildcard : robots.txt
• Destination : <url_of_your_content_node>
• Leave Redirecting URL unchecked

Finally check that the following line in your Apache VHost (or .htaccess file) is commented :

RewriteRule ^/robots\.txt - [L]

Should be :

#RewriteRule ^/robots\.txt - [L]

This would do the job :)