Robots

We can use the txt type renderer to output the robots.txt file. Note that this is only useful for instances that implement a custom domain, as the robots file needs to be at the root of the page.

A basic robots file would have the following content and would be called using the .txt document type suffix.

{my-custom-domain}/robots.txt

ser-agent: Googlebot
Disallow: /nogooglebot/

User-agent: *
Allow: /

Sitemap: http://{my-instance}/public/104146/en/sitemap.xml