Control crawling
Use user-agent groups with Allow and Disallow directives to express crawl preferences.
Robots.txt Generator
Create a custom robots.txt file with multiple user agents, allow/disallow rules, crawl-delay directives, and sitemap URLs.
Configure your robots.txt
Choose crawler rules and add your sitemap locations. Review the output before publishing it at your domain root.
Used to build the Sitemap directive. It does not publish the file to your website.
Not supported by every crawler, including Googlebot.
Enter one absolute sitemap URL per line.
robots.txt preview
Download this file and upload it to the root of your website, usually https://yourdomain.com/robots.txt. This tool does not deploy changes automatically.
Keep private content secure
Robots.txt is not access control. Disallowed URLs may still be indexed if discovered elsewhere.
Include your sitemap
List absolute sitemap URLs so compatible crawlers can discover your XML sitemap.