Robots.txt Generator
About this Robots.txt Generator tool
A robots.txt file tells search engine crawlers which parts of your site they may visit. It sits at the root of your domain, for example https://example.com/robots.txt. It is also the usual place to point crawlers to your sitemap. Note that robots.txt is a request to well-behaved bots, not a security measure.
How to use it
- Choose which crawlers to allow or block and which paths to disallow.
- Add your sitemap URL.
- Copy the file and save it as robots.txt in your site's root.
Example
User-agent: *
Allow: /
Sitemap: https://example.com/sitemap.xmlFrequently asked questions
Where does robots.txt go?
In the root of your domain, so it loads at /robots.txt.
Does Disallow remove a page from Google?
Not reliably. A blocked page can still be indexed if other sites link to it. Use a noindex tag to keep a page out of search.
Can I block AI crawlers?
You can add rules for specific user agents. Check each company's documentation for the exact names, since they change.
Related tools
Slug Generator · Meta Title Generator · Keyword Density Checker