Robots.txt Tester and Validator

Your robots.txt

URL or path to test

Crawler (user agent)

Follows the matching rules Google documents: the most specific (longest) matching rule wins, Allow wins a tie, * matches any text and $ marks the end of a URL. Paste the file content here, because the tool does not fetch other sites.

What does a robots.txt tester do?

A robots.txt file tells crawlers which parts of your site they may visit. Small mistakes in it can block pages you want indexed, or leave open pages you meant to hide. This tester reads the rules you paste, picks the group that applies to a crawler such as Googlebot, and tells you whether a given URL is allowed or blocked and which rule made that decision.

How to use it

Paste your robots.txt. You can copy it from yoursite.com/robots.txt.
Enter the full URL or just the path you want to test, and the crawler name, or leave it blank for any bot.
Press Test URL to see the result and the rule behind it.

How the rules are matched

Disallow: /private/     blocks everything whose path starts with /private/
Allow: /private/a.html   a longer, more specific rule that wins for that page
Disallow: /*.pdf$        * is any text, $ means the URL ends there
Disallow:                an empty Disallow blocks nothing

When several rules match, the longest one wins. If an Allow and a Disallow are equally long, Allow wins. A crawler uses the group with the most specific matching name, such as Googlebot, and only falls back to User-agent: * when none matches.

Frequently asked questions

Does Disallow stop a page from appearing in Google?

Not reliably. It stops crawling, but a blocked URL can still be indexed without its content if other pages link to it. Use a noindex tag, and let the page be crawled, to keep it out of results.

Are the rules case-sensitive?

Yes for the paths. /Admin and /admin are different, while the crawler name is not case-sensitive.

Why does a more specific crawler ignore my * rules?

A crawler obeys only the single group that matches it best. If you have a Googlebot group, Googlebot ignores the * group, so repeat any rules it also needs.

Does this tool fetch my live robots.txt?

No. It tests the text you paste, so you can try changes before publishing them. For the live file, open yoursite.com/robots.txt or use Google Search Console.

Is my robots.txt uploaded?

No. Everything runs in your browser.

Results are estimates for information only and are not financial, tax or legal advice.