Robots.txt tester
This robots.txt tester tells you whether a search engine crawler may fetch a given page according to your robots.txt file. SEO specialists and site owners use it to check new rules before publishing them.
- Runs in your browser
- Free, no account needed
- Copy or download the result
Result
Example
User-agent: * Disallow: /admin/ Allow: /admin/help User-agent: Googlebot Disallow: /private/ Allow: /private/press-kit$ Sitemap: https://www.example.com/sitemap.xml
Result: Blocked User agent: Googlebot Path tested: /private/report.html Group used: User-agent: Googlebot (line 5) Matching rule: Disallow: /private/ (line 6) Sitemaps found: https://www.example.com/sitemap.xml Rules: the longest matching path wins, Allow wins a tie, * matches any characters and $ anchors the end.
How to use the robots.txt tester
- Paste your file into the robots.txt contents box.
- Enter a User agent token such as Googlebot and a URL path to test such as /private/page.html.
- Press Run to see Allowed or Blocked and the matching rule.
- Use Copy result or Download to save the result.
How it works
The crawler uses the group whose User-agent matches its full token, ignoring case. With no matching group and no * group, everything is allowed.
If no group names the token exactly and the token has a hyphen, such as Googlebot-News, the group for the name before the hyphen (Googlebot) is used, which matches how Google handles its crawlers. Otherwise the * group applies.
Among matching Allow and Disallow rules the longest path wins, and Allow wins a tie. * matches any characters and $ anchors the end of the path.
Lines such as Crawl-delay are listed as not used, and /robots.txt itself is always allowed.
Nothing is fetched. The tool reads only the text you paste.
Everything runs in your browser. Your input is not uploaded or stored.
Questions and answers
Does Disallow keep a page out of Google?
Not always. Disallow stops crawling, but a blocked URL can still be listed if other pages link to it. To keep a page out of results, let it be crawled and add a noindex tag.
What happens when Allow and Disallow both match?
The rule with the longer path wins. If both are the same length, Allow wins, as RFC 9309 says.
Is robots.txt case sensitive?
User-agent names are matched without regard to case, but URL paths are case sensitive, so /Private and /private are different.