robots.txt tester
Paste a robots.txt file, enter a URL and the crawler you care about, and see whether the file allows the fetch, which rule decided it and which group answered. The rules are RFC 9309, the standard Google's and Bing's parsers follow. Everything runs in your browser; a browser cannot fetch another site's robots.txt for you, so paste it.
Parsed: 2 groups (* | gptbot, ccbot), 1 sitemap line.
The rules this follows
- A crawler obeys the group whose user-agent token is the most specific match for its own (case-insensitive, by product token); with no match it obeys the
*group; with no*group either, everything is allowed. - Within the group, the rule with the longest path that matches wins. On a tie, Allow wins.
*in a path matches any run of characters and$anchors the end of the URL; every other character is literal. Paths are compared percent-decoded, except%2F.- An empty
Disallow:allows everything.Crawl-delayis not part of the standard; Google ignores it.Sitemap:lines are independent of any group. - A line the parser does not understand is ignored, never fatal, which is why a typo in a field name silently allows what it meant to block.
What robots.txt cannot do
- It is not access control. A disallowed URL can still be indexed from links elsewhere, shown as a bare URL. To keep a page out of results, let it be fetched and send
noindex(anX-Robots-Tagheader or a meta tag). - It only asks. Search and answer-engine crawlers published by their operators honour it; a scraper need not.
- AI crawlers are separate tokens with separate purposes: a search index (
OAI-SearchBot,Claude-SearchBot,PerplexityBot) and a training crawler (GPTBot,ClaudeBot,CCBot) are decided separately. The decision on record for this site is in its website analysis guide and the file itself at/robots.txt. - A CDN can serve a different robots.txt from the one on your origin. Check the file a crawler actually receives, not the one in your repository.