Skip to main content
Free plan, no cardStart free
Free tool · updated 2026-11-04

robots.txt tester

Paste a robots.txt file, enter a URL and the crawler you care about, and see whether the file allows the fetch, which rule decided it and which group answered. The rules are RFC 9309, the standard Google's and Bing's parsers follow. Everything runs in your browser; a browser cannot fetch another site's robots.txt for you, so paste it.

Parsed: 2 groups (* | gptbot, ccbot), 1 sitemap line.

The rules this follows

  • A crawler obeys the group whose user-agent token is the most specific match for its own (case-insensitive, by product token); with no match it obeys the * group; with no * group either, everything is allowed.
  • Within the group, the rule with the longest path that matches wins. On a tie, Allow wins.
  • * in a path matches any run of characters and $ anchors the end of the URL; every other character is literal. Paths are compared percent-decoded, except %2F.
  • An empty Disallow: allows everything. Crawl-delay is not part of the standard; Google ignores it. Sitemap: lines are independent of any group.
  • A line the parser does not understand is ignored, never fatal, which is why a typo in a field name silently allows what it meant to block.

What robots.txt cannot do

  • It is not access control. A disallowed URL can still be indexed from links elsewhere, shown as a bare URL. To keep a page out of results, let it be fetched and send noindex (an X-Robots-Tag header or a meta tag).
  • It only asks. Search and answer-engine crawlers published by their operators honour it; a scraper need not.
  • AI crawlers are separate tokens with separate purposes: a search index (OAI-SearchBot, Claude-SearchBot, PerplexityBot) and a training crawler (GPTBot, ClaudeBot, CCBot) are decided separately. The decision on record for this site is in its website analysis guide and the file itself at /robots.txt.
  • A CDN can serve a different robots.txt from the one on your origin. Check the file a crawler actually receives, not the one in your repository.

Try it against your own app.

Record one flow and get four kinds of evidence back. Judge the results yourself.

  • Free plan, no card
  • 14-day Solo Pro trial
  • Demo replies within one business day
  • Data hosted in India