Is your robots.txt locking out AI search?
Plenty of sites block AI crawlers by accident: an old plugin setting, a copied template, a well-meaning developer. Paste your robots.txt and see exactly who gets in.
An AI crawler checker tests a robots.txt file against the user agents AI companies publish, such as GPTBot and OAI-SearchBot (OpenAI), ClaudeBot (Anthropic), PerplexityBot, Google-Extended and Applebot-Extended, to show which ones are allowed or blocked for a given path and which rule decided it.
How the checker decides
It follows the Robots Exclusion Protocol, standardized as RFC 9309. For each crawler it finds the most specific matching user-agent group, falls back to the * group, then applies the longest matching Allow or Disallow rule for your path, with Allow winning ties. Wildcards (*) and end anchors ($) are supported.
Training bots vs. search bots
Not all AI crawlers do the same job. Some collect data to train models. Others fetch pages to answer a live question and cite the source. OpenAI, for example, documents GPTBot for training and OAI-SearchBot for search separately. You can block one and allow the other. Google-Extended and Applebot-Extended are control tokens: they do not crawl on their own, they tell Google and Apple whether content may be used for AI features and training.
Blocking training while allowing search is a reasonable middle path for many businesses. Blocking everything means answer engines cannot quote you.
robots.txt is a request, not a lock
Reputable crawlers honor it. It does not stop anyone determined to ignore it, and it does not hide pages from search results if other sites link to them. For private content, use authentication.
Where this connects
Questions, answered straight
Which AI crawlers does this check?
Googlebot, Google-Extended, Bingbot, GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, Applebot, Applebot-Extended, CCBot, Meta-ExternalAgent, Amazonbot, Bytespider and DuckAssistBot.
Should I block GPTBot?
Blocking GPTBot stops OpenAI from using your pages for model training. It does not block OAI-SearchBot, which powers ChatGPT search results. Many businesses allow search crawlers even if they block training crawlers.
Does blocking Google-Extended affect Google Search rankings?
Google says Google-Extended does not affect inclusion or ranking in Google Search. It controls use of content for Gemini and related AI features and training.
Why does my result say allowed by default?
If no group in robots.txt matches a crawler and there is no wildcard group, the crawler is allowed. The same applies when no rule matches the tested path.
Does this tool fetch my robots.txt?
No. You paste the file and everything runs in your browser. Nothing is sent to a server.
Not everyone should hire us. Let's find out if you should.
A quick, honest conversation. If it's not a fit, we'll point you somewhere better.