AI crawler access checker
Enter a domain and I'll read its robots.txt and show which AI crawlers it allows or blocks — GPTBot, ClaudeBot, Google-Extended, PerplexityBot, CCBot, and others — in plain English.
How it works
The checker fetches the domain's robots.txt and applies the same rule-matching a compliant crawler uses (RFC 9309): the most specific user-agent group wins, and within it the longest matching path rule decides, with Allow beating Disallow on a tie.
It runs that check for each major AI crawler against the site root, and reports allowed or blocked for each, along with what that bot is used for — training, live browsing, or search indexing. If there is no robots.txt, everything is allowed by default.
This makes visible a decision most sites make by accident: many block AI crawlers without realizing it, or intend to and haven't. Either way, seeing it plainly is the first step to controlling it.
Assumptions and limitations
- robots.txt governs crawling, not indexing, and it is a good-faith standard — not every bot obeys it. A block is a strong signal, not an enforced barrier.
- It checks access to the site root. A site can allow the root while blocking specific sections, which a single check won't reveal.
- AI companies add and rename crawlers over time, so the list reflects the major known agents at time of writing, not every bot in existence.
Frequently asked questions
What is GPTBot and should I block it?
GPTBot is OpenAI's crawler that gathers content to train and improve its models. Whether to block it is a strategy choice: blocking keeps your content out of that training, while allowing it keeps you eligible to be surfaced and cited. There is no universally correct answer.
How do I block AI crawlers in robots.txt?
Add a group per bot: a line 'User-agent: GPTBot' followed by 'Disallow: /' blocks it from the whole site. To allow one, omit it or give it an empty Disallow. You can allow search-oriented bots while blocking training-only ones.
What's the difference between GPTBot and Google-Extended?
GPTBot is OpenAI's crawler; Google-Extended is a control token that lets you opt out of Google's Gemini and AI training without affecting normal Google Search crawling. They govern different AI systems, so you may want different rules for each.
Does blocking AI crawlers affect my Google Search ranking?
Blocking Google-Extended does not affect normal Google Search crawling or ranking — it only controls AI/Gemini use. Blocking Googlebot itself would affect search. This tool distinguishes the AI-specific crawlers from search crawlers.
Is the domain I check stored?
The domain is used to read its public robots.txt for your request. If you request the emailed cheatsheet, your email is stored so it can be sent; the domain is noted only as context.
More free tools like this, by email
I publish new calculators and explainers regularly. Get them when they land — no spam, unsubscribe anytime.