Every site owner now faces a quiet decision: should AI companies be allowed to crawl their content to train and answer with it? Most never make that choice deliberately — it is buried in a robots.txt file few people read. To see where the web actually stands, I check the robots.txt of a fixed list of leading sites every week and record which AI crawlers each one blocks. This is a companion piece to the AI Crawler Policy Tracker, where the live figures update on their own.
Just over half of top sites now block at least one AI crawler
At the most recent scan, 53% of the reachable sites blocked at least one major AI crawler in robots.txt. That is a remarkable figure for a standard that is voluntary and only a couple of years into wide adoption — it means blocking AI crawlers has already become a mainstream default among sites that take their content seriously.
The most-blocked crawler was Common Crawl’s CCBot (which feeds many downstream models), followed closely by Anthropic’s ClaudeBot and OpenAI’s GPTBot. Training-oriented crawlers are blocked far more often than the search-and-answer bots that can send a citation or a click back.
News publishers block aggressively; marketers barely at all
The split by category is the real story. Around 90% of news and media sites block an AI crawler — unsurprising, given the lawsuits and licensing deals in that space. Social and community sites are close behind. At the other end, marketing and SEO sites block the least, often under 15%.
That contrast is rational. A newsroom’s archive is the product, and letting a model train on it for free is giving away the asset. A marketing site, by contrast, wants to be surfaced and cited — being left out of AI answers is the bigger risk than being trained on. The same file, opposite incentives.
Blocking is a strategy choice, not a default to copy
There is no universally correct setting. Blocking training crawlers keeps your content out of models but also removes you from consideration where those systems cite sources. Allowing them keeps you eligible to be surfaced but gives up control over training use. A few things worth separating before you decide:
- Training vs. answering. You can block training-only crawlers while allowing the ones that browse live and cite you, if visibility matters more than exclusivity.
- Google-Extended is not Googlebot. Blocking Google-Extended opts you out of Gemini and AI training without affecting normal Google Search crawling or ranking.
- robots.txt is good faith, not a wall. It signals intent to compliant crawlers; it does not enforce anything technically.
Check your own site
Most sites are blocking or allowing AI crawlers by accident rather than intent. You can see exactly which AI crawlers your site allows or blocks with the free AI Crawler Access Checker, and follow how the wider web is moving on the AI Crawler Policy Tracker. The point is simply to make the decision on purpose.