The AI Crawler Policy Tracker

Block rate by crawler

The share of reachable sites that disallow each crawler at the site root. Training-oriented crawlers are blocked far more often than the search-and-answer bots that can send traffic back.

Where blocking concentrates

Share of sites in each category blocking at least one AI crawler. News and reference publishers — the sites with the most to lose from uncompensated training — lead; commerce and community lag.

Every site we track

The full list, with the specific AI crawlers each site disallows. Domain links open the live robots.txt so you can verify any row yourself.

Methodology

• Every week, an automated job fetches /robots.txt for each tracked domain and evaluates it against each major AI crawler using RFC 9309 rules — the most specific user-agent group wins, and within it the longest matching path decides, with Allow beating Disallow on a tie.
• A crawler counts as “blocked” when the site disallows it at the root (/). Sites with no robots.txt, or that were unreachable at scan time, are treated as allowing all and excluded from percentages where noted.
• robots.txt governs crawling, not indexing, and it is a good-faith standard — a block is a strong signal of intent, not an enforced barrier. Sites can also block sections rather than the whole root, which a root check won’t capture.
• The domain set is a fixed, curated sample of well-known public sites, not a statistical sample of the whole web. It is meant to show direction and contrast between categories, not a precise population figure.