Robots.txt tester
Paste a robots.txt file, enter a URL path and a user-agent, and get an instant verdict on whether that path can be crawled — plus the exact rule that decided it.
✓ Allowed
No rule in the googlebot group matches — allowed by default.
Longest-path-match wins; Allow beats Disallow on equal length. Tests the pasted text, not your live file, and covers crawling only — not indexing.
How it works
The tool parses your pasted robots.txt into user-agent groups, the same way a compliant crawler would under RFC 9309. It then selects the group whose user-agent token most specifically matches the agent you entered, falling back to the wildcard (*) group only when no named group applies. This matters because a common mistake is assuming rules stack: a crawler with its own group ignores the * group entirely.
Within the selected group, every Allow and Disallow rule is compared against your URL path using longest-path-match: the rule matching the most characters of the path wins, regardless of the order rules appear in the file. When an Allow and a Disallow match with equal length, Allow wins. Wildcard (*) and end-of-path ($) patterns are expanded during matching, since these are where hand-written files most often go wrong.
The result tells you whether crawling is allowed or blocked and highlights the specific line that made the decision, so you can trace surprising verdicts back to their source. All parsing happens in your browser — the file you paste is never sent anywhere.
Assumptions and limitations
- This tests the text you paste, not the file your server actually serves. Redirects, CDN rewrites, or a stale deploy can make the live robots.txt differ from what you think is there — always verify the live file too.
- Not every crawler follows RFC 9309 precisely. Some bots interpret wildcards differently, honor nonstandard directives, or ignore robots.txt altogether, so a verdict here describes compliant behavior, not every bot's behavior.
- Robots.txt controls crawling, not indexing. A disallowed URL can still appear in search results if other pages link to it — deindexing requires noindex or removal, which this tool does not evaluate.
- Directives outside the standard Allow/Disallow/user-agent model, such as crawl-delay or sitemap lines, are not part of the allow/block decision and are not evaluated.
Frequently asked questions
Why is Google not crawling a page my robots.txt allows?
Robots.txt is only the first gate. A page that passes it can still be skipped because of noindex tags, canonical tags pointing elsewhere, weak internal linking, or the crawler simply deprioritizing it. If this tool says the path is allowed, look at the page-level signals next rather than rewriting robots rules.
Does Disallow in robots.txt remove a page from search results?
No. Disallow stops compliant crawlers from fetching the page, but the URL can still be indexed from external links. Worse, if you add a noindex tag to a disallowed page, crawlers can never fetch the page to see it. To deindex, allow crawling and use noindex instead.
Which rule wins when Allow and Disallow both match a URL?
Under RFC 9309, the rule with the longest matching path wins, no matter where it sits in the file. If an Allow and a Disallow match with exactly the same length, Allow takes precedence. This tool shows you which rule won and why, which is the fastest way to debug a conflicting file.
Does this robots.txt checker support wildcards like * and $?
Yes. The asterisk matches any sequence of characters within a path and the dollar sign anchors a pattern to the end of the URL. These are the two patterns most often misused in hand-written files, so testing a few real URLs against them is worth the minute it takes.
Is the robots.txt file I paste uploaded or stored anywhere?
No. The parser runs entirely in your browser as client-side JavaScript. Nothing you paste — the file, the path, or the user-agent — is transmitted to a server or stored after you close the page.
More free tools like this, by email
I publish new calculators and explainers regularly. Get them when they land — no spam, unsubscribe anytime.