Which spec does the matcher follow?

RFC 9309 (Robots Exclusion Protocol), which Google and most major crawlers also follow. Longest-match wins, and Allow beats Disallow on equal length.

Are my robots.txt and URL sent to a server?

No. Parsing and matching happen entirely in your browser.

Does it handle Sitemap and Crawl-delay?

Sitemap lines are listed for reference. Crawl-delay is a non-standard directive that different crawlers honor differently; it does not affect Allow/Disallow matching.

What if no rule matches?

Per RFC 9309, paths with no matching rule are allowed by default. The tester shows this as "allowed (no rule matched)".

robots.txt · allow · disallow · simulator

Test robots.txt rules

Paste your robots.txt, pick a user-agent and path, and see exactly which rule matched. Longest-match wins, just like real crawlers.

robots.txt Tester

Verdict

disallowed

match

Googlebot

agent

Path

/private/page

tested

Rule

/private/

matched

input · robots.txt

output · verdict

Disallowed

Matched rule: /private/

Longest matching Allow/Disallow wins. Patterns support * wildcards and $ end-anchor.

Quick Start

How to test robots.txt

Browsers and crawlers obey the longest-match Allow/Disallow rule for the most specific matching user-agent group. Groups without their own match fall back to *.

01
Paste robots.txt
Drop in your live or draft robots.txt content.
Lookups reference the IANA registries (MIME, HTTP status, TLDs) bundled at build time — no live network calls, no rate limits, no leaked queries.
pasteIANAregistryofflinelive
input · queryrules · IANA
```
User-agent: *
```
loaded · IANA bundlespec · ready
02
Type a user-agent
e.g. Googlebot, Bingbot, AdsBot-Google.
Builders emit canonical syntax per the relevant RFC (9110 for HTTP, 5322 for headers, 3986 for URLs) so the output passes spec-compliant parsers.
uaRFC 9110canonicalspec-compliantlive
ua · optionsGooglebot · live
```
Googlebot
```
options applied · livecanonical
03
Type a path
Path-only — no scheme or host. e.g. /private/page.
Validation flags common pitfalls — uppercase header names, smart quotes inside Content-Type, missing ;charset=utf-8 — before they bite production.
pathvalidationlintpitfallslive
path · looked up/private/page · ready
```
/path
```
result · livevalidated
04
Inspect verdict
Allowed or Disallowed, plus the exact rule that decided it.
Copy writes the snippet directly into your clipboard; downloads emit the right extension and MIME so curl/httpie pick them up correctly.
resultclipboardMIME-correctblobcurl-ready
ok · copiedmatched · done
```
Allow: /...
```
shipped · ready to shipsuccess

Pro tips

Web-spec details that bite later

Idempotency saves retries

POST/PUT/DELETE should be safely retried. Use idempotency keys and return the same response on duplicate requests.

Match RFC 9110 status codes

Don’t invent codes. 422 for validation errors, 409 for conflicts, 429 for rate limits. Standard codes work with off-the-shelf clients.

Set <code>Cache-Control</code> deliberately

Default browser caching is conservative. Use max-age + stale-while-revalidate for hot endpoints.

Validate Open Graph in dev

OG tags break silently. Use a previewer like OG Generator + Facebook’s debugger before launch.

CORS is a server concern

The browser sends the preflight; the server decides. Wildcard origins disable credentials — be explicit.

MIME drives behaviour

Servers must send the right Content-Type. Browsers sniff but the spec says they shouldn’t — be explicit to avoid surprises.

Use Cases

When to test robots.txt

Pre-deploy QA, regex sanity, crawl debugging.

Pre-deploy QA

Catch a typo before crawlers de-index live pages. Bundled from IANA registries at build time — no live network call, no rate limit, no leaked queries. The MIME, TLD, and status-code tables are baked in so the tool works offline once loaded.

spec-compliant

Bot-specific rules

Verify Googlebot vs Bingbot vs * behave as intended. Output is canonical per the relevant RFC (9110 for HTTP, 5322 for headers, 3986 for URLs), so spec-compliant parsers accept it as-is. Cross-browser and cross-library behaviour stays consistent.

offline

Wildcard sanity

Confirm * and $ anchors match what you expect. Pairs with the curl-to-fetch converter for end-to-end test stubs — copy the curl from DevTools, paste, get an idiomatic fetch() body. Drop into Playwright/Vitest fixtures without manual translation.

IANA-backed

Crawl budget tuning

See exactly which URLs are blocked across user-agents. Catches common pitfalls (uppercase header names, smart quotes inside Content-Type, missing ;charset=utf-8) before they bite production. Validation flags rather than auto-corrects so you stay in control.

RFC-aligned

Migration QA

Make sure new URL structures aren't accidentally blocked. Validation runs as you type — copy only when the output is shaped correctly. The output drops directly into nginx, Apache, or CDN configs without re-formatting or re-quoting.

production-grade

Documentation

Share a tested rule + verdict in a PR review. Same input plus same options always produces identical canonical output, so the tool is safe to embed in CI scripts that diff against expected configs.

curl-ready

FAQ

Frequently asked questions

Which match wins?

Longest matching Allow/Disallow rule wins, evaluated against the most-specific User-agent group that names your bot.

Are wildcards supported?

Yes — * matches any characters, $ anchors the end of the path.

Do crawl-delay and Sitemap matter here?

Not for path testing. They're parsed but ignored when deciding allow/disallow.

Is robots.txt enforced?

It's advisory — well-behaved crawlers honor it, but malicious bots ignore it. Use auth for real protection.

Does it run locally?

Yes. Parsing and matching run entirely in your browser.

Suggest a tool

Something missing?

Tell us what you need. We ship the top requests every two weeks.

47 in queue·8 shipped this month

Request queued. Thanks!