Is robots.txt a regex format?

Not exactly. It uses two wildcards: * (any sequence) and $ (end of URL). The tester translates these to regex internally to apply most-specific-rule-wins semantics, which is the convention used by Googlebot and most major crawlers.

How is the most specific rule chosen?

By the length of the path pattern before any wildcard. Longer (more specific) paths win, with Allow taking precedence over Disallow at equal specificity — the convention defined by Google's robots.txt spec.

Does the tester fetch the URL?

No. It evaluates whether the URL would be allowed based on the pasted robots.txt. No HTTP request is made.

Is the robots.txt uploaded?

No. Parsing and evaluation are both local.

robots.txt · user-agent · allow/disallow · live

Check whether a URL is crawlable by a bot

Drop in a robots.txt, pick a user-agent and a URL — see the verdict and the winning rule. Runs locally.

robots.txt Tester

Verdict

disallowed

Disallow

Groups

in robots

Googlebot

→ Googlebot

Status

/private/

winning rule

input · robots.txt + UA + URL

robots.txt

User-agentURL or path

Only * and $ wildcards. Longer rules win. Allow beats Disallow at equal length.

output · verdict

Disallowed

winning rule: Disallow: /private/

matched group user-agents: Googlebot

Quick Start

How to test robots.txt rules

Paste a robots.txt, pick a user-agent and URL — see whether the URL is crawlable and which rule wins.

01
Paste robots.txt
Multiple User-agent groups, Allow and Disallow lines. Comments are stripped.
Patterns compile through the JavaScript RegExp constructor, so syntax errors surface immediately and engine quirks like Unicode property escapes (\p{L}) work when the u flag is on.
patternECMAScriptliterallive
input · literalrobots · regex
User-agent: * Disallow: /admin/
loaded · ECMAScriptPCRE · ready
02
Pick UA + URL
Match a specific bot or use * for the default group.
Each flag character mirrors the ECMAScript literal: g enables iteration, i folds case, m anchors per line, s lets . match newlines, u unlocks proper Unicode, y is sticky, and d emits indices.
flagsoptionsg · i · mlive
config · optionsua · flags
Googlebot · /private/page
flags applied · liveECMA · live
03
Read the verdict
Allowed or disallowed, with the winning rule called out.
Watch for catastrophic backtracking on nested quantifiers — a common ReDoS source. The tool caps match counts to keep the page responsive on pathological patterns.
matchgroupshighlightReDoS-safe
run · outputverdict · matches
Disallowed by /private/
matches · livegroups · live
04
Copy or export
Grab the output as JSON, CSV, or plain text — ready for your editor or pipeline.
Copy writes via navigator.clipboard.writeText; downloads use a Blob and object URL, so nothing transits the network and the output is byte-identical to what you see on screen.
copyexportclipboardblob
export · exportedready · saved
Copied
ready to shipsuccess

Pro tips

Regex patterns that don't melt down

Use named groups

(?<year>\d{4}) reads better than $1 and survives reordering. Supported in ECMAScript 2018+.

Anchor strictly

Use ^/$ with the m flag when matching whole lines. Without anchors, partial matches sneak through.

Avoid catastrophic backtracking

Nested quantifiers like (a+)+ can explode. Run patterns through ReDoS Detector first.

Escape every literal

Backslash any meta character you want literally: \.\$\^\*\+\?\[\]\{\}\|\\.

Test edge cases

Empty string, single character, unicode, the maximum length you expect — write fixtures for each.

Patterns aren’t parsers

Don’t parse HTML, JSON, or balanced parens with regex. Use a proper parser when the grammar is recursive.

Use Cases

When to check crawlability

SEO audits and bot policy reviews.

SEO audit

Confirm priority pages aren't accidentally blocked. Compiles through the JavaScript RegExp constructor — engine quirks (Unicode property escapes, sticky matching, named groups) work when their flags are on. Error messages cite the exact position of any syntax issue.

ReDoS-safe

Per-bot tuning

Compare Googlebot vs Bingbot access. Pairs with the ReDoS detector to catch catastrophic backtracking before you deploy. A pattern that handles 1k-character input fine can become a denial-of-service vector on attacker-controlled 1M-character input.

ECMA-compliant

Migration

Verify new robots.txt before going live. Output exports as JavaScript, Python re, Java util.regex, .NET System.Text.RegularExpressions, PCRE, and POSIX — language quirks are highlighted on conversion so you don't ship a JS-only pattern to a Go service.

spec-grade

Docs

Document what bots can crawl. Capture groups (named or numbered) round-trip cleanly into replacement strings, JSON output, or CSV extraction without re-typing. Backreferences ($1, ${name}) preserve semantics across languages.

cross-engine

Teach Allow/Disallow

Show how more specific rules win. Test against edge cases — empty input, single character, max length, multibyte boundaries — to catch off-by-one and zero-length match bugs before they reach production.

test-ready

Block leaks

Catch overshoots that block /admin/public. All matching runs client-side so secret patterns never leave the browser. Useful when the pattern itself is sensitive (PII regexes, internal data formats, security filters).

production-ready

FAQ

Frequently asked questions

Are wildcards supported?

Yes — * matches any chars, $ anchors to end of URL. These are the only two wildcards in robots.txt.

Allow vs Disallow priority?

The longer (more specific) rule wins. At equal length, Allow beats Disallow.

What about Crawl-delay?

Not handled by this tester — it's directive-only, not URL-matching.

Sitemap directive?

Ignored — Sitemap: lines don't affect crawlability.

Is anything sent to a server?

No. The tool runs entirely in your browser — your pattern and sample text never leave the page.

Suggest a tool

Something missing?

Tell us what you need. We ship the top requests every two weeks.

47 in queue·8 shipped this month

Request queued. Thanks!