Is my robots.txt uploaded?

No. It is built locally in your browser.

Does order of rules matter?

Within a group, longest-match wins regardless of order (RFC 9309). Across groups, agents pick the most specific group matching their name, so groups can appear in any order.

Should I block /admin/ in robots.txt?

Don't rely on robots.txt for security: it advertises the path's existence. Use authentication for actual access control. robots.txt only stops well-behaved crawlers from listing it.

Where should robots.txt live?

At the root of each host and protocol: https://example.com/robots.txt. Subdirectory robots.txt files are ignored.

robots.txt · allow · disallow · sitemap · live

Generate a robots.txt file

Add User-agent groups with Allow / Disallow rules, optional crawl-delay, plus Sitemap and Host directives. Copy or download.

robots.txt Generator

Groups

Allow rules

total

Disallow rules

total

Bytes

155

output

input · ua groups

Allow

Disallow

Allow

Disallow

output · robots.txt

User-agent: *
Disallow: /admin/
Disallow: /private/
Disallow: /api/*.json

User-agent: Googlebot
Allow: /public/

Sitemap: https://example.com/sitemap.xml

Quick Start

How to write robots.txt

Add one group per User-agent (or use *), list Allow/Disallow paths, and end with Sitemap + Host.

01
Add User-agent groups
Use * for all bots; name specific ones to override.
Lookups reference the IANA registries (MIME, HTTP status, TLDs) bundled at build time — no live network calls, no rate limits, no leaked queries.
uaIANAregistryofflinelive
groups · queryua · IANA
```
User-agent: *
```
loaded · IANA bundlespec · ready
02
List Allow / Disallow paths
One path per line. Wildcards * and end anchor $ are supported.
Builders emit canonical syntax per the relevant RFC (9110 for HTTP, 5322 for headers, 3986 for URLs) so the output passes spec-compliant parsers.
pathsRFC 9110canonicalspec-compliantlive
rules · optionspaths · live
```
Disallow: /admin/
```
options applied · livecanonical
03
Point to sitemap
Bots use the Sitemap directive to discover URLs efficiently.
Validation flags common pitfalls — uppercase header names, smart quotes inside Content-Type, missing ;charset=utf-8 — before they bite production.
sitemapvalidationlintpitfallslive
sitemap · looked upurl · ready
```
Sitemap: ...
```
result · livevalidated
04
Drop at /robots.txt
Save the file at the site root — Googlebot expects it there.
Copy writes the snippet directly into your clipboard; downloads emit the right extension and MIME so curl/httpie pick them up correctly.
deployclipboardMIME-correctblobcurl-ready
ok · copiedready · done
```
/robots.txt
```
shipped · ready to shipsuccess

Pro tips

Web-spec details that bite later

Idempotency saves retries

POST/PUT/DELETE should be safely retried. Use idempotency keys and return the same response on duplicate requests.

Match RFC 9110 status codes

Don’t invent codes. 422 for validation errors, 409 for conflicts, 429 for rate limits. Standard codes work with off-the-shelf clients.

Set <code>Cache-Control</code> deliberately

Default browser caching is conservative. Use max-age + stale-while-revalidate for hot endpoints.

Validate Open Graph in dev

OG tags break silently. Use a previewer like OG Generator + Facebook’s debugger before launch.

CORS is a server concern

The browser sends the preflight; the server decides. Wildcard origins disable credentials — be explicit.

MIME drives behaviour

Servers must send the right Content-Type. Browsers sniff but the spec says they shouldn’t — be explicit to avoid surprises.

Use Cases

When to write robots.txt

Crawl budget, private pages, bot-specific rules.

Crawl budget control

Block low-value pages to focus crawlers on what matters. Bundled from IANA registries at build time — no live network call, no rate limit, no leaked queries. The MIME, TLD, and status-code tables are baked in so the tool works offline once loaded.

spec-compliant

Block admin & staging

Keep dashboards and test routes out of search. Output is canonical per the relevant RFC (9110 for HTTP, 5322 for headers, 3986 for URLs), so spec-compliant parsers accept it as-is. Cross-browser and cross-library behaviour stays consistent.

offline

Bot-specific behaviour

Different rules for Googlebot, Bingbot, Twitterbot. Pairs with the curl-to-fetch converter for end-to-end test stubs — copy the curl from DevTools, paste, get an idiomatic fetch() body. Drop into Playwright/Vitest fixtures without manual translation.

IANA-backed

Sitemap pointers

Tell crawlers where your sitemaps live. Catches common pitfalls (uppercase header names, smart quotes inside Content-Type, missing ;charset=utf-8) before they bite production. Validation flags rather than auto-corrects so you stay in control.

RFC-aligned

Migrations

Temporarily block legacy URLs during a redirect rollout. Validation runs as you type — copy only when the output is shaped correctly. The output drops directly into nginx, Apache, or CDN configs without re-formatting or re-quoting.

production-grade

AI training opt-out

Disallow bots like GPTBot, CCBot, Google-Extended. Same input plus same options always produces identical canonical output, so the tool is safe to embed in CI scripts that diff against expected configs.

curl-ready

FAQ

Frequently asked questions

Is robots.txt enforced?

It's advisory. Well-behaved crawlers obey it, but malicious bots ignore it. Use authentication for real protection.

Where does the file live?

Always at /robots.txt on the site root. One per host.

Are wildcards supported?

Yes — Google and most major crawlers support * and $ in the path pattern.

What does Crawl-delay do?

Asks the bot to wait between requests. Google ignores it; Bing and Yandex honor it.

Does it run locally?

Yes. Generation runs entirely in your browser — nothing is uploaded.

Suggest a tool

Something missing?

Tell us what you need. We ship the top requests every two weeks.

47 in queue·8 shipped this month

Request queued. Thanks!