Robots.txt Generator
Note: This file belongs at the root of each host and protocol it applies to — example.com and shop.example.com need their own. It must be reachable at /robots.txt exactly; a crawler will not look anywhere else.
The matching rules have their own traps. Paths are prefix matches, so Disallow: /admin also blocks /administrator and /admin-guide. Everything is case sensitive. A bare Disallow: with nothing after it means allow everything, which is the opposite of what it looks like. And the whole file only applies to the exact host and protocol it is served from — a separate subdomain needs its own.
When several groups could match, a crawler uses only the single most specific User-agent group and ignores the rest entirely. So a rule you add to the wildcard group does not apply to Googlebot if a Googlebot group exists anywhere in the file. That surprises people constantly, and the generator warns when a per-agent group would shadow a wildcard rule.
The AI crawler presets are included because the question comes up on every site now. Blocking them is a content decision rather than a technical one, and it is worth knowing they are honoured voluntarily.
Frequently Asked Questions
Does Disallow remove a page from Google?
No. It stops the page being crawled, which is not the same thing — a disallowed URL that other sites link to can still be indexed, shown without a description. To actually remove a page, allow crawling and serve a noindex meta tag or header.
Why is my noindex tag being ignored?
Because the page is also disallowed in robots.txt, so the crawler never fetches it and never sees the tag. The two directives work against each other. Remove the Disallow line and let the crawler read the noindex.
Do I need a robots.txt at all?
Not technically — a missing file means everything may be crawled, which is often correct. It becomes worth having when you need to keep crawlers out of search result pages, faceted navigation, or staging paths, and as somewhere to declare your sitemap.
Why is my wildcard rule not applying to Googlebot?
Because a crawler obeys only the most specific group that names it, and ignores every other group completely. If a Googlebot section exists anywhere in the file, Googlebot never reads the wildcard section — so any rule you want it to follow has to be repeated inside its own group.
Can I block AI crawlers?
You can list them and the major ones honour it, but compliance is voluntary — there is no enforcement mechanism. Anything that must not be accessed by an automated client needs authentication, not a line in a text file.
