Free robots.txt Validator & Generator
Find every line of a robots.txt that a crawler would trip on — then generate a correct one, with or without AI crawlers.
| Verdict | Check | Measured |
|---|
robots.txt generator
Pick a policy, add the paths to keep crawlers out of, and copy the file. Then run the validator above on your live site.
What the validator reports
- The file itself — HTTP status (a 5xx makes Google crawl nothing), plain text rather than an HTML page served as 200, and the Content-Type.
- Every problem line — rules outside any group, misspelled directives, unsupported ones (
noindex,crawl-delayfor Google), full URLs where a path belongs, a relative Sitemap. - Whole-site blocks — the
*group or Googlebot shut out of the home page. - Sitemap lines — whether the file tells crawlers where your sitemap is.
To see what the file decides for one URL and each AI crawler, with the deciding line, use the AI crawler access test.
Questions
What does the robots.txt validator check?
That /robots.txt answers 200 as plain text (not an HTML page), its size against Google's 500 KiB limit, and every line a crawler would trip on: rules before any User-agent, misspelled or unsupported directives, full URLs where a path belongs, a Sitemap given as a path, and a * group or Googlebot blocked from the whole site. Each finding names its line.
Does Google support noindex in robots.txt?
No — Google stopped honouring noindex, nofollow and crawl-delay rules in robots.txt in September 2019. Use a meta robots tag or an X-Robots-Tag header to keep a page out of the index.
Why does blocking a page in robots.txt not remove it from Google?
robots.txt controls crawling, not indexing. A blocked URL can still be indexed from links pointing at it, shown without a description. To remove a page, let Google crawl it and serve noindex.
How are Allow and Disallow decided when both match?
The longest matching rule wins, and on a tie Allow wins (RFC 9309, and how Google reads it). A crawler with a group of its own ignores the * group entirely; several groups naming the same crawler are merged.
Which AI crawlers does the generator block?
"Block AI training" blocks the 11 training crawlers (GPTBot, ClaudeBot, anthropic-ai, CCBot, Google-Extended, Applebot-Extended, Bytespider, meta-externalagent, MistralAI-Training, DeepSeekBot, QwenBot) and leaves the search and answer agents that cite your pages alone. "Block all AI crawlers" blocks all 31 tokens we track.
More free SEO tools
- llms.txt checker — Is there an llms.txt, is it really markdown, and does it link to the pages that matter? No file: a draft from your home page.
- AI crawler access test — What robots.txt tells GPTBot, ClaudeBot, PerplexityBot and every other AI crawler — the deciding line, a tested fix, and whether the server actually lets GPTBot in.
- NAP consistency checker — Your business name, phone and address as your site declares them, compared field by field with your Google Business Profile, the Facebook and directory pages your site links to, and any you paste.
- Free SEO checker — every check on one page, with its threshold.
- Free SEO audit tool — the full report: findings explained, prioritised, with the fix.