AI Crawler Access Test
See what robots.txt tells 33 crawlers about this exact URL — and whether the server lets a GPTBot request through at all.
| Verdict | Check | Measured |
|---|
What the AI crawler test reports
- robots.txt, per agent — for the URL you entered, whether each crawler is allowed, and the exact rule — with its line number — that decided it.
- A tested fix — when a search or answer crawler is blocked, the lines to add to
robots.txt, checked against your own file before they are shown. - Own group or the default — whether the crawler has its own
User-agent:block or falls under*. - Live fetch as GPTBot — the page requested with GPTBot's user-agent: the HTTP status, the server, any Cloudflare mitigation header, and known bot-challenge fingerprints (Cloudflare, Sucuri, DDoS-Guard).
- Googlebot and Bingbot — the same rule check for the two search crawlers, so an AI block that accidentally caught Google is visible.
Every free SEO check runs the same probe as part of its AI-readiness checks, next to llms.txt, structured data and authorship.
How to allow or block an AI crawler
In robots.txt, one group per agent. To allow the search agents and block the training crawlers:
User-agent: OAI-SearchBot User-agent: PerplexityBot User-agent: ChatGPT-User Allow: / User-agent: GPTBot User-agent: CCBot User-agent: Bytespider User-agent: Google-Extended Disallow: /
Then check the edge: in Cloudflare, Security → Bots → AI bots is where the default block lives, and it wins over robots.txt. Re-run this test after changing either — the live row is the one that tells you what happened.
Questions
Which AI crawlers does it check?
All 31 AI crawler tokens we track: GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, anthropic-ai, Applebot, PerplexityBot, Perplexity-User, CCBot, Google-Extended, Applebot-Extended, Bytespider, meta-externalagent, Amazonbot, DuckAssistBot, MistralAI-Index, MistralAI-User, MistralAI-Training, Meta-ExternalFetcher, Google-Agent, GoogleAgent-Mariner, Gemini-Deep-Research, DeepSeekBot, QwenBot, Kimi-SearchBot, YouBot, cohere-ai, PetalBot, OAI-AdsBot — plus Googlebot and Bingbot for comparison, since Copilot grounds on Bing's crawl.
What is the difference between blocking in robots.txt and the live test?
robots.txt is a request the crawler is asked to honour. The live test fetches the page with GPTBot's user-agent and reports what the server actually did. Since July 2025 every new Cloudflare zone blocks AI bots at the edge by default, so a site can publish a welcoming robots.txt and still refuse every AI agent — and the owner never sees it in a browser.
Should I block AI crawlers?
Training crawlers (CCBot, Bytespider, Google-Extended, Applebot-Extended) fetch content to train models and send no traffic; blocking them is a defensible choice. The search agents (OAI-SearchBot, PerplexityBot, ChatGPT-User) fetch pages to answer a user's question and cite the source — blocking them removes you from those answers.
Why might the live test fail on a site that allows GPTBot?
The request comes from our scanner's address, not from OpenAI's published IP ranges. A firewall that verifies bots by IP will refuse it as an unverified GPTBot, which is correct behaviour. A 403 here is a real vantage — plenty of agents arrive exactly like this — but not proof of what the verified crawler gets.
Does robots.txt apply per page?
Yes. The table shows the rule that decides this exact URL: the longest matching Allow or Disallow in the agent's own group, or the * group when the agent has none. A site can allow the home page and block a directory.
More free SEO tools
- llms.txt checker — Is there an llms.txt, is it really markdown, and does it link to the pages that matter? No file: a draft from your home page.
- robots.txt validator & generator — Every line a crawler would trip on, by line number — and a generator for a correct file with your AI crawler policy.
- NAP consistency checker — Your business name, phone and address as your site declares them, compared field by field with your Google Business Profile, the Facebook and directory pages your site links to, and any you paste.
- Free SEO checker — every check on one page, with its threshold.
- Free SEO audit tool — the full report: findings explained, prioritised, with the fix.