Skip to content

Free AI crawler checker · No signup · Part of the AI Visibility Check

Robots.txt checker for AI bots: test GPTBot, ClaudeBot and 12 more

Every website has a small file, robots.txt, that tells search engines and AI tools which pages they may visit. Type in your site and I'll show which AI tools yours lets in, one by one. Then I open your page as four of them, to catch a security setting that turns them away even when the file says yes.

Check AI crawler access

AI crawler check result

Sample result · fictional business

Getting in is one part of four. The full check also tests whether AI tools can read your page, and gives you ten questions to see whether they recommend you.

Show the bot-by-bot detail

robots.txt, bot by bot

Fetched as a browser and as the bots

robots.txt tester: try a draft before you publish

Paste a robots.txt and pick a path. It's checked in your browser; nothing is sent anywhere.

Last verified: October 2026

How the AI crawler checker works

Most robots.txt checkers stop at the file, and plenty of sites allow every AI bot in robots.txt and then block them one layer down at the CDN. This checker does both halves: it evaluates the rules, then sends real requests to see what each bot gets back. Want to do it by hand? My guide to checking AI crawler access walks through it.

Here is the path a bot takes to your page. This checker tests stops 2 and 3. The AI Visibility Check covers 4 and 5.

How an AI bot gets to your page

  1. An AI bot asks for your page

    • It sends its name, like GPTBot or OAI-SearchBot

    What I checkI label each bot by its job: search, training, or fetching a page for a user, so you know which ones to keep open.

  2. Your robots.txt rules

    • Allow: it carries on
    • Block: it should stop here

    What I checkI show which robots.txt line decides each bot.

  3. Your firewall or CDN bot settings

    • Let through
    • Challenge page
    • Block

    What I checkI load your page as a browser and as four AI bots, and compare.

  4. Your page

    • Text in the HTML: the bot can read it
    • Text only after scripts run: a bot that doesn't run scripts misses it

    What I checkThe AI Visibility Check counts the words in your raw HTML, before any script runs.

  5. The AI tool can read and quote you

    What I checkGetting in is one part. The AI Visibility Check gives you ten questions to ask the AI tools yourself, and you score whether they name you.

A bot has to get past every stop to read you. A block at any one of them ends the trip, and the fix is at that stop, not the next one.

The bots, and where each token comes from

Every user-agent token is copied from its owner's own documentation. Where an owner hasn't published one, the bot isn't here. Each AI bot has its own page in my AI crawler directory, with its user agent, IP list and copy-paste rules.

TokenOwnerJobWhat it does
Googlebot Google Search engine Crawls for Google Search.
bingbot Microsoft Search engine Crawls for Bing search.
OAI-SearchBot OpenAI AI search Finds pages to show and link in ChatGPT search answers.
ChatGPT-User OpenAI User request Opens a page when a ChatGPT user asks about it. The owner says these fetches may not follow robots.txt.
GPTBot OpenAI Model training Collects pages that may train OpenAI models.
Claude-SearchBot Anthropic AI search Indexes pages so Claude can find them for search answers.
Claude-User Anthropic User request Opens a page when a Claude user asks a question.
ClaudeBot Anthropic Model training Collects pages that may train Anthropic models.
PerplexityBot Perplexity AI search Finds pages to show and link in Perplexity answers.
Perplexity-User Perplexity User request Opens a page when a Perplexity user asks a question. The owner says these fetches may not follow robots.txt.
Google-Extended Google Model training Controls whether Gemini may train on or ground answers in your pages. It does not affect Google Search. Robots token only; the owner sends no request under this name.
Applebot-Extended Apple Model training Controls whether Apple may use what Applebot crawled to train its generative models. Robots token only; the owner sends no request under this name.
meta-externalagent Meta Model training Crawls for training Meta's AI models and for indexing content in its products.
CCBot Common Crawl Model training Builds Common Crawl's open web archive, which anyone can download.

Bytespider, ByteDance's crawler, is missing on purpose. I couldn't reach a ByteDance page that documents the token, and I won't cite a third-party list for it.

A robots.txt validator that reads the rules the way the standard says

The rules come from RFC 9309, the robots.txt standard published in 2022. A group starts at one or more User-agent lines. A crawler obeys the group that names its token, matched without regard to case (RFC 9309), and, following Google's parser, ignoring a version suffix like GPTBot/1.0. It ignores the * group entirely when one does. That last part trips people up: a specific GPTBot group doesn't inherit the rules you wrote for everyone.

Inside a group, the longest matching path wins. On a tie, Allow wins. * matches any run of characters and $ anchors the end of the URL. An empty Disallow: allows everything. The file's own status matters too: a 404 means no rules, while a 5xx or a timeout means the standard says assume the whole site is off limits, and Google handles a 429 the same way at first. I follow up to five redirects to the file, as the standard asks. A robots.txt that answers with an HTML page is a misconfigured server, and I treat it as missing.

Fetching as the bots

After the rules, I load your page as a current Chrome browser, then fetch the same final address as GPTBot, OAI-SearchBot, ClaudeBot and PerplexityBot, each with the user-agent string its owner publishes. Anthropic documents the ClaudeBot token but not a full string, so that request uses a test string built around the token. I compare the HTTP status and the size of each response with the browser's.

  • Same answer: the bot got what the browser got, even if that's an error.
  • Blocked: the bot got a 4xx or 5xx while the browser got the page.
  • Challenge page: the bot got a "checking your browser" or CAPTCHA page the browser didn't.
  • Sent elsewhere or much smaller: a redirect the browser didn't get, or under a third of the browser's response.

Cloudflare AI Crawl Control and other CDN bot settings

CDNs make blocking easy and sometimes automatic. Cloudflare documents AI bot controls grouped by purpose, and its July 1, 2026 changelog says new domains from September 15, 2026 get training and agent bots blocked on pages that show ads, with search allowed. The same post says crawlers that do both search and training are caught by the training block, so check each one you care about. If your site went through a host or CDN change, this is the first thing to test.

My requests come from a Cloudflare server, not from OpenAI's or Anthropic's networks. A firewall that only admits a bot from its owner's published IP ranges will block my test and still let the real crawler through. A blocked result is a strong reason to open your firewall log, not a verdict on its own.

Before you publish a robots.txt change

Paste the new file into the tester above and pick the path you care about most, usually / or your main service page. Every bot is evaluated in your browser with the same code the live check uses, so you see which line decides each one before a crawler does. Two mistakes show up more than any others: a User-agent: * block with Disallow: / left over from a staging site, and a new bot-specific group that quietly drops the rules everyone else gets.

What to do with the result

Decide by job, not by bot. If you want to be found and cited, allow Googlebot, bingbot and the AI search bots: OAI-SearchBot, Claude-SearchBot and PerplexityBot. OpenAI says blocking OAI-SearchBot keeps you out of ChatGPT search answers, and Anthropic and Perplexity say blocking theirs can cost you visibility. Training bots are your call. OpenAI and Anthropic run training and search under separate tokens. Google-Extended is different: Google says it also covers grounding answers in Gemini apps, so blocking it can cost you there, though never in Google Search. Then make the firewall agree with whatever robots.txt says.

Crawler access is the first of four questions. The AI Visibility Check adds whether AI can read the page, and ten prompts to see whether it names you. If you'd rather have me ask the AI tools for you, that's the free AI Answer Snapshot.

Sources

AI crawler checker questions

What does the AI crawler checker test?
Two things. Your robots.txt rules for 14 crawler tokens, AI bots plus Googlebot and bingbot, evaluated the way RFC 9309 says a crawler must. And a live fetch of your page as GPTBot, OAI-SearchBot, ClaudeBot and PerplexityBot, compared with a normal browser, which catches a firewall that blocks bots robots.txt allows.

Is it free?
Yes. No signup and no email. It runs on my own site and calls no paid API.

Can I test a path other than the homepage?
Yes, two ways. After a live check, enter any path to see which rule decides each bot against the file I just fetched. Or paste a robots.txt draft and test it before you publish. Both run in your browser.

Should I block GPTBot?
That depends on whether you want your pages used to train OpenAI models. Blocking GPTBot doesn't remove you from ChatGPT search; that is OAI-SearchBot, a separate token. The same split holds for ClaudeBot and Claude-SearchBot. Google-Extended is the exception: Google says it also controls grounding in Gemini apps.

Why does my site show blocked when robots.txt allows the bot?
Something in front of the site, usually a firewall or CDN bot rule, is refusing the request before robots.txt matters. Check the bot settings at your host. Some firewalls only admit a bot from its owner's IP ranges, so a test from my server can read as blocked when the real crawler would pass.