How to Check If AI Crawlers Can Reach Your Site

Quick take

If an AI assistant can’t fetch your pages, it can’t read or quote them. The check takes four steps: read the robots.txt your site serves today, look for rules your CDN or platform adds on top of it, fetch your pages with each bot’s user agent, and look for the bots in your logs. It’s the first thing I check in every AI answer audit, and it’s usually the shortest part of the report.

Here is the path a bot takes to your page, and what the checker tests at each stop:

If you want the fast version, the AI crawler checker reads your robots.txt for the bots in my AI crawler directory. This post is the manual method, including the parts a robots.txt reader can’t see.

Know which bots you’re checking for

“AI crawler” covers three kinds of bot, and they don’t carry the same stakes.

Search crawlers build the index an assistant searches when it answers: OAI-SearchBot, Claude-SearchBot and PerplexityBot. Block one and you can drop out of that assistant’s search answers. OpenAI says so directly for OAI-SearchBot. Google’s AI Overviews and AI Mode don’t have their own bot. Google says a page only needs to be indexed and eligible to show with a snippet, with no additional technical requirements, so Googlebot is the one that counts there.

User-triggered agents fetch a page because a person asked the assistant to: ChatGPT-User, Claude-User and Perplexity-User.

Training crawlers and tokens decide whether your pages feed model training: GPTBot, ClaudeBot, Google-Extended, CCBot, Applebot-Extended and meta-externalagent. Google, Apple and OpenAI say blocking their training tokens doesn’t remove you from their search, and Meta says its crawler also indexes content for its products. Google says Google-Extended doesn’t affect inclusion in Google Search, and Apple says pages that disallow Applebot-Extended can still show up in Apple’s search results.

For a local business or a store, the search crawlers are the ones to protect. The training bots are a judgment call.

Step 1: read the robots.txt your site serves today

Open yoursite.com/robots.txt in a browser. Read the live URL, not the file in your theme or repo. Step 2 explains why they can differ.

Robots.txt is made of groups. Each group starts with one or more User-agent lines and lists Allow and Disallow rules under them. The standard, RFC 9309, sets how a bot picks its group. Under it, a bot obeys the group that names it and falls back to the User-agent: * group only when no group names it.

Say your file has this:

User-agent: *
Disallow: /checkout
Disallow: /account

User-agent: GPTBot
Disallow: /blog

GPTBot now follows only its own group. It’s allowed into /checkout and /account, because the * rules no longer apply to it. Every time you add a group for one bot, copy in the general rules you still want it to follow.

What to look for:

1. A User-agent: * group with Disallow: /. That blocks every bot without its own group, the search crawlers included.
2. A group naming OAI-SearchBot, Claude-SearchBot, PerplexityBot or Googlebot with a broad Disallow.
3. A block of rules you didn’t write, added by a plugin, an app or an old template. Read every bot name in it.
4. Googlebot rules, if you care about Apple. Apple says that if your file doesn’t mention Applebot but does mention Googlebot, Applebot follows the Googlebot rules.

Also check the status code. Under RFC 9309, a robots.txt that returns a 4xx error means bots may crawl everything. One that returns a 5xx error, or doesn’t respond, means they must assume they can crawl nothing. A flaky server can block every bot without a single rule in the file.

Step 2: check what your platform and CDN add

The file you wrote isn’t always the file that gets served.

On Shopify, every store starts with a default robots.txt, and you change it by adding a robots.txt.liquid template. Shopify warns that a wrong edit can cost you all your traffic, so read the live URL after any change.

On Cloudflare, managed robots.txt can prepend Disallow rules for AI crawlers to your existing file when you turn it on. The example list in Cloudflare’s docs covers training bots such as GPTBot, ClaudeBot, Google-Extended and CCBot.

Then there’s the firewall, which matters more. Robots.txt asks a bot to stay out. A firewall rule turns it away, and it never shows up in your robots.txt. Cloudflare announced in July 2025 that it was changing its default to block AI crawlers. Its current docs say new domains from September 15, 2026 get defaults that block training and agent bots on pages with ads, while search crawlers stay allowed. Cloudflare’s changelog adds a catch: “multi-purpose crawlers that combine Search and Training will be affected by the new defaults to block Training.” If a bot you care about does both, check it by name. In AI Crawl Control, blocking a crawler creates a WAF rule on your zone.

So log in and look. In Cloudflare, check AI Crawl Control and the bot settings. On any other CDN or security plugin, search the settings for “AI”, “bot” and “crawler”. In June my own Cloudflare firewall was returning 403 errors to five AI crawlers, and nothing in my analytics showed it. I wrote that one up in six AI search audits.

Step 3: fetch your pages as each bot

Now ask your server directly. This curl command requests your home page with OAI-SearchBot’s published user agent and prints only the status code:

curl -s -o /dev/null -w "%{http_code}\n" -A "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36; compatible; OAI-SearchBot/1.4; +https://openai.com/searchbot" https://yoursite.com/

On Windows, run it in PowerShell as curl.exe and change /dev/null to NUL. Swap in the user agent for each bot you care about. The directory lists the full string for every bot whose owner publishes one.

Run it on your home page and two or three pages you want quoted, like a service page and a product page. A 200 is what you want. A 403, 429 or 503 means something turned the bot away. A 200 can still be a challenge page, so drop -o /dev/null once and check that the HTML is your page.

This test has one limit. The request comes from your IP address, not the bot’s. A firewall rule that checks verified bot IP ranges, or blocks data-center traffic, can treat the real bot differently from your test. In my AI answers field report, all 530 fetches across 9 business sites came back 200, and they came from a residential IP, which is why I report them with this caveat. A clean curl result rules out rules keyed to the user agent. Step 4 covers the rest.

Step 4: look for the real bots in your logs

The last check is whether the bots show up. Search your server or CDN logs for the token, like OAI-SearchBot or PerplexityBot, and look at the status codes they got. In Cloudflare, AI Crawl Control shows which AI crawlers visited and how often.

Anyone can fake a user agent, so treat the token as a claim. OpenAI, Perplexity, Anthropic, Common Crawl and Apple publish IP lists or reverse DNS rules you can check a request against. Each bot’s page in the directory links to its list.

If a search crawler never shows up, that doesn’t prove a block. A small site may not get crawled often. If it shows up and gets 403s, you’ve found your problem.

User-triggered agents don’t follow the same rules

Don’t count on robots.txt to control the user agents. OpenAI says robots.txt rules may not apply to ChatGPT-User, because a user started the fetch. Perplexity says Perplexity-User generally ignores them. Anthropic says its bots, Claude-User included, honor robots.txt.

For a local business or store, the better move is to let these through. They’re how an assistant reads your page when a customer asks about you.

What a clean result does and doesn’t tell you

Passing all four steps means the assistants can read you. It doesn’t mean they’ll name you. In the field report, every crawler fetch got through, and businesses still went unnamed on most questions that didn’t include their name. The ones that got named were the ones other sites talked about.

So run this check, fix anything it finds, and move on. The free AI Visibility Check covers access along with the other things a tool can read from outside your site. The next step is finding out what the answers say, and I wrote up how I do that for local businesses. To have me look, start with the free AI Answer Snapshot on my AI answer audit page.

Frequently asked questions

See how I check what AI assistants tell people about a business.