AI crawler list: user agents and robots.txt tokens.
Free reference · 12 bots · Last verified October 2026
Every bot here is documented by the company that runs it, and every fact links to that company's own page. Each entry covers the user agent and IP list where the owner publishes them, whether it respects robots.txt, rules you can copy, and what blocking it means for a local business or a Shopify store.
To check by hand, follow the method I use in every audit.
The 12 bots, sorted by job
-
3 bots
-
Fetching a page when a user asks
3 bots
-
6 bots
Search crawlers
These build the indexes AI assistants search when they answer. Per their owners, blocking one can drop you from that assistant's search answers. For a local business or a store, these are the ones to keep open.
- OAI-SearchBot The crawler behind ChatGPT search. OpenAI says pages that block it won't be shown in ChatGPT's search answers. Respects robots.txt
- Claude-SearchBot Anthropic's search crawler. Anthropic says blocking it may reduce how visible you are in Claude's search results. Respects robots.txt
- PerplexityBot Perplexity's search crawler. Perplexity says it isn't used to collect training data for foundation models. Respects robots.txt
User-triggered agents
These fetch a page because a person asked the assistant to. Two of the three owners say robots.txt may not apply to theirs.
- ChatGPT-User ChatGPT visiting a page because a user asked it to. OpenAI says robots.txt rules may not apply. robots.txt may not apply
- Claude-User Claude fetching a page because a user asked a question. Anthropic says it honors robots.txt. Respects robots.txt
- Perplexity-User Perplexity fetching a page to answer a user's question. Perplexity says it generally ignores robots.txt. Generally ignores robots.txt
Training crawlers and tokens
These decide whether your pages feed model training. For most of these, the owner says blocking doesn't remove you from its search features. Meta's is the one to read closely.
- GPTBot OpenAI's training crawler. It collects pages that may be used to train OpenAI's foundation models. Respects robots.txt
- ClaudeBot Anthropic's training crawler. Blocking it signals Anthropic to leave your future pages out of model training. Respects robots.txt
- Google-Extended A robots.txt control token, not a crawler. It tells Google whether it may use your pages to train Gemini and to ground answers in some of its other systems. robots.txt token only
- CCBot Common Crawl's crawler. Common Crawl describes the result as an open repository of web crawl data. Respects robots.txt
- Applebot-Extended A control token, not a crawler. It decides whether Apple may use what Applebot crawled to train its generative models. robots.txt token only
- Meta-ExternalAgent Meta's crawler for training foundation models and indexing content for its products. Respects robots.txt
Not in the directory yet.
Bytespider (ByteDance). I couldn't find a ByteDance page that documents Bytespider's purpose, user agent, IP ranges or robots.txt behavior. Third-party directories describe it, but this directory only lists bots whose owner documents them, so it stays out until ByteDance publishes something.
Owners rename bots and change what they do. I re-check every entry against the owner's page and update the "last verified" date when I do.
Sources
Last verified: October 2026.
Bot access is the first check. Most of what decides whether an assistant names you is what other sites say about you, and that's what my AI answer audit measures. For a free first read, run the AI Visibility Check.