Skip to content

ClaudeBot

Anthropic's training crawler. Blocking it signals Anthropic to leave your future pages out of model training.

Check if your site lets ClaudeBot in Run the full AI Visibility Check

Last verified: October 2026, against Anthropic's own documentation.

ClaudeBot at a glance.

Owner
Anthropic
Purpose
Training. Anthropic says ClaudeBot collects web content that could contribute to training its generative AI models. When a site restricts it, Anthropic treats that as a signal to exclude the site's future materials from its training datasets.
robots.txt token
ClaudeBot
Full user agent
Not published.

Anthropic's page names the robots.txt token ClaudeBot but doesn't publish a full user-agent string.

IP ranges
claude.com/crawling/bots.json

Anthropic publishes one IP list for its crawlers, not one per bot. It also warns that blocking by IP may not work as an opt-out, because it stops the bot from reading your robots.txt.

Respects robots.txt?
Respects robots.txt. Yes. Anthropic says its bots honor industry-standard robots.txt directives, support the non-standard Crawl-delay extension, and won't try to bypass CAPTCHAs. It asks you to repeat the rule on every subdomain you want to opt out.

The robots.txt rules to copy.

Your robots.txt lives at the root of your domain, like yoursite.com/robots.txt. On Shopify you edit it by adding a robots.txt.liquid template to your theme. Shopify warns that a wrong edit can cost you all your traffic, so change one rule at a time.

Allow ClaudeBot

User-agent: ClaudeBot
Allow: /

A bot with no rule is already allowed. Add this only when a broader rule, such as User-agent: * with Disallow: /, would otherwise block it.

Block ClaudeBot

User-agent: ClaudeBot
Disallow: /

Robots.txt rules apply per host. Repeat them on each subdomain you want covered.

Should you block ClaudeBot on a local business site or a Shopify store?

Same trade as GPTBot. Anthropic treats a block as a signal to leave your future pages out of its training data, and it lists ClaudeBot separately from Claude-SearchBot and Claude-User, so a ClaudeBot rule on its own doesn't block search or user fetches.

I'd allow it on most small business sites. If you do block it, use robots.txt rather than an IP block, since Anthropic says the IP route may not hold.

A rule you didn't write can block it too: an old robots.txt template, a security plugin or a CDN bot setting. The AI crawler checker reads your robots.txt for ClaudeBot and the other bots in this directory.

The 12 bots, sorted by job

  1. Search and answers

    3 bots

    They build the index an AI assistant searches when it answers.

    If you block themPer their owners, blocking one can drop you from that assistant's search answers.

  2. Fetching a page when a user asks

    3 bots

    They fetch your page because a person asked the assistant about it.

    If you block themBlocked at your firewall, the assistant can't read your page for the person who asked. Two of the three may not follow a robots.txt block.

  3. ClaudeBot is in this group

    Model training

    6 bots

    A block here tells the owner to keep your pages out of AI model training. CCBot is here because its open archive can be used that way.

    If you block themFor most, the owner says blocking doesn't remove you from its search features. Meta says its crawler also indexes content for its products.

Where ClaudeBot sits among the 12 AI bots in this directory, grouped by what each one does, with what blocking each group costs you.

Related crawlers.

Sources

Last verified: October 2026. Where the owner doesn't document something, this page says so instead of guessing.

  1. Anthropic, Does Anthropic crawl data from the web, and how can site owners block the crawler?

Crawler access is the first thing I check in an AI answer audit. Most of what decides who gets named is what other sites say about you.

Check your robots.txt Or run the full AI Visibility Check