CCBot
Common Crawl's crawler. Common Crawl describes the result as an open repository of web crawl data.
Last verified: October 2026, against Common Crawl's own documentation.
CCBot at a glance.
- Owner
- Common Crawl
- Purpose
- Training. Common Crawl is a non-profit that maintains an open repository of web crawl data, and CCBot is its crawler. Its FAQ says the data is provided at no cost for research and analysis. Its own pages don't name AI training as a use. I file it with the training bots for one reason: the archive is open, so once your pages are in it, you don't control who uses them.
- robots.txt token
CCBot- Full user agent
-
CCBot/2.0 (https://commoncrawl.org/faq/)Common Crawl warns that some crawlers falsely identify themselves as CCBot.
- IP ranges
- index.commoncrawl.org/ccbot.json
Common Crawl says CCBot runs from dedicated IPv4 and IPv6 ranges. Reverse DNS works for IPv4: real CCBot addresses resolve to hostnames under crawl.commoncrawl.org.
- Respects robots.txt?
- Respects robots.txt. Yes. Common Crawl's FAQ says CCBot checks robots.txt before fetching, obeys Crawl-delay and honors nofollow on links.
The robots.txt rules to copy.
Your robots.txt lives at the root of your domain, like yoursite.com/robots.txt. On Shopify you edit it by adding a robots.txt.liquid template to your theme. Shopify warns that a wrong edit can cost you all your traffic, so change one rule at a time.
Allow CCBot
User-agent: CCBot
Allow: /A bot with no rule is already allowed. Add this only when a broader rule, such as User-agent: * with Disallow: /, would otherwise block it.
Block CCBot
User-agent: CCBot
Disallow: /Robots.txt rules apply per host. Repeat them on each subdomain you want covered.
Should you block CCBot on a local business site or a Shopify store?
A Disallow stops the next crawls. Pages already in past crawls stay in those archives.
This one is a judgment call. It doesn't feed any one assistant's search results that I can source, so blocking it costs you little that's documented. If you're fine with your site being public data, there's no reason to block it either.
A rule you didn't write can block it too: an old robots.txt template, a security plugin or a CDN bot setting. The AI crawler checker reads your robots.txt for CCBot and the other bots in this directory.
The 12 bots, sorted by job
-
Search and answers
3 bots
They build the index an AI assistant searches when it answers.
If you block themPer their owners, blocking one can drop you from that assistant's search answers.
-
Fetching a page when a user asks
3 bots
They fetch your page because a person asked the assistant about it.
If you block themBlocked at your firewall, the assistant can't read your page for the person who asked. Two of the three may not follow a robots.txt block.
-
CCBot is in this group
Model training
6 bots
A block here tells the owner to keep your pages out of AI model training. CCBot is here because its open archive can be used that way.
If you block themFor most, the owner says blocking doesn't remove you from its search features. Meta says its crawler also indexes content for its products.
Related crawlers.
Sources
Last verified: October 2026. Where the owner doesn't document something, this page says so instead of guessing.
Crawler access is the first thing I check in an AI answer audit. Most of what decides who gets named is what other sites say about you.