Start free
Common Crawl · Common Crawl

CCBot

Collects pages to train Common Crawl’s AI models. Here is how to recognise it, verify it, and allow or block it.

Company
Common Crawl
Product
Common Crawl
Why it visits
Collecting for AI training
How to verify
IP list + reverse DNS
User-agent contains
CCBot

How to tell a real CCBot from a fake

  1. The user-agent contains CCBot. On its own this proves nothing: fakes copy it.
  2. The visit’s IP address is inside Common Crawl’s published list (ccbot, below). If it isn’t, it’s a fake.
  3. Or: the address’s reverse DNS name ends in .commoncrawl.org, and that name resolves back to the same address.

Check a CCBot visit

IP ranges: ccbot

5 ranges · updated 12 Oct 2026 · source index.commoncrawl.org

Show all 5 ranges
2600:1f28:365:8000::/56 3.41.188.32/29 18.97.9.168/29 18.97.14.80/29 18.97.14.88/30

Recent changes to these ranges

No changes recorded yet. When a company adds or removes ranges, it shows up here (lists are checked every day).

Allow or block CCBot in robots.txt

Block it everywhere:

User-agent: CCBot Disallow: /

Allow it everywhere:

User-agent: CCBot Allow: /

Fake bots ignore robots.txt: block them by address in your firewall instead.