Anthropic, the company behind Claude, uses three bots:
- ClaudeBot collects public web content that may be used to train Anthropic’s models.
- Claude-SearchBot crawls to improve the search results Claude gives its users.
- Claude-User fetches a page when a person asks Claude something that needs it.
All three come from the addresses Anthropic publishes, and each can be allowed or blocked on its own in robots.txt.
The three bots side by side
| Bot | What it does | If you block it |
|---|---|---|
| ClaudeBot | Collects content that may be used for training | Your future content isn’t collected for training |
| Claude-SearchBot | Crawls to improve Claude’s search results | Your pages may be less likely to come up in Claude’s answers |
| Claude-User | Fetches a page for a person, right now | Claude can’t open your pages when users ask it to |
How to recognise them
Their user-agents contain the bot’s name, for example:
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +claudebot@anthropic.com)In logs, ClaudeBot behaves like a classic crawler: many pages over time. Claude-User shows up as a single page, or a handful, at the moment someone asks Claude about it.
You may also see two older names, anthropic-ai and Claude-Web, in old logs or old robots.txt files. Anthropic’s current names are the three above. It doesn’t hurt to keep the old names in your robots.txt if you already have them.
How to verify them
Anthropic publishes the addresses its bots use. A request that says “ClaudeBot” but doesn’t come from that list is a fake. Fakes are common; a typical one looks like this:
45.148.10.12 "GET /.env" "Mozilla/5.0 (compatible; Claude-User/1.0; +Claude-User@anthropic.com)"Anthropic doesn’t probe for .env files, and the address isn’t Anthropic’s. Paste any visit into the bot checker, or see the current ranges on the ClaudeBot page. For the method in general, read how to verify any AI crawler.
Controlling them with robots.txt
Anthropic says its bots respect robots.txt, including the Crawl-delay rule for ClaudeBot.
Stay visible in Claude, but no training:
User-agent: ClaudeBot
Disallow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: Claude-User
Allow: /Keep Anthropic out completely:
User-agent: ClaudeBot
User-agent: Claude-SearchBot
User-agent: Claude-User
Disallow: /Slow ClaudeBot down instead of blocking it:
User-agent: ClaudeBot
Crawl-delay: 5Robots.txt only works for the real bots. To stop impostors, see fake AI bots and how to block them.
Is blocking ClaudeBot a good idea?
It depends on what you want. ClaudeBot is about training: blocking it doesn’t stop people finding you through Claude’s search, which uses Claude-SearchBot and Claude-User. If your content is your product (a publisher, a course, a paid database), blocking training is a reasonable choice. If you mostly want to be found and cited, allow the search and user bots at least. The trade-offs are covered in should you block AI crawlers?.
People who come from Claude
When someone clicks a link in one of Claude’s answers, they arrive with claude.ai as the referrer. See how to find AI traffic in your analytics to count them.
See Claude’s visits on your site
Create a free account to see how often each Anthropic bot visits, which pages it reads, how many of the “ClaudeBot” visits are fake, and how many people then arrive from Claude.
Check your own website
See which AI bots really visit your site — real or fake, page by page — and how many people AI assistants send you. Free.