These are the AI crawlers and AI agents you are most likely to see in your logs, grouped by company. For each one: the token that appears in its user-agent (use it in robots.txt), why it visits, and how to verify that a visit is real. Each name links to its page in the AI bot directory, which shows its official IP ranges, updated daily.
Why it visits uses three groups: training (collects content for future models), search (indexes pages so an assistant can find and cite them) and live (fetches a page because a person asked, right now). See should you block AI crawlers? for what that means for you.
OpenAI (ChatGPT)
| Token | Why it visits | How to verify |
|---|---|---|
| GPTBot | Training | IP list |
| OAI-SearchBot | Search (ChatGPT search) | IP list |
| ChatGPT-User | Live (a person asked ChatGPT) | IP list |
More: GPTBot vs ChatGPT-User vs OAI-SearchBot.
Anthropic (Claude)
| Token | Why it visits | How to verify |
|---|---|---|
| ClaudeBot | Training | IP list |
| Claude-SearchBot | Search | IP list |
| Claude-User | Live | IP list |
| anthropic-ai, Claude-Web | Older names | IP list |
More: Anthropic’s bots explained.
Perplexity
| Token | Why it visits | How to verify |
|---|---|---|
| PerplexityBot | Search | IP list |
| Perplexity-User | Live | IP list |
More: PerplexityBot vs Perplexity-User.
Google (Search, AI Overviews, Gemini)
| Token | Why it visits | How to verify |
|---|---|---|
| Googlebot | Search, including AI Overviews | IP list or reverse DNS |
| Google-Extended | Not a crawler: a robots.txt token for Gemini training and grounding | Nothing to verify |
| GoogleOther | Other Google crawling | IP list or reverse DNS |
| Google-CloudVertexBot | Crawls for a site owner’s Vertex AI agent | IP list or reverse DNS |
| Google-NotebookLM | Live (a NotebookLM user added the page) | IP list or reverse DNS |
More: Google and AI crawlers.
Microsoft, Apple, Amazon, DuckDuckGo, Mistral
| Token | Company | Why it visits | How to verify |
|---|---|---|---|
| bingbot | Microsoft (Bing, Copilot) | Search | IP list or reverse DNS |
| Applebot | Apple | Search | IP list or reverse DNS |
| Applebot-Extended | Apple | Not a crawler: a robots.txt token for AI training | Nothing to verify |
| Amazonbot | Amazon | Search | Reverse DNS |
| DuckAssistBot | DuckDuckGo | Live (AI answers) | IP list |
| MistralAI-User | Mistral (Le Chat) | Live | IP list |
Open datasets and others
| Token | Company | Why it visits | How to verify |
|---|---|---|---|
| CCBot | Common Crawl | Training (an open web archive many AI models learn from) | IP list or reverse DNS |
| meta-externalagent | Meta | Training | Name only |
| meta-externalfetcher | Meta | Live | Name only |
| Bytespider | ByteDance | Training | Name only |
| cohere-ai | Cohere | Other | Name only |
| AI2Bot | Allen Institute for AI | Training | Name only |
| Diffbot | Diffbot | Other | Name only |
Name only means the company publishes no IP list and no DNS names, so a visit can’t be proven real or fake. Anyone can use these names.
How to use this list
- robots.txt: use the token exactly as shown. Ready-made setups are in how to block or allow AI bots in robots.txt.
- Verifying a visit: check the address with the method in the last column; see how to verify any AI crawler, or paste the visit into the bot checker.
- Blocking fakes: block failed verifications, not names; see fake AI bots.
Keeping up
New AI bots appear every few months, and companies change their IP ranges. The AI bot directory downloads every official list daily and shows recent range changes. To see which of these bots really visit your site, and how many are fake, create a free account.
Check your own website
See which AI bots really visit your site — real or fake, page by page — and how many people AI assistants send you. Free.