The complete list of AI crawler user-agents (and how to verify each)
Every major AI crawler and AI agent — OpenAI, Anthropic, Perplexity, Google, Microsoft, Apple, Meta, ByteDance and more — with its user-agent token, purpose and how to verify it.
Read the guide 13 Oct 2026 · 3 min readHow to see ChatGPT, Perplexity and other AI traffic in your analytics
People now click through to websites from ChatGPT, Perplexity, Claude, Gemini and Copilot. How to find that traffic in Google Analytics 4, why some of it hides, and how to count it properly.
Read the guide 13 Oct 2026 · 3 min readHow to block or allow AI bots in robots.txt (copy-paste rules)
Ready-to-use robots.txt rules to block AI training crawlers, keep AI search visibility, or block AI bots completely, plus the limits of robots.txt.
Read the guide 13 Oct 2026 · 3 min readClaudeBot, Claude-User and Claude-SearchBot: Anthropic’s bots explained
Anthropic visits websites with three bots. What ClaudeBot, Claude-User and Claude-SearchBot each do, how to verify them, and how to control them with robots.txt.
Read the guide 13 Oct 2026 · 4 min readFake AI bots: how to spot them and block them safely
Scrapers and scanners often pretend to be GPTBot or ClaudeBot. How to recognise fake AI bots in your logs and block them without blocking the real crawlers.
Read the guide 13 Oct 2026 · 3 min readGoogle and AI: Googlebot, Google-Extended and Gemini explained
How Google’s crawlers relate to AI Overviews and Gemini, what the Google-Extended token really controls, and how to verify a Googlebot visit.
Read the guide 13 Oct 2026 · 4 min readGPTBot vs ChatGPT-User vs OAI-SearchBot: OpenAI’s three bots explained
OpenAI visits websites with three different bots. What each one does, how to verify it, and how to allow ChatGPT search while saying no to training.
Read the guide 13 Oct 2026 · 3 min readPerplexityBot vs Perplexity-User: what each one does
Perplexity visits websites with two bots. What PerplexityBot and Perplexity-User do, how to verify them with Perplexity’s IP lists, and what robots.txt can and can’t control.
Read the guide 13 Oct 2026 · 3 min readShould you block AI crawlers? A practical way to decide
Blocking every AI bot can cost you visitors; allowing all of them gives your content away. How to decide bot by bot, with recommended setups for different kinds of websites.
Read the guide 13 Oct 2026 · 4 min readHow to verify any AI crawler (GPTBot, ClaudeBot, Googlebot…)
A user-agent proves nothing. Here is how to check whether GPTBot, ClaudeBot, Googlebot or any other AI crawler is real, using official IP lists and reverse DNS.
Read the guide