Don’t decide for “AI bots” as a whole. Split them into three groups and decide for each: training crawlers (which collect content for future models), AI search crawlers (which let assistants find and cite you), and live fetchers (which open a page because a person asked). Most businesses that want visitors allow the last two and decide on the first. Publishers whose content is the product often block training.
The three kinds of AI bots
| Group | Examples | What you give | What you get |
|---|---|---|---|
| Training | GPTBot, ClaudeBot, CCBot, Google-Extended, Applebot-Extended, meta-externalagent, Bytespider | Your content, for future models | Little directly; maybe models that know your brand |
| AI search | OAI-SearchBot, Claude-SearchBot, PerplexityBot (and Googlebot, Bingbot for Search) | Your content, to be indexed | Citations and links in AI answers, and visitors from them |
| Live fetchers | ChatGPT-User, Claude-User, Perplexity-User, MistralAI-User | One page, for one person’s question | A person reading about your page right now |
The AI bot directory lists every bot with its group.
Questions to ask
- Is your content the product? News, courses, research, a paid database: training crawlers take value without sending visitors back. Blocking them is reasonable.
- Do you want to be found and recommended? Shops, SaaS, local businesses, docs: being cited by ChatGPT, Claude and Perplexity brings people who are already looking for what you do. Allow the AI search and live bots.
- Can your server take it? Some crawlers fetch a lot. If load is the problem, slowing a bot down (or blocking just the heaviest) may be enough.
- Are the visits even real? Before deciding based on log volume, check how many of the “AI bots” are fakes. Often a large share is. See fake AI bots.
Recommended setups
Business that wants customers (shop, SaaS, services, local business): allow everything. If you don’t want to feed training, block only the training group:
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: CCBot
User-agent: Google-Extended
User-agent: Applebot-Extended
Disallow: /Publisher or paid content: block training; allow AI search so you’re cited and linked; keep paid pages behind a login (robots.txt doesn’t protect them).
Documentation or open-source project: usually allow everything. You want AI assistants to answer questions about your product correctly.
Private or internal site: don’t rely on robots.txt at all. Put it behind authentication.
All the copy-paste rules are in how to block or allow AI bots in robots.txt.
What blocking won’t do
- It won’t remove what was already collected. Blocking affects future crawling.
- It won’t stop fakes. Scrapers using a bot’s name ignore robots.txt; you block those by address.
- It may not stop live fetchers. Some companies say their user-started fetchers don’t follow robots.txt, because a person asked. See PerplexityBot vs Perplexity-User and OpenAI’s bots.
- It won’t affect Google Search, as long as you leave Googlebot alone. Google-Extended is separate; see Google and AI.
Decide with your own numbers
Opinions are cheap; your logs aren’t. Before changing anything, look at:
- Which AI bots really visit (verified), and how many are fake.
- Which pages each one reads.
- How many people arrive from ChatGPT, Claude, Perplexity, Gemini and Copilot, and on which pages.
If AI search sends you real visitors, blocking it costs you customers. If a training crawler reads thousands of pages and nobody ever arrives from that company, blocking it costs you nothing.
OZMAICO shows all of this for free: verified vs fake bots, per page, next to the people AI assistants send you. Run it for a couple of weeks, then decide.
Check your own website
See which AI bots really visit your site — real or fake, page by page — and how many people AI assistants send you. Free.