How to block or allow AI bots in robots.txt (copy-paste rules)

To block an AI bot, add a group for its user-agent token to your site’s robots.txt with Disallow: /. For example, User-agent: GPTBot followed by Disallow: / asks OpenAI’s training crawler to stay away. Well-behaved crawlers follow these rules, but robots.txt is a request, not a lock: fake bots ignore it.

Below are ready-to-use setups, then the details that trip people up.

The basics in 60 seconds

• The file must be at the root of each host: https://example.com/robots.txt. A subdomain such as shop.example.com needs its own file.
• Rules are written in groups. A group starts with one or more User-agent: lines, followed by Disallow: and Allow: lines.
• A crawler uses the group that names it most specifically. If none names it, it uses the User-agent: * group.
• Disallow: / blocks the whole site. Disallow: /private/ blocks one folder. An empty Disallow: blocks nothing.

Setup 1: no AI training, but stay visible in AI search

This is the most common choice. Training crawlers are blocked; the bots that let AI search engines find and cite you are allowed.

# AI training: no
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: CCBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: meta-externalagent
User-agent: Bytespider
Disallow: /

# AI search and answers: yes
User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
Allow: /

Google-Extended and Applebot-Extended are special: they are tokens, not crawlers. Google and Apple crawl with their normal bots, and these tokens only tell them whether your content may be used for AI training (and, for Google, for grounding Gemini). Blocking them does not affect Google Search or Apple’s search features. See Google and AI crawlers.

Setup 2: block AI bots completely

User-agent: GPTBot
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: ClaudeBot
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: Perplexity-User
User-agent: CCBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: meta-externalagent
User-agent: Bytespider
Disallow: /

Think before you use this one. You will stop appearing as a source in ChatGPT search, Claude and Perplexity, and you lose the visitors those answers send. See should you block AI crawlers?.

Leave Googlebot, Bingbot and Applebot alone unless you also want to leave Google Search, Bing (and Copilot) and Apple’s search features.

Setup 3: keep AI out of some folders only

User-agent: GPTBot
User-agent: ClaudeBot
User-agent: CCBot
Disallow: /members/
Disallow: /courses/
Allow: /

Useful when your blog can be read by anyone but your paid content shouldn’t be collected. Remember that robots.txt doesn’t protect content: if a page is public, anyone who ignores the file can still fetch it. Put paid content behind a login.

Things that trip people up

• One group per crawler, or several names on one group. Both work. Listing several User-agent: lines above one set of rules (as above) keeps the file short.
• Names are matched loosely, but spell them right. Crawlers match their token case-insensitively; a typo like GPT-Bot matches nothing.
• A specific group replaces *, it doesn’t add to it. If you have User-agent: * with Disallow: /admin/ and a separate User-agent: GPTBot group, GPTBot follows only its own group. Repeat shared rules in it if you need them.
• Live, user-started fetchers may not obey. Bots that fetch a page because a person asked (ChatGPT-User, Perplexity-User) are treated differently by their companies; some say robots.txt may not apply to them. Check each company’s documentation.
• It takes time. Crawlers cache robots.txt, usually for up to a day.
• robots.txt can’t stop fakes. A scraper pretending to be GPTBot ignores your rules. Block those by address; see fake AI bots and how to block them.

Check that it works

1. Open https://your-site/robots.txt in a browser and make sure it shows your new rules.
2. Watch your logs, or your OZMAICO dashboard, for the next few days. Verified visits from a blocked bot should stop. If “GPTBot” keeps coming from addresses outside OpenAI’s list, those are fakes, and robots.txt won’t stop them.
3. Each bot’s page in the AI bot directory has copy-paste rules for that bot alone.