How to verify any AI crawler (GPTBot, ClaudeBot, Googlebot…)

To verify an AI crawler, check its IP address, not its name. The company behind the bot either publishes a list of the addresses its bot uses, or lets you confirm the address with reverse DNS, checked in both directions. If the address passes, the bot is real. If it fails, someone is borrowing the name.

You can do this in a few seconds with the free bot checker. The rest of this guide shows how it works, so you can do it yourself or build it into your own stack.

Why the user-agent isn’t enough

Every request carries a user-agent string, and the sender writes it. Nothing stops a scraper from sending Mozilla/5.0 (compatible; GPTBot/1.2; +https://openai.com/gptbot). Scrapers and vulnerability scanners do this all the time, because many sites let AI crawlers through.

So the user-agent tells you who a visitor claims to be. The address tells you who it is.

The two ways companies let you verify their bots

1. Published IP lists

Most AI companies publish a JSON file with the address ranges their bot uses. The format is the same almost everywhere:

{
  "creationTime": "2026-10-12T00:00:00.000000",
  "prefixes": [
    { "ipv4Prefix": "20.171.207.0/24" },
    { "ipv6Prefix": "2001:db8:1234::/48" }
  ]
}

Companies that publish lists include OpenAI (one list per bot), Anthropic, Perplexity (one per bot), Google, Microsoft (Bingbot), Apple (Applebot), DuckDuckGo, Mistral and Common Crawl. Each bot’s page in the AI bot directory links to its official list.

2. Reverse DNS, confirmed both ways

Some companies, notably Google, Microsoft, Apple and Amazon, also let you check the name behind an address:

1. Look up the reverse DNS name of the address. It must end in the company’s domain, for example .googlebot.com or .search.msn.com.
2. Look up that name’s forward DNS address. It must be the same address you started with.

The second step matters. Anyone who controls an IP address can set its reverse DNS to any name they like, including crawl-1-2-3-4.googlebot.com. Only the real company controls what googlebot.com names resolve to.

Step by step

1. Read the claim. Find the bot’s token in the user-agent: GPTBot, ClaudeBot, PerplexityBot, Googlebot and so on.
2. Find how that company lets you verify it. Use the AI bot directory, or the company’s own crawler documentation.
3. Check the address against that bot’s list, or with reverse DNS in both directions.
4. Decide:
   • Verified: the address is in the list, or the DNS check passed.
   • Fake: the company publishes a way to check, and the address fails it.
   • Name only: the company publishes nothing (Meta and ByteDance, for example), so the visit can’t be proven either way.

Checking an IP list by hand

A few lines of Python download a list and test an address:

import ipaddress, json, urllib.request

def in_list(ip, url):
    data = json.load(urllib.request.urlopen(url))
    addr = ipaddress.ip_address(ip)
    for p in data["prefixes"]:
        net = p.get("ipv4Prefix") or p.get("ipv6Prefix")
        if addr in ipaddress.ip_network(net):
            return True
    return False

print(in_list("20.171.207.12", "https://openai.com/gptbot.json"))

Use the list for that exact bot. OpenAI and Perplexity publish separate lists for each of their bots, and a ChatGPT-User request should come from the ChatGPT-User list, not the GPTBot one.

Checking reverse DNS by hand

On Linux or macOS, the host command does both lookups:

$ host 66.249.66.1
1.66.249.66.in-addr.arpa domain name pointer crawl-66-249-66-1.googlebot.com.

$ host crawl-66-249-66-1.googlebot.com
crawl-66-249-66-1.googlebot.com has address 66.249.66.1

The name ends in googlebot.com, and it points back to the same address, so this Googlebot is real.

Mistakes to avoid

• Checking the wrong address. Behind a CDN or load balancer, REMOTE_ADDR is the proxy, not the visitor. Use the header your proxy sets (for example CF-Connecting-IP on Cloudflare), but only trust it when the request really came from that proxy’s own addresses.
• Stale lists. Companies add and remove ranges. Download the lists every day, and if a download fails, keep yesterday’s copy rather than marking every real bot as fake. Recent changes are shown in the directory.
• Reverse DNS without the forward check. As explained above, the reverse name alone can be faked.
• Calling “name only” bots fake. If a company publishes no list and no DNS names, a failed check means nothing. Label these visits as unverified instead.
• Blocking by user-agent alone. A rule like “block everything that says GPTBot” blocks the real GPTBot along with the fakes. If you want to block fakes, block failed verifications. See how to block fake AI bots.

Doing it automatically

Checking every visit by hand isn’t practical. Two options:

• Paste any single visit into the free bot checker.
• Create a free account and add your website. Every AI bot visit is then checked against all the official lists and DNS rules, and labelled verified, fake or name only, page by page.