Diffbot · Diffbot

Diffbot

Crawls the web for other Diffbot products. Here is how to recognise it, verify it, and allow or block it.

Company: Diffbot
Product: Diffbot
Why it visits: Other crawling
How to verify: Name only
User-agent contains: Diffbot

How to tell a real Diffbot from a fake

Diffbot publishes no IP list and no DNS names for Diffbot, so a visit can’t be proven real. Anyone can put “Diffbot” in a user-agent. Treat it as unverified, and watch how much it crawls.

Check a Diffbot visit

Allow or block Diffbot in robots.txt

Block it everywhere:

User-agent: Diffbot Disallow: /

Allow it everywhere:

User-agent: Diffbot Allow: /

Fake bots ignore robots.txt: block them by address in your firewall instead.