ByteDance · ByteDance (Doubao)

Bytespider

Collects pages to train ByteDance’s AI models. Here is how to recognise it, verify it, and allow or block it.

Company: ByteDance
Product: ByteDance (Doubao)
Why it visits: Collecting for AI training
How to verify: Name only
User-agent contains: Bytespider

How to tell a real Bytespider from a fake

ByteDance publishes no IP list and no DNS names for Bytespider, so a visit can’t be proven real. Anyone can put “Bytespider” in a user-agent. Treat it as unverified, and watch how much it crawls.

Check a Bytespider visit

Allow or block Bytespider in robots.txt

Block it everywhere:

User-agent: Bytespider Disallow: /

Allow it everywhere:

User-agent: Bytespider Allow: /

Fake bots ignore robots.txt: block them by address in your firewall instead.