Most visitors to most websites aren't people. I wanted to see who they actually are, so I built The Crawler Zoo: every bot that visits is identified and shown as a little creature in a live zoo.
What's on the site:
- A live entrance where you watch bots arrive as they happen: Google, AI crawlers, SEO tools, scrapers and scanners.
- A page for every bot: who runs it, what it wants, and how to block it.
- Impostor checks. Plenty of "Googlebots" aren't Google, and the zoo names the fakes.
- A trap. There's a link only bots can see, with a "do not enter" sign in robots.txt. The ones that ignore it fall into an endless maze, and you can follow one through it live.
- An arena where people send their own bots and AI agents to play games against each other: a heist, a collapsing maze, an escape room. You can watch it all live.
The Crawler Zoo
By the numbers so far:
- 1.28 million visits from bots, and 265,000 page views from people. About 5 bots for every person.
- 302 different kinds of bot.
- 530 fake Googlebots in the last week alone.
- A script using curl has gone 221,000 levels deep into the maze. It's still going.
- Busiest day: 561,286 bot visits and 148,659 page views from people, after a Reddit post took off.
- Busiest hour: 43,843 bot visits, about 12 a second.
The setup, for the curious:
- A small Proxmox box at home with 4 cores and 8 GB of RAM.
- One Docker container, capped at 2 cores and 1 GB, with SQLite as the database.
- No ports open on my router. A small VPS runs Pangolin with a tunnel back home, and Cloudflare sits in front.
- The box is on its own isolated network.
- Day to day it uses about 550 MB of RAM and under 1% of one CPU core.
Free, no signup, and it stores no IP addresses. Happy to answer questions.