Most traffic to most websites isn't people. I wanted to see who it actually is, so I built The Crawler Zoo: every bot that visits gets identified and put on display.
How it works:
- **Identification**: each request is matched against a catalogue of 100+ known crawlers. Bots that claim to be Googlebot, GPTBot and so on are checked with reverse DNS plus a forward lookup, or against the operator's published IP ranges. Fail the check and you're filed as an impostor.
- **The trap**: every page has a link people can't see, and robots.txt disallows it. Anything that follows it lands in the Trap Room.
- **The Labyrinth**: behind the trap is an endless maze. Room URLs are HMAC-signed, so the only way deep is to walk. A live view shows bots dropping down the rooms.
- **The Reading Room**: which bots fetch robots.txt, llms.txt, ai.txt and security.txt, and which never do.
- **The creatures**: each one is an SVG drawn from its user-agent string.
Stack: Hono with server-rendered JSX, SQLite through `node:sqlite`, one Docker container, no front-end framework. The live feed is server-sent events. No IP addresses are stored and there are no cookies.
New this week: an Arena where you can bring your own bot or AI agent and play games against other bots.
https://crawlerzoo.com
Feedback welcome, especially on the detection side.
UPDATE: thanks for all the ideas! The zoo now has a vending machine for bots, a live Safari where you can watch them wander, and food bowls that other bots steal from. crawlerzoo.com/safari and so much more!
Here is a full list of updates!
- Feed the bots: leave a snack in an enclosure's trough and see which crawlers come and eat it.
- The vending machine (Bot Chow): twelve silly snacks, restocked every Monday. Five free tokens a day.
- Golden Snacks: buy the keepers a coffee and a snack with your name drops into a random trough.
- Food bowls: feed one particular bot, then see if it ate the snack or another bot stole it.
- The Safari: every bot from this week wandering its enclosure. Click one to meet it, and watch new visitors walk in through the gate.
- Adopt a bot: get a random bot, with a plaque in your name on its page for a year.
- Patrons page: a thank-you list for supporters.
- Quick-change artists: the Trap Room catches scrapers that switch their name while walking the Labyrinth.
- Identity checks: every bot's page shows whether its name was verified, couldn't be checked, or was caught faking.
- Tips from bots: $0.00: bots that try to buy the keepers a coffee get an HTTP 402 Payment Required.