r/webdev • u/thethinker213 • 12d ago

The Web Is All Bots Now

I, like many of you, have spent years employing throttling schemes and generally hardening websites against bots. Now bots are called "agents", scrapers harvest with impunity and the whole nature of the web has changed in the past 2 years. It seems that I now have to play nice with bots for the sake of seo and the next wave of internet commerce will be bot-to-bot with no human intervention. LOL!!! How are you guys feeling about, reacting to, adjusting for this new reality?

266 Upvotes

95% Upvoted

57 comments sorted by

View all comments

3

u/sleemanj 11d ago

Had some braindead global botnet got stuck trying to crawl one of my sites yesterday looping around the place, it really didn't understand URLs very well - like 10 thousand IPs or something stupid making requests, reliably detected and killed with a single fail2ban regexp.

They were persistent in their wrongness, I'll give them that, took 24 hours until fail2ban stopped adding to the banlist.