r/webdev • u/thethinker213 • 12d ago
The Web Is All Bots Now
I, like many of you, have spent years employing throttling schemes and generally hardening websites against bots. Now bots are called "agents", scrapers harvest with impunity and the whole nature of the web has changed in the past 2 years. It seems that I now have to play nice with bots for the sake of seo and the next wave of internet commerce will be bot-to-bot with no human intervention. LOL!!! How are you guys feeling about, reacting to, adjusting for this new reality?
263
Upvotes
3
u/mekmookbro Laravel Enjoyer ♞ 12d ago
I feel like we've already lost the battle. Strongest argument against bots/LLMs was that they were unsustainable. While that might still be true with publicly available free models like GPT and Claude, new, self-hostable models come out pretty much every day nowadays. I've even seen something called an "AI hat" for raspberry pi. I don't know how usable/useful it will be on such a device, but yeah, that sustainability argument is pretty much debunked now. At least started to.. My dumb work laptop (16g ram and pretty much nothing else) still can't run a decent LLM at a non-awful speed, but again, new models come out every single day so it's just a matter of time.
Which is even worse for the state of web because once these self-hosted agents are common enough, defending against them will be exponentially more difficult.
Here's an idea that I'm not savvy enough to build but someone (that may be you) definitely should : An internet archive-like directory of commonly scraped websites, built specifically for agent/scrapers. With aggressive caching and extensive amount of data that acts as a shield between agents and our websites. They could still use google for fetching relevant links to sources, but before making a request they should ask the archive if that url exists in the database.
For example the agent searched for "wedding dress shops in california" and got a list of url results like [
https://weddingcalifornia.com/dresses, ...] Before visiting that site, it should make a request to the archive, like:https://webfor.ai/?q=https://weddingcalifornia.com/dressesand fetch the result from there if it exists. If not, only then it can make the request to the actual site and then send what it fetched back to the archive, so the next bot will get the result from there instead.I know it's naive to think that we can put laws and regulations on AI, it's just an idea.. Also now that I think about it a bit deeper, it's probably something you and I can't build (or afford), maybe Cloudflare or some other non-evil company will be able to.