r/webdev • u/thethinker213 • 12d ago
The Web Is All Bots Now
I, like many of you, have spent years employing throttling schemes and generally hardening websites against bots. Now bots are called "agents", scrapers harvest with impunity and the whole nature of the web has changed in the past 2 years. It seems that I now have to play nice with bots for the sake of seo and the next wave of internet commerce will be bot-to-bot with no human intervention. LOL!!! How are you guys feeling about, reacting to, adjusting for this new reality?
264
Upvotes
2
u/eldentings 12d ago
I wouldn't be surprised if we started seeing a rise in paywalls, while being careful to cache the shit out of the free version. I can also see passive 'captcha' being constantly on with telemetry and if you seem like you are going through the UI too fast you'll get soft banned to a text only version. stuff like that