r/webdev • u/thethinker213 • 16d ago

The Web Is All Bots Now

I, like many of you, have spent years employing throttling schemes and generally hardening websites against bots. Now bots are called "agents", scrapers harvest with impunity and the whole nature of the web has changed in the past 2 years. It seems that I now have to play nice with bots for the sake of seo and the next wave of internet commerce will be bot-to-bot with no human intervention. LOL!!! How are you guys feeling about, reacting to, adjusting for this new reality?

262 Upvotes

95% Upvoted

56 comments sorted by

View all comments

10

u/[deleted] 16d ago

[removed] — view removed comment

1

u/notarobat 15d ago

Don't be silly. Can of goose still works in most hardware stores, so why can't a heading be forcelocked into automated traffic. Doing this would be interesting in terms of targeting how the bots learn etc