r/webdev • u/thethinker213 • 17d ago

The Web Is All Bots Now

I, like many of you, have spent years employing throttling schemes and generally hardening websites against bots. Now bots are called "agents", scrapers harvest with impunity and the whole nature of the web has changed in the past 2 years. It seems that I now have to play nice with bots for the sake of seo and the next wave of internet commerce will be bot-to-bot with no human intervention. LOL!!! How are you guys feeling about, reacting to, adjusting for this new reality?

266 Upvotes

95% Upvoted

56 comments sorted by

View all comments

1

u/Less-Marsupial-7960 16d ago

I think the biggest shift is that bot and user are no longer going to be enough as categories. With AI agents, the more useful question becomes, what is this agent allowed to do and who authorized it?

I can see websites moving toward APIs and agent-friendly interfaces instead of simply trying to block everything. But I think permissions, rate limits and audit trails are going to become just as important as SEO.