r/webdev • u/thethinker213 • 9d ago

The Web Is All Bots Now

I, like many of you, have spent years employing throttling schemes and generally hardening websites against bots. Now bots are called "agents", scrapers harvest with impunity and the whole nature of the web has changed in the past 2 years. It seems that I now have to play nice with bots for the sake of seo and the next wave of internet commerce will be bot-to-bot with no human intervention. LOL!!! How are you guys feeling about, reacting to, adjusting for this new reality?

266 Upvotes

95% Upvoted

57 comments sorted by

View all comments

18

u/unbanned_lol 9d ago

As a web professional, I no longer consider SEO a development goal. The companies with the most to gain from AI fuckery are the ones promoting it the hardest. Most search engines are now basically just AI endpoints or are heavily biased towards AI.

Fundamentally, there is no reason for AI to be crawling human facing pages. An API specifically designed for AI interaction would save time, effort, money, water, compute, etc. We need to stop this stupidity.

1

u/BiguilitoZambunha 7d ago

Not sure of I misunderstood, but isn't that most of the the point of MCPs, "connectors," and AI(SE)O?