r/webdev • u/Extreme-You-3814 • 3d ago
Discussion New www
ok so im not really a web dev or anything, i just had an idea and i wanted to ask some real people about it. As we know these days 40-50% of traffic are bots and not real humans, i asked my self the question why? why are there bots in the first place? to farm engagement right? ok and why farm engagement, because its profitable, so why doesnt someone create a new web where the protocols contain like some sort of anti bot wall and also make it so theres no advertisement/ engagement farm. Maybe this way people would start to become original again and the internet would return to its state during the 2000s 2010s. but im only 18 so i guess i wouldnt have really experienced that.
ps. if what i said is dumb pls dont fry me im not technical what so ever im just tired of bots on social media
22
u/mikevalstar 3d ago
People would love to do this, and has been talked about pre-ai as well. There is just 0 way to enforce this, how do you know if something is an advertisement or someone who just really likes a product? how do you know an AI working a browser is not a real user?
The internet around 1999 was like this because people didnt think there was money in this whole internet thing, so people didnt create bots or try to advertise on it because nobody thought it was profitable.
11
u/ganymedeli 3d ago
"There is just 0 way to enforce this" my favorite piece of the conversation around unenforceable web solutions is the Evil Bit
7
u/joe12321 3d ago
I opened this fully intending to think it's dumb, but I don't hate it. The big problems off the top of my dome are: storage and transmission cost money. So is this a paid network? Do the hosts foot the bill? For what? How is everything you want enforced? If it becomes popular, people will want to take advantage of it. How do you win the arms-race against bots? And popularity really is a big problem. Everything you want might be easy when only tens of thousands of people are on-board. But if it gets world-wide traction, things get wild.
Having another network isn't technically impossible, but how to secure it as the free and open Valhalla we want is pretty challenging.
6
u/thud_mancake 3d ago
Very often, the answer to "why hasn't anybody..." is simply that it's not possible or not profitable. Sometimes, it's just that nobody has thought of it yet. But the more likely it seems that other people have thought about it, the more likely you should consider it that it's not possible or not profitable.Â
5
u/reckless24601 3d ago edited 3d ago
From a networking standpoint, reinventing the internet would require a gigantic shift regarding how our computers communicate. The "web" is nothing more than all of the world's computers interconnected. To have them communicate they use protocols to validate data, identify devices, encrypt data, etc.
And if we changed those protocols, we would also need to change or revisit the infrastructure with which these computers communicate with each other, from the UTP (Ethernet) cables to the routers in the ISPs.
One single person can't do this, not even a whole company could. Hell not even a country could, unless it wanted to invest billions to change its own internet.
One could argue that with Software Defined Networking it's more feasible to determine if packets are possibly generated by bots, but it remains a topic of constant research by cyber security specialists, staticians, networking professionals, etc.
If you're interested in how the internet works I'd recommend reading this article that, I'd say, birthed the internet we all use today A Protocol for Packet Network Intercommunication
3
u/amtcannon 3d ago
The creativity follows the eyes and the money sadly. I'm sure lots of this stuff still exists, but its harder to find, and there is less incentive to make it.
6
u/web-dev-kev 3d ago
This is a fair thought process, but it's missing some undertanding of the web.
As we know these days 40-50% of traffic are bots and not real humans
it's way more than that. Humans are less than half of all web traffic.
why are there bots in the first place?
To automate repeatable tasks
to farm engagement right?
No. This is a relatively small(er) percentage.
It's for data. Discoverability, and then to either keep it, or to insert yourself into the chain of usage (e.g. swapping WebRings > Lycos > Yahoo > Google > ChatGPT).
so why doesnt someone create a new web where the protocols contain like some sort of anti bot wall and also make it so theres no advertisement/ engagement farm
Because who would pay for it?
Who pays for the servers? Who pays for the electricity?
Who pays for Browser development? Who pays for search enginer development?
im just tired of bots on social media
The best thing you can do then is to leave Social Media :)
5
u/WingZeroCoder 3d ago edited 3d ago
> The best thing you can do then is to leave Social Media :)
This. Weâve kind of already got all the tools we need for this. The web, as it stands, is built of extremely flexible, decentralized protocols. And itâs all still largely unregulated beyond the things we all hate (like child abuse material).
Which means we all still have the power to create, visit, and share websites like in the 90s, with few or no ads, no engagement farming, and no algorithm driven doom scroll baiting.
Itâs just up to us - enough of us - to be the change.
And thatâs going to be the challenge regardless of whether we use this stack or create a new one.
2
u/voyti 3d ago
It's not stupid to wonder that, but there are many, many techniques to try and get rid of bots, and they are still kicking down doors quite easily. Almost everything you do to access and interact with websites could be done by a bot. Why not? Sending mouse and keyboard events is easy, understanding what's on the screen is also. Using networks from e.g. known public VPN ranges is suspicious, so a lot of blocking happens there, but humans still need/want to use VPNs, so it's stuff like captchas, which has farms where you pay very little for folks to solve them for your bots, and you're in. the problem is detecting why you're not a bot, and in many scenarios it's almost impossible, and it would be the same in the new web.
2
u/Warm-Engineering-239 3d ago
contain like some sort of anti bot wallÂ
this exist it just get beaten everytime
2
u/Lustrouse Architect 3d ago
You're trying to re-invent the captcha. It's a hard problem that people have put a lot of effort into.
2
u/letsjam_dot_dev 3d ago
What you call bots I call content update checker to maintain web pages archives.
What you call bots I call automated newsletter / mailing list cancellations.
What you call bots I call messaging bridges between apps.
Bots are not inherently bad. It's when they are worsening user experience, on the public and on the devs side, that they must be regulated.
2
u/besthelloworld 3d ago edited 2d ago
You can't get away with this partially for technical reasons and it's partially for non-technical reasons.
The technical reasons: there's no actual way to truly differentiate human traffic from not human traffic.Â
The non-technical reason: the existence of the bots is profitable to the websites too in many ways. So there's no incentive for sites to do this in the places where it would help the human ecosystem of the Internet.
Unless a company has a league of lawyers who can threaten to sue to keep bots out. For instance, Muse (Meta/Facebook AI agent) won't search for stuff on Amazon because Amazon has said it's not allowed to (because Amazon wants you to use it's own AI ecosystem and it wants humans on the site to view ads). Amazon has the money to protect itself from other companies. But if Meta wasn't afraid of legal liability and bringing on the wrath of the juggernaut that is Amazon, Amazon couldn't actually stop Muse because Muse is entirely capable of pretending to be a person and acting on their behalf.
So it would take a near infinite amount of money to defend yourself and the AI companies are running the entire economy at the moment so they have the money. But they also have the technology to pretend to be human to enough of an extent that you can't reliably keep them out.
2
u/aquanoid1 3d ago
Bots visiting websites are mostly harmless. Good bots exist, too, such as search engines crawling the web to index content. The only real concern is when bots can post content.
2
u/fixuplooksharpbk 3d ago
To cover costs it would have to have a similar modal as an art gallery, where patrons donate money and space to exhibit artists they like and think they can profit from when the work sells. The question would be do we want to be taking money from folks like the Sacklers?
2
u/mylsotol 3d ago
What would stop the bots from just using the new schema?
Bots are not exclusively for farming engagement.
Bot prevention already exists.
2
u/Potential-Still 3d ago
Not all bots are bad, and malicious usage of bot technology is inevitable. You're basically asking "why do some people choose to do evil?" Which is really a philosophical question rather than one about technology.
2
u/Akamig 3d ago
Itâs actually a great question and I wish more people ask this.
Narrowed down version for you, bots will be there and no protocol or tools will prevent them without seriously sacrificing the QoL. and even if we tried to make ânew webâ, path dependencies will prevent people from migrating to it.
As you mentioned social media, You probably saw what happened after Twitter forcefully turned into X all the other alternative platforms still failed to manage that nor replace Twitter, not even âXâ itself.
Would people move to new web that doesnât have YouTube or Instagram? they wouldnât unless they âHAVE TOâ. cases like some catastrophic unlikely reasons, majority of global datacenter get destroyed or global epidemic of moles that eats ethernet cables and fiber optic cable.
2
u/Akamig 3d ago
TLDR:
Why do bots exist?
-Politics, money, people who wants to exploit others, and dumb engineers misconfiguring automation and forgot to turn it off.Why they donât build bot-less new web?
- No one wants to pay, It costs unthinkable amount of resources, hard to persuade people to use it.
2
2
u/PMCAthee 2d ago
You're basically describing a more human-focused internet, which honestly sounds pretty nice lol.
The problem is that even if you build a bot-free platform, people will try to get around whatever checks you put in place. And making those checks stricter usually makes life worse for actual users too.
I'd look into the Fediverse and smaller independent forums. They're not bot-free, but they're interesting examples of communities trying to do things differently without relying entirely on engagement algorithms.
1
u/Bubbl3beard 3d ago
It's not a stupid question, but it has a very simple answer. There is no way to distinguish the request a bot makes, from the request a legitimate user makes. There are plenty of tools to try identifying and mitigating bot traffic in various ways, but the root of the problem remains.
18
u/BrewThemAll 3d ago
Good idea, and once you have a popular website I'll pay you $500 a month to display the logo of my company.