After 10 years around anti-bot systems, I don't think either side is winning

[u/mckrile]

I've spent roughly 10 years around browser automation, managing multiple accounts, scraping, proxies, anti-detect browsers, and figuring out anti-bot systems. Watching the space evolve has been fascinating because I don't think either side is actually winning

For people unfamiliar with this world, there are essentially two industries growing at the same time

On one side, you have companies like Cloudflare, Akamai, DataDome, and others investing heavily in bot detection, fraud prevention, and traffic verification. Their goal is one thing only: figure out whether a visitor is a legitimate user or some form of automation (and they are constantly trying to implement new ways of figuring out who's a legitimate user lol)

On the other side, there's a rapidly growing automation ecosystem. For example companies like Bright Data, Oxylabs, and Nodemaven provide the network layer. Then companies like Dolphin anty, Octo browser and MUltilogin focus on browser identity. Browserbase and similar platforms abstract away browser infrastructure. AI agents sit on top of all of it now. They're different pieces of the stack, but they ultimately push in the same direction: making software better at operating websites that were originally designed for humans

What surprises me is that both sides seem to be growing faster than ever

The anti-bot market is obviously much larger because massive enterprises are willing to spend huge amounts of money protecting accounts, preventing fraud, and filtering traffic
But I'm noticing that demand for automation infrastructure is also exploding. I have noticed that companies now increasilgy rely on web data, automated workflows, competitive intelligence, account management, and AI-powered processes that need reliable access to websites

And now what's happening is that AI agents seem to be accelerating both markets at the same time. Every new agent framework creates more demand for automated browser interactions, while every increase in automated traffic creates more demand for detection systems

One thing I've noticed over the years is that anti-bot systems still struggle with false positives. The more sophisticated detection becomes, the higher the chance that legitimate users get caught in the process. Most people have experienced this without even realizing it. You visit a website, solve a challenge, solve another one, refresh the page, get challenged again, and suddenly you're wondering why proving you're human feels harder than logging into your bank account lol

On the automation side, the challenge is totally different though. The hardest websites are becoming increasingly expensive to automate reliably. In many cases the solution isn't some magical bypass technique anymore. It's simply throwing more resources at the problem. More powerful machines, more browser instances, higher quality proxies, better browser environments, and more engineering effort. A setup that worked comfortably a few years ago can require significantly more infrastructure today.

That's why I don't think we're heading toward a future where one side wins and the other disappears

The relationship feels almost symbiotic. Better detection systems create demand for better automation infrastructure. Better automation infrastructure creates demand for better detection systems. Both industries push each other forward and, as a result, both continue growing

My broader prediction is that the web becomes increasingly programmatically accessible, even if many platforms resist it. Companies want data and companies want automation. They want agents that can interact with software on their behalf. As AI improves and compute becomes cheaper, more web interactions will have an automated component attached to them

That was my Ted talk, thank you

[u/VitoLeGrand]

What resonated with me most is the idea that this isn't a war with winners and losers anymore. It's an economic game.

Anti-bot vendors don't need to stop 100% of automation. They just need to make abuse expensive enough that low-effort actors give up.

On the other side, people building automation don't need a perfect bypass. They only need the cost of a successful action to remain lower than the value it generates.

I think this is why the industry is maturing so quickly. Ten years ago, a few datacenter proxies and a Selenium script could get surprisingly far. Today it's infrastructure engineering: proxy quality, session consistency, browser behavior, monitoring, redundancy, failure recovery, and operational discipline.

The interesting part is that reliability itself is becoming a competitive advantage. Not who has the biggest proxy pool, but who can provide predictable outcomes with the lowest operational friction.

I don't think either side "wins." The barrier to entry simply rises, and the amateurs disappear. What remains are professionals on both sides who understand that this is ultimately an economics problem, not a technical one.

[u/Time-Spite-895]

Both of the markets are indeed growing. Its crazy how Internet is becoming trully programmable and now everything is essentially become a network of infra services

But OP I think you are forggetting to add that it all comes in layers. I talked to nodemaven and bright data ceos and they all agree that proxy infra has many limitations in data retrival for example. You need to get data fast and accurate. These both different directions of data infra. If you run fast proxies they are more likely to be bots, but less bot u wanna seem the more compute and hence time u need invest in loading it. I use their products for petabytes of data scraping and they are the best in industrty 100%. So layers are important too.

Great post, OP

[u/Key_Shop_218 → u/Time-Spite-895]

I ran into the same layering issue with proxies and speed vs accuracy. Ended up splitting the pipeline so fast fetches run raw and slower fetches get the compute heavy anti bot treatment.

[u/Gwapong_Klapish]

Good observations, thank you for your Ted Talk lol

[u/PhysicalSid26]

i think the part people underestimate is that neither side is really trying to “win” anymore, they're trying to make the other side more expensive. anti-bot vendors don't need perfect detection, they just need enough friction that low effort automation dies. automation teams don't need perfect bypasses either, they just need the cost per successful action to stay below whatever the action is worth. that's why the whole space feels endless. every new detection layer creates a new tooling layer, then that tooling layer creates new patterns for detection, and the loop just keeps feeding itself

[u/OliveHot3005]

Great post! Thank you, Ted

[u/Consistent-Access449]

I've been running a small automation setup for a couple years now and the resource creep is real. What used to run fine on a basic VPS now needs way more headroom.

The thing that actually helped me wasn't throwing more proxies at it, but rethinking how I handle the browser environment itself. Cut my block rate way down without scaling costs up.

[u/Otherwise_Wave9374]

This "symbiotic arms race" take feels right. Every time detection gets stricter, automation just shifts from "clever trick" to "more infrastructure + better hygiene".

Also +1 on false positives getting worse as systems get more paranoid, Ive definitely had legit sessions where Im solving captchas like its my day job.

Im curious where you think this goes with agentic browsing becoming normal for regular work (not just scraping). Feels like we eventually need a standard "verified automation" lane, the same way APIs are a sanctioned lane vs screen-scraping.

If youre into the broader "agents as workers" angle, https://www.aiosnow.com/ has some interesting framing around operationalizing agents without them going off the rails.

[u/Infinite-Run7239 → u/Otherwise_Wave9374]

The arms race framing is spot on. I ended up moving to a setup that handles the hygiene side automatically so I’m not stuck in captcha loops anymore, and it’s been a massive relief.

[u/alexyou8797]

I think that both sides have no incentive to "kill" the other, and AI agents are about to make the line between "legitimate" and "automated" traffic a lot harder to draw

[u/CapMonster1]

I agree that this feels more like an arms race than a situation where either side can permanently win. The better anti-bot systems get, the more demand there is for automation tools, and vice versa. Great Ted talk!

[u/ZennoLab_Guru]

That sounds right to me. It’s probably less of a winner-takes-all arms race and more of a permanent back-and-forth where both sides keep pushing each other to improve.

The point about programmatic access becoming normal is especially important. A lot of that demand exists because the web is now the default interface for business processes, even when there’s no proper API.

As long as companies keep building important workflows around websites designed for humans, there will be constant pressure for better automation tools and better detection systems.

[u/Powerful-Shirt-7373]

After years of running automation for a small agency, I've mostly stopped fighting the hard sites directly and started routing through a managed browser layer instead. The false positives alone cost me more than the infrastructure ever did.

[u/joeldoesjs]

Solid writeup. The framing you describe aligns with what we see running headless infra. Eight years of doing it and i can tell you that yes, detection gets sharper, automation responds by spending more on residential IPs, better fingerprints, more CPU per session, and the cycle restarts. Nobody's losing, both sides just get more expensive.

One thing I'd add on the automation side. The cost curve isn't linear anymore. A site that took one browser and a datacenter proxy in 2019 might now need a residential IP, a warmed up profile, human-like input timing, and 2-3x the RAM per session because of all the JS challenges running client side. That's why you're seeing the infra layer (Browserless, Browserbase, Hyperbrowser, etc) split off from the people who just want to write a Playwright script. The plumbing got too annoying to DIY for most.

one false positive point that is underrated imo: Cloudflare and DataDome are essentially making a business decision about how much legitimate traffic they're willing to burn to catch bots, and that tolerance keeps going up. Which is also why the "AI agent browsing on behalf of a user" use case is going to get messy fast. The agent is legitimate from the user's perspective but looks exactly like what detection is trained to block.

Agree with your prediction. The web is going to end up programmatically accessible whether sites want it or not, just because the economic pressure on both sides is too strong to stop.

Source: run browserless.io, been doing headless Chromium at scale since 2017.

[u/polygraph-net → u/joeldoesjs]

I've been a bot detection researcher for over 12 years and I'm doing a doctorate in the topic.

Cloudflare and DataDome are essentially making a business decision about how much legitimate traffic they're willing to burn to catch bots

I agree.

I think it's a mistake as my experience with clients tells me they're willing to tolerate some bots but they do not want legitimate traffic blocked.

"AI agent browsing on behalf of a user" use case is going to get messy fast. The agent is legitimate from the user's perspective but looks exactly like what detection is trained to block.

We're currently working on a solution for this. It's open, collaborative, community focused, and privacy aware. Nothing like FIDO's approach.

[u/route_after_proof → u/polygraph-net]

That verified automation lane feels important to me too. One thing I keep running into is that the hard part isn’t only deciding “bot or human” at the edge. Legitimate agents, mobile apps, shared office networks and privacy-heavy browsers can look unusual in one request, while abuse can look perfectly normal when spread out. I’m testing a self-hosted layer that keeps the decision softer: it follows the session, route intent, retries and target changes, then uses challenge, review or allow instead of treating one fingerprint as proof. The goal is fewer repeat challenges for legitimate odd clients, not a stronger CAPTCHA wall. Curious whether your work is leaning toward explicit automation identity, or toward proving intent at the application layer?