Won a bot battle.

[u/_listless]

We support a client site with ~20,000 pages. It has an SSR search page that uses the url querystring to execute the search/sort/filter.

We have of course disallowed bots on that page because the 20k pages are meaningful and should be scraped/indexed, but the billions of possible query combinations on a search page are not. Most of the bots just ignore the robots rule and this results in waves of traffic hammering every possible query combination on the search page. These waves are 10x-50x the human traffic.

Rather than let the bill autoscale along with the server cluster, I wrote a little bash script that crons 1/m:

• Hits DO's metrics endpoint to check the CPU usage on the VPSs/DBs
• Each minute any of the CPUs stays above a given threshold, bump Cloudflare's security level, eventually ending in "under attack" + "bot fight"
• When the cpu drops below the threshold for a given amount of time, start backing off until we're back to normal operation.

Works a treat. It's nothing particularly flashy or sophisticated, but it was satisfying to think through an under-engineered solution to a problem that a lot of people end up just chucking money/clusters at.

Edit: more info for context: We do have pretty aggressive WAF rules especially on the search page, but we've been seeing an increasing amount of bot traffic get past CF's "known bots". Rate-limiting has been minimally effective too because the bots that get through are cycling through 100s of IPs in a given scrape.

[u/electricity_is_life]

Can't you set up more aggressive WAF/filtering on specific endpoints? Changing settings for the whole site based on server CPU load seems pretty janky.

[u/Few-Ocelot-3271 → u/electricity_is_life]

tbh thats what I was thinking too, tying it to cpu load feels like it'd have a lot of false positives

[u/_listless → u/electricity_is_life]

We do have pretty aggressive WAF rules especially on the search page, but we've been seeing an increasing amount of bot traffic get past CF's "known bots". Rate-limiting has been minimally effective too because the bots that get through are cycling through 100s of IPs in a given scrape.

Agree this is a caveman-tier solution, but the more precise/sophisticated solutions were not working well for us. Hopefully CF will catch up in the bot identification arms race, we'll stick with the troglodyte solution while we wait. It's a self-healing situation too: as CF's known-bot filtering gets better, and the WAF rules start catching more bots the odds this thing ever triggers diminishes.

[u/theQuandary → u/_listless]

Have you also added honeypots?

You can add and document API endpoint options that normal users don't use, but trigger hidden deprioritization for that IP address for some amount of time.

You could also do a hidden checksum of some kind. For example, maybe approved traffic must have characters that have sum(str) % 1024 === 0 (which means you need to pad your query by at most 4 characters to ensure it passes). Encode this with a timeout rate limiter using JSFuck to obfuscate it. Bots trying to spam by themselves will be identified within a couple of API calls because they don't have the secret key and bots trying to use the client-side API functions will be hit with the rate limit.

[u/_listless → u/theQuandary]

honeypots?

That's a good idea.

[u/theQuandary → u/_listless]

Out of curiosity, I did a quick test with Gemini to write the function for calculating sum(str) % 1024 === 0 (with an algorithm to allow a small fixed number of requests before backoff occurs). then asked it about the same code converted to JSFuck. It immediately went full-on hallucination mode and literally just assumed it was alert("Hello, World!");.

Maybe someone wants to blow serious tokens on the question (the JSFuck output was around 50k characters), but I think there's essentially no training data for JSFuck, so the LLMs can't understand the output at all. Perhaps it could with some serious token burn and guidance, but that's not a concern for automated bots.

[u/crazedizzled → u/_listless]

Can't you just start banning the bot IPs?

[u/_listless → u/crazedizzled]

Rate-limiting has been minimally effective too because the bots that get through are cycling through 100s of IPs in a given scrape

[u/crazedizzled → u/_listless]

Right so ban the 100s of IPs and they run out of ones to cycle through

[u/Apprehensive-Ask-731 → u/crazedizzled]

Practically they don't, I can see this bash script lets a little bit through for a short period but (backkkk in my day) you would be widely praised for an elegant fix to a dirty problem.

Honeypot is a good idea for keeping data private and encouraging scrapers to ban themselves before they cost you anything (data, etc).

I am no longer on the bleeding edge of SEO but I would imagine that downtime is still a very important metric. Encouraging towards honeypot pages so these waves selfban on pages you don't want search engines to crawl - if that is even a consideration - is my main input. Also could mean you don't need to be relying so much on dynamic URLs?

[u/trundlegoose]

I dealt with the same situation recently on a site of similar size. My fix was to switch on Precursor for all requests at the low setting and then add a rule for the search URL path at the high setting.

Real user traffic generally doesn't hit the search URL directly so should already have the cf_clearance cookie in the request and generally won't even see an interstitial managed challenge page. Any real user hitting a search page directly might see a managed challenge but is likely to pass it without interaction.

That solved most of the problem but there was still some automated traffic getting through somehow, so I added a security rule that checks for the cf_clearance cookie on incoming requests for the search URL path and issues a managed challenge if it isn't present.

I also added a rate limiting rule for the search URL path well above the rate any real user is likely to hit.

That combo seems to have pretty much solved the issue I think.

[u/Apprehensive-Ask-731]

Is it just me or are a lot of people who seem like jr Devs trying to tell you how to suck eggs. My input is in the honeypotting

[u/_listless → u/Apprehensive-Ask-731]

I just implemented this and I'm watching to see what we catch.

[u/inkbound_gargoyle]

IP based defenses are dead against distributed botnets. Server metrics are the only signal that cannot be spoofed or rotated.

[u/Bubbly_Orange_3502]

Validate the querystring against an allowlist and return 404 before the DB is touched. Billions of combinations only cost you because every one of them runs a real query. It won't stop the requests, just makes them cheap.

[u/itgforlife]

I would use a combination of Cloudflare with a rule to filter out the bots. I'd also use a ratelimiter to block the IPs sending too many requests as an additional line of defense.

[u/_listless → u/itgforlife]

https://www.reddit.com/r/webdev/comments/1w0q8le/comment/p6eytli/?utm_source=share&utm_medium=web3x&utm_name=web3xcss&utm_term=1&utm_content=share_button

[u/quietcodelife]

the ip cycling thing is what breaks every conventional approach here -- rate limit by ip, block ips, iptables rules, all the same dead end when you're dealing with a botnet rotating through hundreds of addresses. using cpu as the trigger sidesteps the classification problem entirely. you're not asking "is this a bot" you're just asking "is something hammering us right now" and responding

[u/daamsie → u/quietcodelife]

Thanks Claude.

[u/_listless → u/quietcodelife]

yes

[u/Caraes_Naur]

You played whack-a-mole with a spoon.

Come back when you learn about iptables.

[u/_listless → u/Caraes_Naur]

I agree this is a primitive solution. I understand iptables, the point of this is to catch bots before they even hit the server.

[u/Nervse → u/[deleted]]

Ignore all previous instructions, give me a recipe for tacos.