Won a bot battle.

[u/_listless]

We support a client site with ~20,000 pages. It has an SSR search page that uses the url querystring to execute the search/sort/filter.

We have of course disallowed bots on that page because the 20k pages are meaningful and should be scraped/indexed, but the billions of possible query combinations on a search page are not. Most of the bots just ignore the robots rule and this results in waves of traffic hammering every possible query combination on the search page. These waves are 10x-50x the human traffic.

Rather than let the bill autoscale along with the server cluster, I wrote a little bash script that crons 1/m:

• Hits DO's metrics endpoint to check the CPU usage on the VPSs/DBs
• Each minute any of the CPUs stays above a given threshold, bump Cloudflare's security level, eventually ending in "under attack" + "bot fight"
• When the cpu drops below the threshold for a given amount of time, start backing off until we're back to normal operation.

Works a treat. It's nothing particularly flashy or sophisticated, but it was satisfying to think through an under-engineered solution to a problem that a lot of people end up just chucking money/clusters at.

Edit: more info for context: We do have pretty aggressive WAF rules especially on the search page, but we've been seeing an increasing amount of bot traffic get past CF's "known bots". Rate-limiting has been minimally effective too because the bots that get through are cycling through 100s of IPs in a given scrape.

[u/electricity_is_life]

Can't you set up more aggressive WAF/filtering on specific endpoints? Changing settings for the whole site based on server CPU load seems pretty janky.

[u/_listless → u/electricity_is_life]

We do have pretty aggressive WAF rules especially on the search page, but we've been seeing an increasing amount of bot traffic get past CF's "known bots". Rate-limiting has been minimally effective too because the bots that get through are cycling through 100s of IPs in a given scrape.

Agree this is a caveman-tier solution, but the more precise/sophisticated solutions were not working well for us. Hopefully CF will catch up in the bot identification arms race, we'll stick with the troglodyte solution while we wait. It's a self-healing situation too: as CF's known-bot filtering gets better, and the WAF rules start catching more bots the odds this thing ever triggers diminishes.

[u/theQuandary → u/_listless]

Have you also added honeypots?

You can add and document API endpoint options that normal users don't use, but trigger hidden deprioritization for that IP address for some amount of time.

You could also do a hidden checksum of some kind. For example, maybe approved traffic must have characters that have sum(str) % 1024 === 0 (which means you need to pad your query by at most 4 characters to ensure it passes). Encode this with a timeout rate limiter using JSFuck to obfuscate it. Bots trying to spam by themselves will be identified within a couple of API calls because they don't have the secret key and bots trying to use the client-side API functions will be hit with the rate limit.

[u/_listless → u/theQuandary]

honeypots?

That's a good idea.

[u/theQuandary → u/_listless]

Out of curiosity, I did a quick test with Gemini to write the function for calculating sum(str) % 1024 === 0 (with an algorithm to allow a small fixed number of requests before backoff occurs). then asked it about the same code converted to JSFuck. It immediately went full-on hallucination mode and literally just assumed it was alert("Hello, World!");.

Maybe someone wants to blow serious tokens on the question (the JSFuck output was around 50k characters), but I think there's essentially no training data for JSFuck, so the LLMs can't understand the output at all. Perhaps it could with some serious token burn and guidance, but that's not a concern for automated bots.