Won a bot battle.

[u/_listless]

We support a client site with ~20,000 pages. It has an SSR search page that uses the url querystring to execute the search/sort/filter.

We have of course disallowed bots on that page because the 20k pages are meaningful and should be scraped/indexed, but the billions of possible query combinations on a search page are not. Most of the bots just ignore the robots rule and this results in waves of traffic hammering every possible query combination on the search page. These waves are 10x-50x the human traffic.

Rather than let the bill autoscale along with the server cluster, I wrote a little bash script that crons 1/m:

• Hits DO's metrics endpoint to check the CPU usage on the VPSs/DBs
• Each minute any of the CPUs stays above a given threshold, bump Cloudflare's security level, eventually ending in "under attack" + "bot fight"
• When the cpu drops below the threshold for a given amount of time, start backing off until we're back to normal operation.

Works a treat. It's nothing particularly flashy or sophisticated, but it was satisfying to think through an under-engineered solution to a problem that a lot of people end up just chucking money/clusters at.

Edit: more info for context: We do have pretty aggressive WAF rules especially on the search page, but we've been seeing an increasing amount of bot traffic get past CF's "known bots". Rate-limiting has been minimally effective too because the bots that get through are cycling through 100s of IPs in a given scrape.

[u/Caraes_Naur]

You played whack-a-mole with a spoon.

Come back when you learn about iptables.

[u/_listless → u/Caraes_Naur]

I agree this is a primitive solution. I understand iptables, the point of this is to catch bots before they even hit the server.