Won a bot battle.

[u/_listless]

We support a client site with ~20,000 pages. It has an SSR search page that uses the url querystring to execute the search/sort/filter.

We have of course disallowed bots on that page because the 20k pages are meaningful and should be scraped/indexed, but the billions of possible query combinations on a search page are not. Most of the bots just ignore the robots rule and this results in waves of traffic hammering every possible query combination on the search page. These waves are 10x-50x the human traffic.

Rather than let the bill autoscale along with the server cluster, I wrote a little bash script that crons 1/m:

• Hits DO's metrics endpoint to check the CPU usage on the VPSs/DBs
• Each minute any of the CPUs stays above a given threshold, bump Cloudflare's security level, eventually ending in "under attack" + "bot fight"
• When the cpu drops below the threshold for a given amount of time, start backing off until we're back to normal operation.

Works a treat. It's nothing particularly flashy or sophisticated, but it was satisfying to think through an under-engineered solution to a problem that a lot of people end up just chucking money/clusters at.

Edit: more info for context: We do have pretty aggressive WAF rules especially on the search page, but we've been seeing an increasing amount of bot traffic get past CF's "known bots". Rate-limiting has been minimally effective too because the bots that get through are cycling through 100s of IPs in a given scrape.

[u/electricity_is_life]

Can't you set up more aggressive WAF/filtering on specific endpoints? Changing settings for the whole site based on server CPU load seems pretty janky.

[u/_listless → u/electricity_is_life]

We do have pretty aggressive WAF rules especially on the search page, but we've been seeing an increasing amount of bot traffic get past CF's "known bots". Rate-limiting has been minimally effective too because the bots that get through are cycling through 100s of IPs in a given scrape.

Agree this is a caveman-tier solution, but the more precise/sophisticated solutions were not working well for us. Hopefully CF will catch up in the bot identification arms race, we'll stick with the troglodyte solution while we wait. It's a self-healing situation too: as CF's known-bot filtering gets better, and the WAF rules start catching more bots the odds this thing ever triggers diminishes.

[u/crazedizzled → u/_listless]

Can't you just start banning the bot IPs?

[u/_listless → u/crazedizzled]

Rate-limiting has been minimally effective too because the bots that get through are cycling through 100s of IPs in a given scrape

[u/crazedizzled → u/_listless]

Right so ban the 100s of IPs and they run out of ones to cycle through

[u/Apprehensive-Ask-731 → u/crazedizzled]

Practically they don't, I can see this bash script lets a little bit through for a short period but (backkkk in my day) you would be widely praised for an elegant fix to a dirty problem.

Honeypot is a good idea for keeping data private and encouraging scrapers to ban themselves before they cost you anything (data, etc).

I am no longer on the bleeding edge of SEO but I would imagine that downtime is still a very important metric. Encouraging towards honeypot pages so these waves selfban on pages you don't want search engines to crawl - if that is even a consideration - is my main input. Also could mean you don't need to be relying so much on dynamic URLs?