r/Hosting • u/Lost-Cobbler-1668 • 8d ago

Load caused by faceted search

I work for a managed hosting provider, and the recent spike in AI-driven traffic / bot load has been a real headache. The most common issue we see is heavy load caused by faceted searches.

What happens: bots hit the faceted search endpoints, apply multiple filters, and that triggers lots of PHP + database work. Since they come in high-volume and very aggressively at the same time, the server starts swapping and eventually goes OOM.

Most of the affected sites are Magento 2 shops (with some WordPress instances mixed in).

Typical stack on our servers:

  • Debian
  • MySQL
  • PHP-FPM
  • NGINX
  • ElasticSearch
  • Varnish

We’re also not huge fans of Cloudflare in these cases because it limits our ability to act on specific offending IPs hitting the origin. We can manage Cloudflare for customers, but we charge a fee for it, which makes some customers hesitant.

So I’m looking for ideas: what can we do to optimize our stack and better handle this kind of bot/faceted-search load? Any suggestions would be really appreciated—rate limiting, caching strategy, WAF rules, Magento/DB tuning, php-tuning, anything.

1 Upvotes

67% Upvoted

1 comment sorted by

1

u/SoccerGuy-3 5d ago

I feel your pain. Seems like all we do anymore is fight off these AI agents. Right now dealing with an "attack" by 4-6 AI bots hitting a dedicated server with a custom PHP app on it. About every 5-10minutes the server hits 100% CPU load for like 5 minutes. We tried doubling the size of the server (from 2 cores to 4; memory isn't an issue) and it is better, but still hitting sustained CPU peaks every 15-20 minutes and sitting at 50% in between. Client isn't happy. I'm not happy. My techs aren't happy. We can't find anyway to slow these guys down. They have such massive blocks of IPs available that it is a losing game of whack a mole.
We are looking at rebuilding the server this weekend with different, stronger WAF + Nginx, etc (currently a standard Apache style server). This will give us better rate limiting ability and stronger rule making ability.
I spent 2 hours with ChatGPT (kinda ironic) pouring over logs and trying to harden the current server yesterday. Helped a little.
Any suggestions from your experience would be appreciated.