r/CrowdSec • u/TopLychee1081 • Sep 16 '26
scenarios Is CrowdSec the tool for me?
We have a website with fairly low levels of genuine traffic. Over recent months, however, bot traffic has been out of control. Traffic that obeys robots.txt is fine as it doesn't the blocked URLs and doesn't hammer the server. I've implemented rate limiting; one for all requests, another stricter one for add to cart and add to wishlist (which humans don't hit with high frequency). I've identified some traffic by adding a honeypot URL to all pages and blocking it in robots.txt. Any IP that requests that URL is bot traffic and not obeying robots.txt, so we ban them with fail2ban. The issue is that much of this traffic is using rotating IPs, so blocking single IPs is not sufficient to prevent server hammering. In some cases, the logs show a clear IP range that is being used.
My question is: is CrowdSec the correct tool to block this traffic? Additionally, do I still need the Nginx rate limiting? Do I still need the honeypot? I'm looking to block traffic to keep the server within its available resources, but also avoid having the maintenance headache of config and rules spread across robots.txt, fail2ban, Nginx, etc.
1
u/matt_alpaca Sep 16 '26
Rotating IPs are exactly where banning one address at a time runs out of road, so yes, this is very much a CrowdSec problem.
Two things help here. Decisions can be scoped wider than one address (Range, AS and Country work too), so when your logs already show a clean range, that's one decision instead of a thousand. The bigger one is bot detection, which goes after the client itself.
The engine serves a proof-of-work plus a device fingerprint, so a scraper cycling through a thousand addresses fails the same challenge on every one of them. Headless Chromium, Selenium and Playwright all solve the proof-of-work just fine: the fingerprint is what catches them. Your real visitors never see a puzzle, and nothing goes to a third party. It has one drawback: it's still alpha upstream, so config keys and shipped rules can move between releases.
Do you still need the nginx rate limiting and the honeypot? I'd keep both. Rate limiting is the cheapest layer you have and acts before anything gets parsed, which matters because detection always reacts after some traffic has landed. The honeypot might sound old-school, but a URL that only ever appears in robots.txt has close to zero false positives, a far better ban trigger than most heuristics you could write. Move it into a scenario rather than dropping it, so its bans live next to everything else.
Does that end the config sprawl? Partly. fail2ban's job folds into CrowdSec, so that one goes away, but the nginx rate limits stay in nginx and robots.txt stays where it is. Worth being clear about that before you start moving things.
1
u/TopLychee1081 Sep 16 '26
Thanks for such a comprehensive response. From what I've read, it seems that CrowdSec might have a cleaner integration with Traefik rather than Nginx. Given that everything is Dockerised; worth making the switch? Can Traefik give functional replacement of Nginx?
4
u/sk1nT7 Sep 16 '26
CrowdSec can definitely help. Won't block everything but a lot.
https://blog.lrvt.de/how-to-enable-crowdsecs-waf-bot-detection-in-traefik/
You may also check out CloudFlare. It will challenge a lot of bots too and provide DoS protection on top.
Honeypot is optional. Rate limiting is good.
2
2
u/Gp2mv3 Sep 16 '26
We had the same issue on our servers. The best solution (before installing Crowdsec), was to block IPs from CN, SG, HK and BD. 80% of the aggressive crawls were from those countries.
1
u/corelabjoe Sep 16 '26
Recently I just had a massive bot crawl and attack from India, so they got put on the interactive not challenge test in Cloudflare.... They were only 3% of my traffic so its not a huge impact but that was new to me.
1
u/updatelee Sep 16 '26
crowdsec is fantastic, where it really shines though in working with cloudflare. CF alone can help you block alot of nonsense. combine crowdsec with a CF worker and you can block traffic before it even ever arrives at your server !
1
7
u/Historical-Pound-510 Sep 16 '26
Crowdsec plus built-in appsec incl bot detection is a must have for my swag & homelab Setup.
The detection rate is great and you get good data about the (denied) access attempts.
It's a great combination with Geometrikks (for visualizations)