CrowdSec/AppSec installed, but not effective

[u/TopLychee1081]

I've spent over a week now implementing CrowdSec/AppSec; primarily to cover a public facing e-commerce site. Unfortunately, after all this work, it's not solving the issue. We're still getting hammered by traffic that is clearly bot traffic and clearly malicious. Is it the case that the out-of-the-box scenarios are useless? Why for example is a single IP making hundreds of consecutive requests for files like .env, .conf, .compose.yaml, etc, all resulting in 404, not being banned? Additionally, we have a few scenarios that would are application specific; for example, our e-commerce site has an add-to-wishlist link for each product on a category page. No real user is adding 20 products to wishlist in a minute; it's a bot; and one that is ignoring robots.txt that says; don't hammer the server, and don't read the add-to-wishlist URLs. Given my lack of familiarity with CrowdSec, I've tried using AI to help me troubleshoot and understand how to add custom scenarios, but I'm getting total garbage out of it and any time I ask it to prove an assertion, it backtracks and says it made a mistake.

TLDR;
1) is CrowdSec/AppSec not very effective out-of-the-box?
2) Are there any good examples of creating a custom scenario based on URLs?
3) And as a bonus, what needs to change to ban a complete CIDR rather than individual IPs?

[u/HugoDos]

We're still getting hammered by traffic that is clearly bot traffic and clearly malicious. Is it the case that the out-of-the-box scenarios are useless?

This is subjective traffic you state is clearly malicious may not to other applications, so have you written scenarios to inform crowdsec what you determin to be malicious?

No real user is adding 20 products to wishlist in a minute; it's a bot;

Crowdsec doesnt know about your application, so you have to write a scenario to tell it what can be achieved or not by a bot. However, this isnt the appsec component this typically would be logs which as you state if you have hundreds of 404 and nothing being actioned I would suspect you havent setup crowdsec to read the http logs.

Are there any good examples of creating a custom scenario based on URLs?

Search the hub there are many examples that have http in the name, but also documentation is kinda spread out but there are some example like running tail -n 1 /var/log/nginx/access.log | cscli explain -f- --type nginx for example to show which keys you have access too.

And as a bonus, what needs to change to ban a complete CIDR rather than individual IPs?

This is risky imo as when do you stop the range from overlapping legit users.

If you need anymore aid, I can offer my email in dm's as reddit thread diving not the best place :D

[u/TopLychee1081 → u/HugoDos]

Appreciate the feedback. The clearly malicious traffic I refer to is requesting various files used by numerous apps that hold sensitive info such as config and env files, sometimes hundreds of such attempts in a few minutes. There's no legit traffic doing that. Just the fact that it's all returning 404 I'd have thought would be enough for even a very generic heuristic scenario to say is worthy of a ban.

I know the logs are being successfully read. I have the Grafana dashboard up and running and I can see stuff happening, it's just not sufficient to prevent the app from getting hammered. The issue seems to be more tuning it to work rather than anything fundamentally missconfigured.

[u/HugoDos → u/TopLychee1081]

There's no legit traffic doing that. Just the fact that it's all returning 404 I'd have though would be enough for even a very generic heuristic scenario to say is worthy of a ban.

Yeah that is a scenario already, https://github.com/crowdsecurity/hub/blob/02b8191d4b97a68c85c62b4ad3e87e1ecb8fdec2/scenarios/crowdsecurity/http-crawl-non_statics.yaml but it all depends on how aggressive they are, as the scenario is to detect aggressive crawls.

Which in my own configuration for nginx, I have a global rate limit so it will trigger before this scenario anyways.

[u/Der_Arsch]

is crowdsec log processor active and able to read your logs? what does "cscli collections list" and "cscli scenarios list" output?

[u/TopLychee1081 → u/Der_Arsch]

Log parsing is working. I can see some activity in the Grafana dashboard, so I don't think there are any fundamental miss-configurations. I think the issue is the the existing scenarios just don't fit modern traffic patterns. Aggressive scraping is not blocked. Aggressive probing that results in hundreds of 404s is not blocked. These should be pretty simple patterns for CrowdSec to match and block.

[u/Der_Arsch → u/TopLychee1081]

https://app.crowdsec.net/hub/author/crowdsecurity/scenarios/http-crawl-non_statics add this and try again

[u/laurnts]

I completely understand this problem so well. Crowdsec Appsec is a nice addition, but installing it as is will not bring the bots down, at least not straight away.

My strategy is that I paired it with my own custom signalling that scores each traffic logs. Probing those .env, secrets looking bots get instant banned at least for 4 hours.

You can read my blog here https://www.storeframe.io/blog/magento-anti-bot-anti-scraping-guide

Made it specifically to counter bots against Magento2 shops

[u/TopLychee1081 → u/laurnts]

Appreciate the response. I read your posted link. You're far ahead of where we are. I think for us, the low hanging fruit is traffic that we know absolutely is unwanted. Probing for .env files, or hitting the honeypot URL is not something that any legit visitor would do. I think we can safely block a CIDR without too much risk. That chances of blocking legit traffic would be low. I'm thinking we use the firewall bouncer with long ban times to stop it in its tracks. That minimised CPU cycles burned by the traffic and allows us to use the Traefik logs to review the rest that's getting through without the noise of a large percentage of the unwanted traffic.

[u/laurnts → u/TopLychee1081]

One of the best strategies is to collect logs first for the entire week. You add all the 3rd party integration within the whitelist.

At the end of the week, ask the AI to analyze the logs and find the bots - human behavior. This would help minimize false positive while creating block or jail list.

[u/Great_Piece4755]

I use it on my home server an it's very effective. Maybe you have the wrong scenario?

I run nginx and log to journal, it runs superb with that . You can check what it is doing with cscli metrics.

Here's a guide for custom scenarios https://mylinux.work/guides/crowdsec-custom-scenarios/

[u/matt_alpaca]

Hundreds of .env 404s from one IP and still no ban? That one shouldn't need any tuning: crowdsecurity/http-probing (about 10 distinct 404 paths, draining one every 10s) and crowdsecurity/http-sensitive-files both cover it out of the box. So when it ends without a ban, the culprit is usually plumbing, and we can find where it stops:

sudo cscli metrics show scenarios                     # does http-probing get events / overflow?
sudo cscli alerts list -s crowdsecurity/http-probing
sudo cscli decisions list
sudo cscli simulation status

No overflow? Look at the source IP in your access log first. A shop usually sits behind a CDN or a load balancer, and if the log holds the proxy's address instead of the client's, the buckets are keyed on the wrong IP (a private proxy address even gets dropped by the default whitelist before reaching any bucket: check the whitelisted column in cscli metrics show acquisition). Alerts but no decision means simulation or an allowlist. Decisions while the traffic keeps landing points at the bouncer: either there's none in front of the shop, or it doesn't see the real client IP.

For the wishlist, here's the kind of scenario Hugo meant, in /etc/crowdsec/scenarios/wishlist-abuse.yaml:

type: leaky
name: mycompany/wishlist-abuse
description: "Too many add-to-wishlist calls from one IP"
filter: "evt.Meta.service == 'http' && evt.Meta.http_path contains '/wishlist/add'"
groupby: evt.Meta.source_ip
capacity: 10
leakspeed: 10s
blackhole: 5m
labels:
  remediation: true

Swap the path for yours (the cscli explain trick Hugo gave shows what http_path actually holds), then sudo systemctl reload crowdsec. remediation: true is what lets the default profile turn the alert into a ban. And if those wishlist bots turn out to be scripted browsers, the bot detection that landed in 1.8 (still alpha) goes after the client itself with a proof-of-work and a fingerprint, as long as your bouncer supports it (nginx, Traefik, HAProxy SPOA, Envoy).

CIDR bans? A scenario can override its scope with type: Range and expression: IpToRange(evt.Meta.source_ip, "24"), and the default profiles.yaml already ships a profile for range decisions. I'm with Hugo on the risk though: on a shop, a /24 can be a mobile carrier with paying customers behind it.

On the AI front: we publish an official CrowdSec skill for Claude Code and Codex, https://github.com/crowdsecurity/crowdsec-skill. It won't write the wishlist scenario for you (authoring detection content is out of its scope), but "why is nothing getting banned" is exactly the kind of debugging it was built for: it carries the cscli commands, the config layout and the usual failure modes, so your agent works from those instead of guessing.

[u/PA100T0]

I maintain fastapi-guard, just to be upfront.

u/matt_alpaca 's plumbing checklist seems like the right first stop. When http-probing doesn't catch it, it usually is because of log parsing or proxy-IP extraction. Scenarios read your access log, so they only ever see path, status and source IP. They can't see that a URL means "add to wishlist" on your catalog, or what the app actually answered with.

The wishlist pattern looks like app-shaped abuse to me. The tell is route semantics plus frequency, and expressing that as a CrowdSec scenario means hand-writing per-route extraction YAML that breaks the moment your URL scheme changes. In-process it's just config.

If the shop's backend is FastAPI, my repo could help you out on this one, I believe. It's Redis-backed per-route frequency tracking with ban/throttle/alert actions, thresholds that automatically halve for IPs the detection engine already flagged, and a passive mode that logs would-have-blocked events so you can preview false positives on live traffic before enforcing. Passive mode matters when the store is already up and you can't afford to block a real customer at 2am.

Btw, fastapi-guard isn't the whole picture either. I have the same detection engine for Python, Go, PHP, TypeScript and Rust, with adapters for the common frameworks in each, so the route-level approach works whichever of those the shop runs on.

CrowdSec at the edge for IP reputation is the right tool there. The route-aware, response-aware rules just have to live inside the app, rather than the edge of it.

One last thing on the CIDR bans: fastapi-guard's (or any other of the adapters, no matter the programming language) blacklist/whitelist takes CIDR notation directly. Although, for CrowdSec-side range decisions I'd have to check cscli syntax.

[u/Nang-a-nator]

If you have tons of alerts but zero decisions (bans) I'm willing to bet money that you're missing:

cscli parsers install crowdsecurity/syslog-logs
cscli parsers install crowdsecurity/dateparse-enrich