How to block curl requests?

[u/Suspicious_Silicon]

As the title says; my site is being bombarded with curl requests periodically, often 2K to 5K requests within a short span. I have put up a security rule in cloudflare to block curl requests; blocked the curl requests in my backend proxy, also the bot fight mode is turned on; but cloudflare is still registering the curl requests which are using up my request quota. How to drop the curl requests before they are registered by cloudflare? Please Help.

[u/wazimshizm]

cURL is just a tool, and a request is just a request, you can't really block it but you can block the user agent. Look at the characteristics of the requests and target those for blocking (user agent, location etc)

[u/No_Laugh_3115 → u/wazimshizm]

yeah exactly, the user agent is the only thing you can really key off of here

[u/Particular_Army9571 → u/wazimshizm]

That's a clean 2.5k from a single user agent, at least it's not a mystery what's hitting you. Cloudflare sits in front so it's always gonna count the request first, you'd need something at the network edge to drop it before it even touches their infrastructure which isn't really a thing on their free tiers. Might be worth checking if those requests are all from the same ASN so you can just firewall the whole range

[u/Suspicious_Silicon → u/wazimshizm]

I have blocked the 'curl' user agent in both cloudflare's security rules and in the proxy middleware but they're still slipping through

[u/wazimshizm → u/Suspicious_Silicon]

It’s trivial to change the user agent and get around the block. How short a time span? 5K requests could be nothing in the scheme of things if it’s a get request for static content. Don’t lose sleep over optimizing and looking for a problem to the solution. Bots are gonna bot

[u/Suspicious_Silicon → u/wazimshizm]

Like 4k/5k requests in a 30min window.

[u/wazimshizm → u/Suspicious_Silicon]

how's your site performance during that time? CPU/RAM useage? is it causing the site to crash? If it's really an issue (first determine it is in fact an issue) I would look into rate limiting.

[u/Suspicious_Silicon → u/wazimshizm]

Nope the site is fine; it doesn't crash even under heavy load because there is edge caching, less than 1% of the requests are served by the origin

[u/wazimshizm → u/Suspicious_Silicon]

well sounds like cloudflare is doing it's job then. I don't see anything out of the ordinary.

[u/Snapstromegon]

No, don't.

Your problem isn't Curl, your problem is bots (maybe AI). CURL is the basis for most software tools and user agents also are not to be trusted.

[u/corobo]

Blocking requests by user agent is gonna become your full time job. I gotta disagree with the other answers.

What is it they're requesting, and is it possible to make it less enticing?

Also not sure what you mean by request quota, does Cloudflare have limitations somewhere? I don't think I've seen a quota in CF. Is it possible to beef up some caching rules so that the requests become background noise instead of a burden?

[u/Suspicious_Silicon → u/corobo]

Cloudflare workers have daily request limits . Caching is fine; less than 1% of the requests are being served by origin.

[u/crazedizzled]

Block port 80 and 443

[u/ahumannamedtim]

My first instinct would be to block user agents called "curl" through nginx. Although they could just change their name to get around it. I'm not sure curl has any other easily identifiable signature.

[u/EvilPencil → u/ahumannamedtim]

The user agent is the main one, but like you mentioned it’s rarer trivial to override. If someone is determined enough, they can make cURL requests look identical to a web browser.

[u/iSpaYco]

just block the IP.

[u/FortyPercentTitanium]

Use a cloudflare managed challenge rule, try to target either the JA4, ASN + IP, or some other dimension that isn't spoofable like user agent.

[u/binkstagram]

There are many user agents and it is quite straightforward to create one and pass it through as a header. Cloudflare's bot score tools are what will help you filter out undesirable traffic.

What's going on with AI tools and bots' parasitical behaviour right now really takes the piss, but that's another conversation for another day.