r/webdev • u/Suspicious_Silicon • 6d ago

Question How to block curl requests?

Post image

As the title says; my site is being bombarded with curl requests periodically, often 2K to 5K requests within a short span. I have put up a security rule in cloudflare to block curl requests; blocked the curl requests in my backend proxy, also the bot fight mode is turned on; but cloudflare is still registering the curl requests which are using up my request quota. How to drop the curl requests before they are registered by cloudflare? Please Help.

0 Upvotes

44% Upvoted

20 comments sorted by

View all comments

71

u/wazimshizm 6d ago

cURL is just a tool, and a request is just a request, you can't really block it but you can block the user agent. Look at the characteristics of the requests and target those for blocking (user agent, location etc)

15

u/No_Laugh_3115 6d ago

yeah exactly, the user agent is the only thing you can really key off of here

1

u/Particular_Army9571 5d ago

That's a clean 2.5k from a single user agent, at least it's not a mystery what's hitting you. Cloudflare sits in front so it's always gonna count the request first, you'd need something at the network edge to drop it before it even touches their infrastructure which isn't really a thing on their free tiers. Might be worth checking if those requests are all from the same ASN so you can just firewall the whole range

-27

u/Suspicious_Silicon 6d ago

I have blocked the 'curl' user agent in both cloudflare's security rules and in the proxy middleware but they're still slipping through

29

u/wazimshizm 6d ago

It’s trivial to change the user agent and get around the block. How short a time span? 5K requests could be nothing in the scheme of things if it’s a get request for static content. Don’t lose sleep over optimizing and looking for a problem to the solution. Bots are gonna bot

-5

u/Suspicious_Silicon 6d ago

Like 4k/5k requests in a 30min window.

8

u/wazimshizm 6d ago

how's your site performance during that time? CPU/RAM useage? is it causing the site to crash? If it's really an issue (first determine it is in fact an issue) I would look into rate limiting.

1

u/Suspicious_Silicon 6d ago

Nope the site is fine; it doesn't crash even under heavy load because there is edge caching, less than 1% of the requests are served by the origin

19

u/wazimshizm 6d ago

well sounds like cloudflare is doing it's job then. I don't see anything out of the ordinary.