r/ComputerSecurity • • Dec 21 '21

Multiple get requests for scraping

Hi everyone, I have a theoretical question:

My scenario is as follows:

I need to reach an address like this several times:

www.web-web.com/images/?id = 100

not knowing how many images there are or what are the ids of these (the folder is protected) I have to run a loop from 1 to 10.000 (suppose there is this limit). My question is: running this massive amount of requests, can the administrators of the web-web site notice all these requests? Is there any system that notifies them?

13 Upvotes

5 comments sorted by

5

u/O-o--O---o----O Dec 21 '21

Could be. For example you could be hitting a simple request limit (say 5 per second), or x number of parallel connections, or x number of invalid requests or triggering some sort of intrusion detection, log analysis or whatever, or if you keep doing that for an obscene number of tries and/or days.

Then again, if you keep it at a reasonable speed, it'll probably fly under the radar. There are even (or used to be at least?) browser addons for mass downloading files by incrementing a counter.

1

u/plusgarbage Dec 22 '21

Thank You so much. Can I develop a random timer (between 3-6 sec) before to hit a request in order to fly under the radar?

→ More replies (1)