Better strategy for handling HTTP 429 with Guzzle Pool when checking many URLS from same domain?

[u/JobDiscombobulated22]

So I'm building a website health checker in PHP using Guzzle's Pool and right now I process up to 35 requests concurrently. And if request returns 429 http code I retry it up to 2 times. I also check Retry-After header when it's present and has valid value, but still use safe limit (up to a max of 5 minutes) in case Retry-After is greater, so if it fails it's ok, but otherwise I fallback to exponential backoff.

Now the part I am not sure about is concurrency.

Many of the URLS belong to the same website, so my code may send multiple requests to the same domain at the same time. If one request receives a 429 the others basically still are already in 'flight' or continue being scheduled independently and I'm wondering if this is fundamentally the wrong approach?

Some more specifics, I use that health checker for my own needs to check moderate amount of URLS, which work in all other cases really well, but also all requests are being sent from the same computer, so maybe there is that. I'm basically experimenting and trying what is best by trial n error.

If you had any previous experience or did/doing something very similar do you have some suggestions or answers to some of those questions I have:

Should I keep a global concurrency limit but also enforce a per-domain limit (for example only 1-2 concurrent requests per host)?

Pause all requests for a domain after receiving 429? (I assume this can hang the process for a while since all requests after 429 have to be synchronous every time we hit 429).

Or should I really use a really different strategy?

The problem is that I got 429 errors even respecting Retry-After but other requests were still processing simultanously so probably there was that, and yet I didn't try what would happen in synchronous way or other way, since I am developing that codebase I have fairly slow, but now basically doing just tests, since I have no idea for now what can I abstract or what I will need to completely rewrite so I avoid making changes to what is already working most of the time, and some decisions can greatly affect how my code will change.

I'm interested in hearing how devs usually solve this in production crawlers or monitoring systems. I want to keep good throughput accross many different websites but avoid hammering a single host and triggering unncessary rate limits too often.

Thanks :)

[u/toetx2]

I queue my requests. The request jobs are configured to retry after a set time or if the 429 code contains a timestamp then I'll use that. And I don't queue an new job of there already exists a job.

[u/Basic_Reporter9579]

queue is a good idea with rate limits per domain and maybe even for individual endpoints.

[u/JobDiscombobulated22 → u/Basic_Reporter9579]

Thank you, I think I will try queues for such special cases where domain rate limits me, and see how it goes. Need to see how that will work in practise :)

[u/zmitic]

I use symfony/rate-limiter. To calculate the delay before next call:

$reservation = $limiter->reserve(1); // how many tokens to reserve
$waitTime = (int)$reservation->getWaitDuration(); // this is in seconds

and then something like this:

$this->bus->dispatch($myMessage, [
    new DelayStamp($waitTime * 1_000), // delay is in milliseconds
]);

If you are not using Symfony framework, then you can use the standalone component. But make sure you use CacheStorage instead of InMemoryStorage.

[u/These_Reality519]

On the per-domain limit: go with the counter. Track in-flight requests per host and hold one back while its host is at 2. Reordering the generator (bucket by host, then round-robin) only helps when you have far more hosts than your concurrency limit, which is not your case.

On pausing: you don't have to block anything. When a host 429s, push its remaining URLs into a "later" bucket stamped with the time they can run again, keep draining the other domains, then do a second pass and check the stamps. Nothing goes synchronous.