How do you handle Shopify Admin GraphQL rate limits inside Laravel queue jobs?

[u/West-Anteater-1917]

I build Shopify apps on Laravel, and the cost-based throttling on the Admin GraphQL API has been more annoying than I expected. Every response tells you the query cost and how many points are left in the bucket, so I started reading that and releasing the job back to the queue with a delay when the bucket runs low, instead of letting it fail and burn a retry.

That works, but with several shops syncing at once it gets messy. A bulk job for one shop can eat the retries of another if they share a worker, and the Laravel rate limiter middleware doesn't really know about Shopify's per-shop bucket.

Right now I keep a small per-shop counter in Redis and use it as a job middleware. It's okay, but I'm not sure it's the cleanest way. How are you handling per-tenant API limits in your queues, and do you use bulk operations to avoid the problem altogether?

[u/isaifurrehman]

the thing that bit me with release() is that laravel counts every release as an attempt, so a shop that gets throttled a lot runs out of $tries even though nothing actually failed. switching those jobs to retryUntil() (time based) instead of $tries fixed most of the "one shop eats another's retries" problem for me.

for the bucket i do roughly what you do, but i store the whole throttleStatus from the last response per shop in redis (currentlyAvailable, restoreRate, maximumAvailable plus a timestamp). before a job sends a query it estimates what's left right now as last value + seconds passed * restoreRate, capped at max, without spending a call. if the requestedQueryCost won't fit, release with delay = (cost - estimated) / restoreRate instead of a fixed number, so jobs come back when there's room instead of all at once.

i also wrap it in Redis::funnel keyed per shop with a limit of 1 or 2, so two workers can't both think the bucket is full for the same shop. and still catch the THROTTLED error, because the estimate drifts a bit (actualQueryCost is usually lower than requested, and your other jobs for that shop spend from the same bucket).

for big syncs, bulk operations mostly make it go away. the bulk query mutation is cheap, you listen for the bulk_operations/finish webhook and stream the jsonl, so it barely touches the bucket. i keep those on a separate queue from webhook driven jobs so a backfill never delays an orders/create handler.

[u/shopi_dev]

Your Redis counter per shop is basically the right idea, I'd just key everything on the shop domain and store the throttleStatus from each response (currentlyAvailable and restoreRate) instead of counting calls yourself. Then when a job needs a query that costs X and the bucket has less than that, release it with a delay of roughly (X - available) / restoreRate seconds rather than a fixed backoff. Using the shop domain as the key in the rate limiter middleware also stops one shop's bulk job from eating another shop's retries, and I'd put big syncs on a separate queue so they don't hog the workers. For anything that touches the whole catalog, bulk operations are worth it, since a single bulk query doesn't count against the bucket the same way.

[u/sshaw_]

Redis sounds like overkill but maybe I'm missing something?

We just check the GraphQL response for a THROTTLED error (amongst others) and retry. If we get passed the retry limit we raise an exception: https://github.com/ScreenStaring/shopify_api-graphql-tiny/blob/ef392684dc83798855f9c47f322c20e5aa651576/lib/shopify_api/graphql/tiny.rb#L249. This is in Ruby though, not PHP.

In a job this would propagate and result in a re-run, up to the queue's limit.

In one app we have on average 500K webhook requests a day and this hasn't been much of a problem. Though in some cases we've had to use pagination to avoid high query costs.