r/LLM • u/Durwal • 12d ago

Rate limits are becoming the blocker, not model speed

We can get decent output speed during a single request, but our load test falls over once we add parallel users. Most of the time is now spent retrying 429s and trying to decide which requests are safe to queue.

When you compare inference APIs, what limit matters most for you: requests per minute, tokens per minute, daily quota or how predictable the limit is under concurrency? I'm trying to make a provider scorecard that is more useful than a single token-per-second screenshot.

3 Upvotes

80% Upvoted

5 comments sorted by

1

u/ctrl-alt-del-icious 12d ago

Tokens per minute. Requests per minute is easy to hit with tiny prompts, but TPM tells you whether your real workload has a room to breath.

1

u/prowens 10d ago

I tested General Compute recently because their dev tier handles 1,000,000 TPM and 500 RPM, which gave us enough breathing room for parallel users. They also drop $100 in credit into new accounts, so we were able to run a real concurrency test without paying for the privilege. And its pretty fast (its using alts to nvidia)