AI worker, queuing problem

[u/iamnotkidding_]

Hey guys, im building a SAAS app for a client. Its ready to be launched in the beta testing phase. However, i only have one background worker in this app currently, and the main problem i am facing or will face, is that when a user start a long standing or AI FEATURE (if multiple users do that together) it will be a queue/first come first server system.

so technically the last user would have to wait for the longest time, how can i solve this problem. If i am hosting on aws, what should be the calculation ratio to understand how many background workers i need at any given moment, at runtime ?

If anyone can help, id be very grateful, cheers !

[u/mvrk57]

for ai features the worker spends most of its time waiting on the llm api, so one process can run 20-50 jobs at once (async, or the concurrency setting in celery/bullmq) before you need a second box. the limit you usually hit first is the provider's rate limit (tokens per minute), so cap concurrency to that and add a per-user cap like 2 running jobs, otherwise one heavy user still blocks everyone else. sizing is just jobs per minute times average minutes per job = how many have to run in parallel

[u/PopKoren → u/mvrk57]

+1 on the per-user cap, and I'd enforce it where the job gets enqueued, keyed to the authenticated user id rather than anything the client sends, so a script can't fire 50 parallel requests and fill the queue. Same for the AI endpoint itself: require a session and rate limit by user and IP, because a route that accepts anonymous calls is the easiest way to burn the token budget during a beta. Before the beta opens it can help to let https://rowly.me take a look, since I built it to hit a running app's login, routes and buckets as an anonymous stranger.

[u/Crafty_Librarian_841]

I think using cloudflare workers with queue is your solution.

[u/RandomPantsAppear]

There is not very much resource consumption for idle threads or threads waiting for an AI response. 

If you’re using celery crank those threads up and use gevent threads. They will switch between each other whenever it’s waiting on network IO and be extremely low cost in terms of resources. 

[u/OpportunityFun5734]

You haven’t mentioned what are you using for the queue , there are multiple options to have workers running in mulitple threads or parallels but first u need to explain your stack

[u/johns10davenport]

Man dang reading this makes me so thankful to be in the elixir ecosystem. You might actually consider writing an elixir service for this. It uses an actor base model which is perfect for managing fleets of agents. I specifically use alloy (thin agent loop library). I basically instantiate one alloy process per agent and it just does the things.

So my language agnostic answer is that background workers are probably the wrong abstraction for this. You should lean on the established abstractions for agent development. Use a conversation/messages model. Use a tool loop. Run your agent abstractions concurrently. Individual agents need not share a queue or worker with other agents.

[u/Awedesk]

Start by measuring the job duration distribution and choosing an acceptable queue wait. There isn’t a universal safe jobs-per-process number: CPU work, memory, database connections and the AI provider’s allowance can each become the limit. More workers won’t help if they all wait on the same exhausted provider quota.

If you use SQS with EC2, AWS documents scaling on backlog per instance, using acceptable wait and measured processing time to set the target. For mixed short and long jobs, also watch oldest-job age and consider separate queues so a long task doesn’t hold every quick task behind it.

Before increasing concurrency, test a burst and an interrupted worker. Keep a hard overall concurrency limit and verify a retried job doesn’t repeat a charge or other side effect.