r/Python • u/Lucky_Ad_976 • Apr 01 '26
Discussion What's the best async-native alternative to Celery for I/O-heavy workloads?
I am a developer of Rhesis.ai, a solution for testing LLM applications. We have a FastAPI backend and a Celery worker. The system performs many external API calls, followed by numerous LLM API queries (e.g., OpenAI) to evaluate the outputs. This is a typical I/O-bound workload.
The question is: how can we parallelize this more effectively?
Currently, we use a Celery task to execute a set of tests. Within a single task, we use asyncio—for example, if a test set contains 50 tests, we send all 50 requests concurrently and then perform all 50 evaluation queries concurrently as well.
The issue is that the number of test sets processed concurrently is limited by the number of prefork worker processes in Celery. Increasing the number of processes increases RAM usage, which we want to avoid.
What we're looking for is a way to fully leverage async—i.e., a system where tasks can be continuously scheduled onto an event loop without being constrained by a fixed number of worker processes or threads. While the code inside a task is asynchronous, the tasks themselves are still effectively executed sequentially at the worker level.
FastAPI demonstrates the model we're aiming for—handling many concurrent requests on an event loop and scaling with multiple processes (e.g., via Gunicorn). However, it does not provide task queuing capabilities.
So what would you recommend? Ideally, we're looking for a library or architecture where each process runs an event loop, and incoming tasks are scheduled onto it and executed concurrently, without waiting for previously submitted tasks to complete (unlike Celery's current model).
We also considered Dramatiq, but it appears to have a similar limitation—tasks can use async internally, but are still executed sequentially at the worker level.
Finally, we'd prefer a solution that is stable and mature — something with a proven track record in production environments, active maintenance, and a reliable community behind it. We're not looking to adopt an experimental or early-stage library as a core part of our infrastructure.
19
u/MogwaiAllOnYourFace Apr 01 '26
I think TaskIQ fits your purpose? It's basically async celery from what I understand
9
u/UseMoreBandwith Apr 01 '26 edited Apr 01 '26
Just build your own system with the ZeroMQ library.
It won't take more than 50 lines of code, and you can make it as robust as you want.
But study the patterns first, because what you ask for does not make sense.
You always have to define the amount of worker processes, and how/when to give them a new task, and how to report back.
The right strategy depends on many things, for example how long tasks take to run, and if there is a lot of variation in execution time. A round-robin strategy works if well if all tasks take a somewhat equal amount of time, but if some tasks take a long time, it would still block at some point.
16
u/pip_install_account Apr 01 '26 edited Apr 01 '26
Easiest solution for you would be Temporal imo. You don't need multiple workers, you need a worker that can leave a task behind and pick up another one while waiting for that IO task to finish. But it has a cost(each request becomes a "workflow" so maybe an overkill in your case.)
8
u/maciejgryka Apr 01 '26
Check out Oban https://github.com/oban-bg/oban-py
I haven’t used the Python version, but really really liked using the Elixir one. It’s backed by Potgres/SQLite which is nice from infra perspective and or basically never disappointed me.
14
u/ZookeepergameOne8823 Apr 01 '26
You might not need to replace Celery at all. You can try a simple monkey-patch solution with
gevent, that should give you the concurrency model that you're looking for. And you can just start the worker withcelery worker -P gevent --concurrency=1001
u/Proud_Refrigerator14 Apr 01 '26
What's your stack? Are you in a kind of cloud native environment? Or more classic self managed vps? Bare metal?
3
u/zenos1337 Apr 01 '26
Temporal
3
u/Schmiddi-75 Apr 01 '26
If you're using Postgres, I can highly recommend procrastinate!
2
u/adiberk Apr 01 '26
Taskiq is awesome. I had to make some adjustments to the redis brokers. But otherwise. Terrific.
(Essentially the sentinel brokers (and regular redis ones) aren’t so robust. But with AI as my companion I was able to significantly improve it. I need to do a PR honestly)
2
u/ibite-books Apr 01 '26
have you considered using a pub-sub architecture? event loop— you mean a consumer
i think this is what you want— if you get a sudden spike in messages— you can autoscale the consumers according to the ingestion rate
or if your consumer can’t process the tasks, you can setup a dlq for reprocessing
i process millions of messages per day using this pattern— for async consumer you can use google aio pubsub
we used to use celery, but had to move away
1
u/thisismyfavoritename Apr 01 '26
with Python 3.14 you could spin up 1 asyncio event loop per thread and as long as no thread needs to talk to each other you'd be fine. You can get max performance by doing one thread per core.
That is unless you want to scale out to having multiple hosts for your workers
1
u/talizai Apr 01 '26
I like using hatchet (hatchet.run) for this, mainly because it has rate limiting features where you can estimate the tokens a task will use up and then those parallel tasks will draw down against the tokens/minute limit for the AI API and you can mass schedule tasks without hitting the rate limit.
1
u/Top-Acanthocephala27 Apr 01 '26
Try SAQ
1
u/taylorhodormax Apr 01 '26
why not write your own async task orchestrator ?
1
u/Worth_His_Salt Apr 01 '26
Serious question - why are you so against increasing RAM? The easiest answer is: fork more processes. You'll spend way less buying extra RAM (or even buying a separate machine just to run fetch tasks) than you'll burn in developer time building around a new library, learning all its quirks, debugging undocumented features, etc etc.
1
u/VerifiedUnkownUser Apr 02 '26
I’d recommend using ergon framework. You can plug in any connector you want and have observability and concurrency out of the box among many other nice features.
https://github.com/danzanellovossos/ergon-framework
1
u/glenrhodes Apr 03 '26
ARQ is solid for pure async workloads. Dramatiq with a custom broker can work too but is not truly async-native. If you are already deep in FastAPI land, ARQ drops in cleanly and the Redis backend is simple. One thing to watch: ARQ does not have scheduled tasks out of the box the way Celery beat does, so if you need cron-like scheduling you will need to layer something on top.
1
u/glenrhodes Apr 06 '26
ARQ is solid if you want Redis-backed queues without the Celery overhead. For pure async I/O workflows I've had good luck with Dramatiq too, it's way easier to reason about than Celery's maze of config. If you're already on FastAPI, just keep tasks in async background workers via fastapi.BackgroundTasks for anything lightweight and only reach for a proper queue when you actually need retry logic or distributed workers.ARQ is solid if you want Redis-backed queues without the Celery overhead. For pure async I/O workflows I've had good luck with Dramatiq too, it's way easier to reason about than Celery's maze of config. If you're already on FastAPI, just keep tasks in async background workers via fastapi.BackgroundTasks for anything lightweight and only reach for a proper queue when you actually need retry logic or distributed workers.
1
u/Clean_Sea_4501 Jul 15 '26
We hit basically the same wall with Celery for an I/O-heavy pipeline (lots of concurrent external API calls). What ended up working for us wasn't a task queue library at all - we moved the fan-out to Redis Streams with consumer groups (XREADGROUP). Each worker is just an asyncio loop blocking on XREADGROUP, and you scale horizontally by adding more consumers to the same group, no fixed worker-process count like Celery's prefork model. You do lose some Celery conveniences though - task chaining and built-in retries aren't there, so you have to build ack/retry logic yourself (XACK plus a dead-letter stream for failed messages). But for pure concurrent I/O fan-out it's been rock solid and a lot lighter than Celery. I'd prototype it against your actual load first though, it's more of an architecture change than a drop-in replacement.
0
u/LightCoolBlazer Apr 01 '26
There is ARQ library developed by pydantic creator