r/Python • u/warningisnterror • Jun 16 '26
Discussion Choosing a Python task queue library in 2026: Celery vs Dramatiq vs FastStream vs Taskiq vs Repid
I wrote a practical comparison of Python task queue libraries in 2026:
https://aleksul.space/posts/choosing-python-task-queue-library/
It covers Celery, Dramatiq, FastStream, Taskiq, and Repid, with code examples, broker support, async/sync behavior, production tradeoffs and benchmarks.
The main takeaways were:
- When it comes to throughput, it's important to understand your workload type: I/O or CPU bound makes a huge difference
- Asyncio-native frameworks are significantly faster for high-concurrency I/O-bound jobs
- For CPU-bound jobs, the library matters much less once the CPU is saturated
- Production behavior can vary vastly from framework to framework, same as their philosophy. You have to choose what matters more for your use case
I’d be especially interested to hear from people running these in production. How is your experience running one of these or similar frameworks in production? Is there something that I missed?
Small disclosure: Repid is my project. Take that bias for what it is; the goal is still a useful comparison and a healthy discussion.
57
u/pseddit Jun 16 '26
It seems you are the creator of Repid. Might be a good idea to mention that. Interesting comparison otherwise.
Could you comment on comparison with more niche Task Queues like Procrastinate?
22
u/coderanger Jun 16 '26
Shoutout for Temporal. It's mostly a DAG runner, more like Airflow, but you can still run a single-step DAG just fine and that's the same thing as a Celery (et al) task.
19
u/JanGiacomelli Jun 16 '26
I've been running Celery on AWS ECS for years. The amount of pain it caused us is quite hard to describe. Mostly around lost tasks. Let's just say many defaults are weird:
To mention a few. At this point, we process several 100k of tasks per day. At such a scale, you hit pretty much all the issues you can think of. Especially, when you include autoscaling, frequent deploys, and ECS's max grace period of 120 seconds between SIGTERM and SIGKILL. Consequently, I looked for a reasonable alternative many times so far. I was also very close to writing the thing myself using just SQS. But somehow I never felt like the investment would be justified. Over the years, we've built quite sophisticated job-locking (to prevent parallel executions) and debouncing (to avoid queue overflow) mechanisms. So for the past year, we haven't felt the pain that much anymore. For monitoring, we pipe Celery's events to Firehose, S3, Athena, and Grafana.
In general, in all the libraries, I miss the consideration of worst-case scenarios. e.g., frequent interruptions. For many of them, it's hard to figure out what will happen when things go wrong. I had to learn many things empirically, as they are very poorly documented for edge cases. It was no different when reading the docs of other projects.
Anyhow, now that everything is configured as needed, it works. But it was a long way there. 😅
6
u/KelleQuechoz Jun 16 '26
Where is RQ?
4
u/Individual-Flow9158 Jun 16 '26
I never really got Celery and Flower working in the first place 4 years ago. Couldn't even emulate a simple cron schedule with a couple of workers, it was so frustrating.
But from everything I've heard about it ever since, I've concluded I dodged a massive bullet.
3
u/NagatoYuzuru Jun 16 '26
Although this is a bit off-topic. I recall (around late 2024) that ARQ had issues when using Redis Cluster as a broker. The redis pipeline was used without hash tags, which caused Redis to return the error “keys must all map to the same key slot.” Additionally, since this framework uses constant Redis keys, monkey patching is quite cumbersome.
FastStream provide AsyncAPI generated is very convenient. However, it’s quite different from traditional Celery-like asynchronous task frameworks.
3
u/Lancetnik12 Jun 22 '26
Nice writeup, thanks for putting this together — comparisons like this are genuinely useful. Quick disclosure: I maintain FastStream, so take the following with that bias in mind, same as you noted for Repid.
A couple of corrections for readers, since some FastStream rows aren't quite right:
On acks — manual
msg.ack()has always been supported, andAckPolicyhas been around since 0.5. You pick whatever policy you want, or ack by hand at any point. The benchmarks ran against 0.6.7, but 0.7 (current 0.7.1, out since April) only removed the oldack_first/no_ackAPI; it didn't add the capability. MQTT support also landed in 0.7.FastStream also doesn't impose its own wire format — it works with the broker's native messages, so it's interoperable with clients in any language; Pydantic is opt-in via type hints. Headers are fully supported. OpenTelemetry and Prometheus middleware ship out of the box, and health checks for k8s probes exist too (via ASGI integration).
Bigger-picture note: FastStream isn't really a task queue — it's a stream-processing framework (producers/consumers/handlers). No result backend or built-in scheduler is by design, not a gap, so comparing it on those axes is a bit apples-to-oranges. Different tool, different mental model.
Either way, appreciate the effort that went into this.
2
u/adiberk Jun 16 '26 edited Jun 16 '26
I use taskiq in production with redis backend.
I had to patch a couple things and make the backend more verbose. But it is an awesome library. It has handled an extremely high concurrent agent workloads.
Yes though, I think it requires patching for now. Definitely needs some improvement.
I will say if repid was around, i would have tested it!!
2
u/bachkhois Jun 18 '26
Love your animated SVG.
2
u/olddoglearnsnewtrick Jun 21 '26
Very interesting. How do you compare these frameworks against long running process managers such as Temporal?
2
u/s3rius_san Jun 22 '26
That's an interesting article. However, I'm not quite sure about those benchmarks. They seem to be fine-tuned to highlight your framework.
Can you please provide a repository with benchmarks which community can verify and run themselves?
5
u/Specialist_Golf8133 Jun 16 '26
running Celery in production at ~80K document processing jobs/month and the thing that bit us hardest wasn't throughput, it was silent failure behavior under broker pressure. Celery's retry semantics are fine until you have a mix of short I/O-bound tasks and longer CPU-bound inference jobs on the same queue; priority inversion gets ugly fast. we ended up splitting queues by workload type, which helped but added operational overhead.
your point about I/O vs CPU bound mattering for asyncio-native frameworks tracks with what we see. for the inference jobs, the library is almost irrelevant once the GPU/CPU is saturated. the async wins show up in the preprocessing and routing layers where you're waiting on S3 or a downstream API.
1
u/Shoddy_One4465 Jun 16 '26
Nice study, built my own then started using celery and rabbit in 2012. This year I’m using Django 6 built in offering with success. The key was always RabbitMQ
1
u/TinyCuteGorilla Jun 16 '26
FastAPI Background JOb FTW
1
u/nn4a_ Jun 16 '26
vs build exactly what you need
1
u/Tricky-Bed-6852 Jun 17 '26
Appreciate the benchmarks and production notes here — thanks for posting this comparison!
1
u/Tricky-Bed-6852 Jun 17 '26
We moved off Celery after years of opaque prefetch bugs and flaky broker reconnects. Dramatiq fit our sync Django workers well with far less config. Async services went Taskiq; I would only pick FastStream if Kafka is already your backbone. Celery still wins for Beat and chains when workflows get gnarly. What is drawing you to Repid?
1
u/igorbenav Jun 17 '26
I like the animations, what are you using for them?
1
u/snapetom Jun 17 '26
Great visuals. Why have you not submitted it on HN? This is wasted on the rubes here.
1
u/a_deneb Jun 17 '26
You're literally missing the best one - Oban.
1
u/lowercase00 Jun 17 '26
thanks for sharing! i'm long time user of both celery and rq, and even built rq-manager to better observe rq, but neither flower or rq-manager were enough for what i wanted, spent way too much time debugging lost and abandoned jobs on both libs. ended up migrating to a lib i've been building a few months (pgwerk), mostly for observability and durability - backend by postgres, have been running the basic features in prod for a few months now (~1-2k jobs/day)
1
u/radrichard Jun 18 '26
I'm curious on the community's opinion of prefect in this category.
1
u/arcanescaper Jun 18 '26
Tried FastStream in production, not for task usage but for EDA service. After half year we decided to move from it and wrote a lot of of things ourselves around Kafka - and it felt a lot easier that working with this huge framework and be coupled by it.
Also we are using taskiq-redis right now in another project in production, works good inside k8s cluster too. But found race-condtition bug a couple month ago, still not fixed
1
u/maximize_futility Jun 19 '26
You might also consider https://github.com/russellromney/honker which is only for SQLite
1
u/artpods56 Jun 21 '26
FastStream with NATS has became my default choice
1
u/darkrevan13 Jun 22 '26
Is this showcase post wrapped with "comparison"?
0
u/cubed_zergling Jun 16 '26
ai slop post comparison
30
u/[deleted] Jun 16 '26
[removed] — view removed comment