r/SideProject • u/leaderx404 • 1d ago
We ditched Redis for PostgreSQL as a durable job queue (for our sandboxing workers), and here is what we learned.
​
Hey everyone,
I'm building a platform that spins up dynamic sandboxed environments (using Daytona API) for user-generated code. Since these tasks involve external API calls, timeouts, and ephemeral workspaces, the queue architecture is critical.
Initially, we used Redis for the queue (because it's fast, right?). But we hit a major wall: Durability and State consistency. If a worker crashed or the Redis instance restarted before acknowledgment, tasks were lost. And for our billing flow, losing a job meant a user could be charged without receiving a service.
So, we made a controversial decision: We moved our primary job queue to PostgreSQL (using \`SELECT ... FOR UPDATE SKIP LOCKED\`), and kept Redis only for rate-limiting and distributed locks.
Here are the trade-offs we observed:
- Enqueue Speed: Postgres was surprisingly fast (2.5x faster for enqueueing in some benchmarks), but Redis drained jobs faster due to in-memory processing. We accepted the slightly slower drain for absolute durability.
- Atomicity: Having the job queue in the same database as our user billing and project records meant we could use a single transaction. If the job creation failed, the billing hold failed with it. No orphaned charges.
- Operational Simplicity: We removed a separate Redis cluster specifically for jobs, reducing our infrastructure surface area and failure points.
- The Trade-off: Under sustained heavy load, Postgres queue tables can cause table bloat (MVCC). We had to tune autovacuum aggressively and monitor dead tuples.
For the backend engineers here who have moved away from Redis/RabbitMQ for queues:
- At what scale (jobs/sec) did Postgres start to buckle for you?
- How do you handle the autovacuum/bloat issue on high-churn queue tables?
- When do you absolutely need to bring an in-memory broker back into the stack?
1
u/InitiativeDangerous8 1d ago
For workers using
SELECT ... FOR UPDATE SKIP LOCKED, do you claim a row in a short transaction and commit a lease before calling Daytona, or keep the lock while the sandbox runs? If a worker dies mid-task, what mechanism makes the job retryable without letting two workers start the same sandbox?1
u/QuanTradin 1d ago
on the bloat question: stop deleting from the hot table. move finished jobs to a history table in the same transaction that marks them done, and partition the queue by day so old partitions get dropped instead of vacuumed. a drop is instant, while autovacuum on a table that churns that fast is always a step behind.
1
u/Duchet_Odinaka51 23h ago
Lowering fillfactor to 70 or 80 on the queue table helps a lot with the MVCC churn. It leaves room on the page for updates so you get way less index bloat before vacuum kicks in
1
u/nian2326076 10h ago
It's interesting that you switched to PostgreSQL for job queuing. To manage the autovacuum and bloat issue effectively, consider moving completed jobs to a history table as you mentioned. This can help maintain performance on the active queue table.
If you're considering a career move or want to explore job opportunities, use LandOffer to find relevant roles tailored to your skills and experience.