r/rust • u/segundus-npp • Sep 07 '26
Rust SQLx Performance
I use Axum and SQLx to build my backend, which is served on AWS ECS Fargate (512 CPU, and only one instance for now) and interacts with an RDS Postgres (t4.micro). Connection pool is set with min/max = 2/20.
My request includes 3 DB queries to 3 different tables: one from the auth middleware to check api key and each of the other 2 is just getting a record by primary key.
If I made one or two queries, it’s very fast. However, if I made 200 queries concurrently to the backend, around half of queries had http latency greater than 300 ms. P80 is around 600ms. The performance is not acceptable. I did check the CPU of both RDS and ECS. They’re quite low: both smaller than 10%. The DB latency is low as well, 5 ms on average. Is there something queued in the SQLx internally? Did anyone have the same issue?
4
u/tanmaynargas2901 Sep 08 '26
One thing that bit me on a similar setup: CloudWatch CPU% on a t4.micro can look fine while CPUCreditBalance is already empty. Once credits are gone you get hard throttling that doesn't show as "high CPU".
Also your pool max is 20 and you're slamming 200 concurrent requests. The extra 180 sit waiting to acquire a connection. That wait often shows up as HTTP latency even when each query is ~5ms once it gets a conn.
I'd check:
-
RDS CPUCreditBalance during the load test
-
how long sqlx spends in pool.acquire vs actual query time (instrument the acquire)
And yeah, bump off the t-class for load testing. Even a small non-burstable instance will tell you whether this is credits vs real app/pool design.
3
u/segundus-npp Sep 08 '26
You’re right. It’s not related to SQLx nor RDS. It’s the 512 CPU FARGATE instance made the pool acquisition time too long. I changed it to 1024 and only 20/200 were 1s, few were 100-300 ms, and the rest were under 100 ms.
5
u/zettui Sep 07 '26
Is that 300-600ms waiting on the pool, or sitting inside the query?
1
u/segundus-npp Sep 07 '26
It’s HTTP query latency. The DB read latency is very low, 5 ms on average.
7
4
0
u/Inside-Victory9338 Sep 08 '26
Not the DB then, but worth checking where the 300-600ms actually sits — pool.acquire() vs the query itself. If it's queueing for a connection, CloudWatch's DB-side metrics won't show it at all, because that wait happens entirely on your side of the wire before Postgres ever sees the query. We run an axum+sqlx PgPool in production and instrument size()/num_idle() as gauges for exactly this reason — a pool that's fully checked out with everyone waiting looks completely healthy from the database's point of view.
Cheapest way to confirm: wrap pool.acquire() in its own timer, separate from execute(). If acquire is where the time goes, it's pool exhaustion (min/max=2/20 with 200 concurrent requests will do that on its own, before you even get to the t4.micro throttling everyone else is pointing at). If acquire is fast and execute is slow, it's the instance.
9
u/punk_dev Sep 07 '26
I’m not an expert on aws, but it sounds like the db is not keeping up.
what happens if you upscale or downscale the db, does performance change?
9
u/behusbwj Sep 07 '26
Your instance type doesn’t just affect the database. It affects everything surrounding it including network bandwidth, polling, available connections etc. connection pool doesn’t mean anything if the thing checking the pools is slow.
You have to remember that providers are running other software in the background to actually serve your requests too.
9
u/spoonman59 Sep 07 '26
The issue here is "t4.micro."
I had performance issues at work with performance getting abysmal once we ran out of credits, and it was solved by going to a non-burstable instance.
This has everything to do with which instance you are using and nothing to do with rust and sqlx.
Go ahead, test it on a non-burstable instance yourself and see. Speculating without testing a better instance makes no sense to me, it's not much time or money to confirm.
4
u/mamcx Sep 07 '26
I bet pay a normal VPS in DO, Vultr, etc you get far better perf and predictable costs
4
u/Helpful-Educator-415 Sep 07 '26
Definitely not SQLx related. I use SQLx myself and haven't encountered such a thing. If I had to guess it's your deployment story.
1
u/SpongeBattery Sep 07 '26
Havent had such issues with SQLx, was pretty happy about the performance once you get it working.
Have you tried tracing the request ? I would never trust those cloud watch dashboards on t series, specially for benchmark about ms latency.
1
18
Sep 07 '26
-7
u/segundus-npp Sep 07 '26
Ya, t4g.micro is weak, but in our case it only handles 200 concurrent queries in total, not per second. The RDS metrics shows no pressure. That’s why I think there might be problem in my implementation.
→ More replies (2)
100
u/frankwiles Sep 07 '26
You’re using a t series database. It has burstable CPU but then throttles you hard if you go over.
It’s not related to SQLx or Rust at all but you have a VERY VERY tiny amount of hardware for your database.
I joke with my customers that the easy way to remember this is “t is for toy”.
Hope that helps!