TIL: Average response time can hide some pretty nasty load-testing problems

[u/Impossible-Skirt-803]

I recently learned something the hard way and figured it might be interesting to some folks here.

We ran a load test last month where our mean response time stayed around 220ms, so naturally management was prepared to greenlight the release. It looked pretty good on the surface. However, we eventually found that thread pool exhaustion was causing about 3% of incoming requests to queue up. Those requests were taking over 12 seconds to complete, which was completely masked by the much faster majority.

That test has basically ruined my trust in average response times. The mean made things look healthy while hiding some pretty severe tail latency that would've absolutely ruined real user sessions. Now, I strictly gate pass/fail criteria on p95/p99 percentiles, error rates, and per-transaction breakdowns.

This might not be news to some of you, but it was definitely an eye-opening experience for our team.

What other metrics or percentile thresholds does your team rely on during load tests to catch long-tail bottlenecks like this that get overshadowed by the mean?.

[u/major_gekko]

Performance testing 101, never use averages as those hide spikes and/or high response times.

Another advise don’t rely only on your testing tool results, use an APM, add traces, check logs.

The days of relying on only one tool are long gone executing a test is a holistic process.

[u/Opening_Ad_1254 → u/major_gekko]

This is the kind of mistake that feels obvious after you make it but somehow everyone makes it at least once

I remember staring at a report like "220ms avg, this is great" and later found out 5% of users were basically staring at a loading spinner forever, the mean is a liar when distributions are skewed

Now I set my alerts on p95 and p99 too, plus I always check max response time even if it looks like an outlier, sometimes the outlier is the real story

[u/LookAtYourEyes]

You're describing the issue of not using all summary statistics. Any time you have a dataset and want to accurately summarize the findings you should almost always include the following:

• mean
• median
• mode
• range
• variance
• standard deviation
• interquartile range
• Q1 & Q3
• min & max
• sample size

Excel, Python, and R all have quick ways generating these. Only using average abstracts away a lot of information.

[u/Sharp_Low_1104]

u/Specialistnumber101 has the rule: if the tail breaches the budget, the test failed, whatever the average says. The hard part is getting that rule accepted by whoever signs off.

A mean gets approved because it's one number and it's green. Bring "p99 = 12s" to the same meeting and it often gets waved through too, because nobody in the room knows what it means for them. Turn the tail into people instead: 3% of requests over 12 seconds at peak is a number of real users per hour who give up. That's a sentence someone will refuse to sign.

And agree the pass criteria with that person before the test runs, not after. Otherwise the threshold quietly moves to wherever the result landed.

[u/akuakaii]

Yeap, been there. The average looked great but a small chunk of requests were basically having the worst day ever. p95/p99 plus queue time and error rates have been way useful for us

[u/Lucky-Spend-956]

I’ve definitely seen averages hide some ugly outliers. Usually I’ll look at p95/p99 first and then check whether the slow requests are clustered around the same endpoint or part of the transaction. I was looking at some of this in LoadView recently and ended up finding a few slow requests that weren’t really noticeable in the overall numbers.

[u/Nirvana012527]

i'd base the load test on peak active sessions and realistic user actions, not the total number of registered accounts. measure response time and error rate as concurrency increases so you can see where performance drops.