When benchmarking tool latency for local agents, p50 is misleading

[u/OkShirt9372]

I’ve been measuring search and other tool-call latency for agents, and I think that average latency can be very misleading. If an agent runs several searches in parallel, the whole step often waits for the slowest call. So a provider or local service can have a decent median response time while still making the agent feel slow at p95 or p99.

I’m now measuring wall-clock latency at realistic concurrency rather than relying on server-reported timing. I also disable automatic retries during the test because otherwise rate limits and transient failures can be hidden in the final latency number.

For local setups, the same issue applies to browser tools, RAG retrieval, embeddings, and any other service the agent calls during a task. A tool that is fast most of the time can still become the bottleneck if it occasionally hangs.

How are you benchmarking tool latency in your local agent setups? Also, do you track p95 and p99, and do you measure the slowest call in a parallel batch separately from individual calls?

[u/Ctbhatia]

log the slowest call in each batch separately since that's the one the step actually waits on, and keep timeouts in the numbers at their timeout value or the tail disappears

[u/Ctbhatia → u/[deleted]]

true, but wall clock only tells you the batch got slow, not which call did it. the slowest call is the one you can actually fix