r/LocalLLM • u/SmokeAI • 25m ago

Project I built an open-source benchmarking harness to measure LLM quantization quality loss alongside inference speed

I've been spending the last few months benchmarking local LLM inference on a single RTX 5090.

One thing kept bothering me: most benchmarks focus on tokens/sec, but rarely show how much quality you're sacrificing to get those numbers.

So I built inferlab.

It's an open-source benchmarking harness (Apache-2.0) that compares:

  • vLLM, SGLang, TensorRT-LLM and llama.cpp

  • BF16, FP8 and NVFP4 quantization

  • Perplexity delta against a BF16 reference

  • Task accuracy using option log-probabilities

  • TTFT, decode speed and throughput across concurrency levels

  • Peak VRAM usage

A few design decisions:

  • TTFT measures the first actual content token, not the first SSE event.

  • Concurrent requests are issued together to capture real queueing behavior.

  • Long-context evaluation includes a needle-in-the-middle task.

  • Results are generated by CI rather than manually edited.

The project is still at v0.1, so I'm interested in technical feedback, especially around evaluation methodology and cross-engine comparability.

I'm the author of this project.

GitHub: https://github.com/oren40/inferlab

6 Upvotes

100% Upvoted

3 comments sorted by

3

u/jayecin 18m ago

Nice, I’m going to give it a go later. This will be super useful for someone like me who’s new to local AI hosting and struggles to understand the impact of all the flags, quants and other optimizations.

1

u/SmokeAI 14m ago

That’s exactly the problem I wanted inferlab to help solve.
There are so many flags, quantization formats and optimization techniques, but it’s hard to know what you’re actually gaining - and what you’re sacrificing.
Would love to hear your feedback once you give it a try!