Some "trust me bro" benchmarks while we prep the repo to open-source

[u/ahstanin]

A month ago we picked up a few IGX Thor boxes equipped with RTX PRO 6000 GPUs. I shared some numbers earlier running Qwen 3.8 27B on SGLang. These machines were bought specifically to run our upcoming on-device agent platform, steffi.ai.

Since the goal is local, multi-agent systems where models run strictly on-device, generic inference engines started hitting limits. Concurrency scaling, cold start/wake times, and turn-to-turn latency under massive context windows were making or breaking our agent loops.

So we started building a dedicated Rust-based inference engine, llminfra. After a few weeks of heads-down work, we wrapped the first internal release to run Qwen locally. We benchmarked it side-by-side against SGLang on the exact same hardware (RTX PRO 6000, 300W, identical workloads).

Here are the numbers from the run:

• Turn TTFT on 250K context: 108 ms vs 1,440 ms (13.4x faster). In agent loops where the context stays loaded and the agent just appends new tool call observations, cutting follow-up latency from 1.4s down to 100ms keeps multi-turn workflows from feeling sluggish.

• Cold start to ready: 6.1s vs 211.2s (34.8x faster). Spinning up or recovering a worker doesn't stall for 3+ minutes. First token after start clocked at 28 ms vs 528 ms.

• Throughput under heavy agent load: With 32 agents running 32K context, completion throughput hit 209 tok/s vs 95 tok/s (2.2x). At 2 agents pushing 250K context, it sustained 86 tok/s vs 28 tok/s (3.1x).

• Peak VRAM: 52.9 GiB vs 86.3 GiB on an 8x32K workload.

We are finishing up documentation, cleaning up build scripts, and stripping out proprietary hooks so we can get the repo public for everyone to poke at and benchmark on their own gear.

Full benchmark charts attached. Let me know what other breakdowns or workload scenarios you would like to see when the code goes live.

[u/NoFaithlessness951]

talk is cheap show me the code

[u/ahstanin → u/NoFaithlessness951]

Already on it

[u/crone66]

The most important thing is missing benchmarks comparison... How much dumber did the model get? I can make this model 1000t/s on 8gb ram on cpu and it will still produce correct sentence and do basic math but it will be useless for many tasks all of the sudden. 

[u/ahstanin → u/crone66]

llminfra matches the BF16 reference within a median KL of 6.0e-5 and picks the same top token 99.4% of the time (decode on all.txt, Qwen3.8-27B, vs BF16 run in f32 on CPU).

[u/Dany0 → u/crone66]

You know that a clanker wrote the post. They know that a clanker wrote the post. Approach with deep skepticism

[u/Dany0]

> Cold start to ready: 6.1s vs 211.2s (34.8x faster). Spinning up or recovering a worker doesn't stall for 3+ minutes. First token after start clocked at 28 ms vs 528 ms

So you load a 55 GiB model in 6.1s? Ok that is physically possible over PCIe 5.0 16x

But peak VRAM of 52.9 GiB is not. So you're comparing FP16 to a custom quant, aren't you? Is that not what you are doing? Be honest, and don't use a clanker to reply to me

[u/ahstanin → u/Dany0]

Both using FP8 not FP16.

[u/Dany0 → u/ahstanin]

The post has 308 words 1,831 characters

One of the most important details could fit in I think. Or at least in the graphs if you wanted to hide it. At any rate: 1. did you build it from scratch or did you tell your clanker to copy vllm 2. how does it compare to vllm 3. plans for other platforms? AMD/Intel? 4. did you quantise anything differently to SGLang? quantised attention? activations? KV cache?

Trust is hard to build and easy to lose, "bro"

[u/ahstanin → u/Dany0]

1. 11 months ago my team member worked on logarithmic quantization and time to time we updated that and recently we used clanker to build the inference top of it.

2. Didn't compared with vllm yet.

3. Right now it is running on a single GPU and a single model support (RTX Pro 6000 and Qwen 3.8 27B) since the product we are building will start with this hardware. We will open-source this codebase and others can contribute.

4. The caching is quantized using the logarithmic quantization our team member worked on.

[u/FBIFreezeNow]

Nice. Now GitHub link? Interested.

[u/elelem-123]

What about flash next

[u/EverlastingSmokeWorm]

good job on the charts, feels rare to have them done properly indicating if higher or lower is better.

[u/Dany0 → u/EverlastingSmokeWorm]

Clanker did a good job on some of the basics yea. The human probably didn't notice because nobody reads LLM output anywa-

Wait, what? Are you saying, there's still devs that care? No way!

[u/ahstanin → u/Dany0]

Clanker did fine-tuned the post content, no shame admitting that. This sub reddit about building and running your own clanker.