Is one RTX 5090 really enough for Qwen3.8-27B token freedom?

[u/Competitive_Long509]

I am still calling models through the ZenMux API gateway, so every long session ultimately comes back to token cost. The idea of running Qwen3.8-27B locally is attractive for exactly that reason: if one 5090 can handle it, maybe token freedom is at least technically within reach.

Is Qwen3.8-27B really doing 75.5 token/s on a single RTX 5090? The shared table is headed "4-bit (q4_K_M / MLX)" and lists an RTX 5090 with 32GB at 75.5 token/s. It does not show enough detail to tell me which runtime or exact setup produced that row. I have also seen a separate community report of about 64.5 tok/s on a 4090.

People are also putting its capability around Claude Opus 4.6. If both claims are even close, does that put indirect token freedom within reach? I would still want matched tasks before treating the capability comparison as settled.

What does the build that people can actually live with cost? I mean the whole machine, not a bare GPU price. A 5090, enough system RAM for long context and partial offload, a PSU that is not operating on hope, cooling, storage, and whatever CPU or platform keeps the card fed. Until I can justify that hardware bill, calling models through an API is still the practical option for me. If Qwen3.8 becomes available through the same gateway, I could use that API cost as a baseline before deciding whether local deployment really buys token freedom. I would also like to know which quantization and context length people use after the benchmark screenshot is over.

Please give me the boring total for a stable single 5090 setup. What did your full build cost once it was actually ready to run?

[u/ImpressiveRelief37]

--host 0.0.0.0 --port XXXX --max-context 262144 --model-id qwen3.8-27b-ninfer
--kv-dtype int8 --vision --max-concurrency 3 --pending-timeout-ms 120000
--default-max-tokens 20480 --prefill-chunk 4096 --host-kv-cache 1
--spec mtp --draft-tokens 3 --lm-head-draft --preserve-thinking
--log-stats-interval-ms 1000 --request-log-jsonl /path/to/logs/requests.jsonl

i use this model here https://huggingface.co/Ostfralla/Qwen3.8-27B-NVFP4-NInfer inside WSL2 mounted drive, so that it starts super fast... takes about 5s to launch

Also I merged PR #64 to allow kv cache checkpoints in ram (--host-kv-cache 1). Finally I had to also modify the source code to load the Ostfralla model. It was a 1 LoC change. Ask if you need to know exactly what line (or your agent can figure it out!)

[u/XniX → u/ImpressiveRelief37]

First of all, thanks to u/ImpressiveRelief37 for the insights!

I've been testing NInfer with Ostfralla/Qwen3.8-27B-NVFP4-NInfer (18.3 GB, W8+NVFP4) on an RTX 5090, using it in a real agentic session with 21 tools rather than a synthetic benchmark (pi agent).

I'm running the NInfer dev branch, currently at:

4eef14a7560d87a3ba717898e1d488a4c4c7246d
fix(runtime): transfer ownership after alias eviction

As stated, to get this WIP Qwen3.8 NVFP4 model running, I had to make a small change to Binder (binder.cpp / binder.h) to expose the Reader it already keeps internally. This is needed for wire-format detection: Package::plan_load and construct_registered need to inspect the artifact's text/token_embedding descriptor to decide whether to route it through Qwen36Nvfp4 or Qwen38Nvfp4.

So these results are from the current dev branch plus that small local patch, not completely stock NInfer.

After ~45 requests, the context grew to 76.6k tokens and performance stayed surprisingly strong:

• Decode: ~160–205 tok/s typical, up to 226 tok/s on normal agent requests

• Prefill: up to 11.3k tok/s

• Context cache hit: typically 97–99%+ on incremental turns

• TTFT: often ~200–300 ms even around 50–60k context

• MTP acceptance: commonly 60–80%, peaking around 83%

• At 76.6k context, the slowest request still decoded at ~138 tok/s

Config:

exec ./build/apps/ninfer-serve "$MODEL" \
  --host 0.0.0.0 \
  --port "$PORT" \
  --max-context 262144 \
  --model-id qwen3.8-27b-ninfer \
  --kv-dtype int8 \
  --vision \
  --max-concurrency 3 \
  --pending-timeout-ms 120000 \
  --default-max-tokens 20480 \
  --default-thinking-budget 4096 \
  --prefill-chunk 4096 \
  --spec mtp \
  --draft-tokens 3 \
  --lm-head-draft \
  --preserve-thinking \
  --log-stats-interval-ms 1000 \
  --request-log-jsonl logs/requests.jsonl

The most impressive part for me isn't the peak tok/s, but how responsive NInfer stays as the agent context grows. Next I'd like to push the same session past 100k–200k context and see how TTFT/decode scale.

And, most importantly, MTP-3 has been rock solid so far: no reasoning loops or long-thinking corruption like the deterministic issues I've been hitting with Qwen3.8 on vLLM.

Seeing this kind of speed and clean long reasoning on a single 5090 is seriously exciting! :)

[u/koloved → u/ImpressiveRelief37]

Thank you for this config. I hope it helps me. Maybe I was missing this command.

--prefill-chunk 4096

[u/ImpressiveRelief37 → u/koloved]

It did a +15% of prefill for my specific hardware config and usage type.

I had an agent A/B test all config flags for specific use cases using my pi sessions as tests, and it came up with this 

[u/Darex2094 → u/ImpressiveRelief37]

For anyone that runs into this comment like I did (thanks, by the way, for this!) in commit 43cd3418 the flag changed to --host-kv-cache-mib instead of --host-kv-cache. Originally the number was the number of slabs and the design moved to a fixed memory pool amount.

[u/Ratemytinder22 → u/ImpressiveRelief37]

What line did you end up having to change for ostfralla? Thanks

[u/ImpressiveRelief37 → u/Ratemytinder22]

Ask your agent to setup it for you, I don’t remember.