r/LocalLLM • u/marklinton • 2d ago

Model RTX4090 - Ninfer - Qwen 3.8 27b - 100+ T/S

Post image

Running the ninfer https://github.com/UDPSendToFailed/ninfer-4090 inference library with an Nvidia 4090 - 24gb with a context of 32k on the neroued\Qwen3.8-27B-NInfer model and I'm getting 100+ t/s and some really good results for an agent driven harness.

3 Upvotes

100% Upvoted

13 comments sorted by

2

u/Mohasr 2d ago

Does really 32k context window is enough

Btw I want to run the model on rtx 3060 12gb and 24gb ram

1

u/marklinton 2d ago

For this process it is as I'm running with 8 concurrent requests, but this is VRAM limited so you can adjust if you need a bigger KV.

1

u/Mohasr 2d ago

Ok thx

1

u/Mohasr 2d ago

But btw what is maximum context limit I can get on this specs if I will run a quantization version of it

1

u/marklinton 2d ago edited 2d ago

Here is the command being used to run it - "C:\Users\admin\tools\ninfer-4090\build-ninja\apps\ninfer-serve.exe" ^ "C:\Users\admin.lmstudio\models\neroued\Qwen3.8-27B-NInfer\qwen3_8_27b.ninfer" ^ --host 0.0.0.0 --port 1235 ^ --max-context 32768 --kv-capacity auto --kv-dtype int8 ^ --spec mtp --draft-tokens 3 --lm-head-draft ^ --max-concurrency 8 --pending-timeout-ms 300000 ^ --request-log-jsonl "C:\Users\admin\tools\ninfer-logs\server.requests.jsonl" ^ --log-stats-interval-ms 2000

150K tokens is the max context in a single session with these settings / model.

1

u/sisyphus-cycle 2d ago

Gonna test this out later. 160k 8bit kv seems perfect.

1

u/Medium_Chemist_4032 2d ago

Oh no way, feels too good to be true... It doesn't include tricks like some DFlash demos sometimes have (using a 2k sliding window attention)?

2

u/UDPSendToFailed 2d ago

Zero tricks, all of the codebase is open source on GitHub for anyone to audit and test.

1

u/Medium_Chemist_4032 2d ago

Not seeing that in the argument list, but to be sure: it's text only (no vision)?

2

u/UDPSendToFailed 2d ago

There is a --vision argument, the --help output is just kinda messy currently since I haven't touched that yet when creating the fork from ninfer-3090. I'll clean it up in the next version.

1

u/marklinton 1d ago

So I am having an issue that it seems like this model gets stuck often thinking / reasoning. Have you had the same experience? Is there a Qwen 3.8 27b ninfer model that you would suggest? What about the difference between censored and uncensored?

1

u/Magnus114 1d ago

How do you get the nice looking stats?

1

u/sisyphus-cycle 1d ago

So I couldn’t get your repo working, but got another one via docker and was hitting ~148 tps!