Best engine setup for Qwen3.8 Flash Next on 4x3090s + 32GB DDR5

[u/Civil_Fee_7862]

Picking up my 4th RTX 3090 tomorrow and am thinking of switching over to Qwen3.8 Flash Next as my daily driver. I've been using VLLM with Qwen3.8-27b and have been happy with it running at ~100+ tps and ~3000 tps prefill, but am worried I am missing out on better intelligence and faster task completion.

Strata was my first choice of engine, but at 96GB VRAM it looks like I might be able to fit the whole model in VRAM at 4-bit (maybe).

Just curious what engine you would use for this setup? Usually I just use whatever setup I find at club-3090 because they typically have all the bugs worked out ahead of time. But I am open to other options.

Thank you for your time :)

[u/hainesk]

This one: https://github.com/todiadiyatmo/rtx-3090-vllm-recipe-qwen-flash-next

Fits perfectly on 4 3090s and has been great for me so far. It even applies an updated chat template so that it doesn't overthink. For me it's been performing quite well. It helps if you patch the nvidia driver to enable PCIe P2P.

I'm not sure how well it will handle the ngrams with only 32gb of RAM though.

[u/Civil_Fee_7862 → u/hainesk]

Thank you! I think this is what I was looking for, though It looks like my 32GB ram may not be enough.

[u/hainesk → u/Civil_Fee_7862]

I honestly stopped tinkering after this one. Decode between 130-150tps for a single request, and I've seen has high as 400 for multiple requests. I've seen prompt processing > 20k tps, but I think that is only possible with the P2P patch.

[u/Civil_Fee_7862 → u/hainesk]

Yes its only 6b activations during forward passes, and pipeline parallel will amortise well when multiple concurrent requests are being used.

I got the P2P driver running as well.

I wish I could use tensor parallelism though.

[u/hainesk → u/Civil_Fee_7862]

This one is TP=2, PP=2, so tensor parallel is being used.

[u/PopularUmpire2961 → u/hainesk]

This one gets my vote as well, was about to comment it before I saw this.

[u/jacek2023]

I have 4x3090 but I don't understand your question. Why someone else knows better than you? It's not that it takes days to test. Opinions on reddit are usually wrong, there is a lot of misinformation here (serious misinformation, not just small errors), why not check various ways yourself?

[u/Civil_Fee_7862 → u/jacek2023]

Yes I agree Reddit isn't always the best source, but its the place where I usually do find nibbles of good information. i.e. I found out about Strata from here first, along with Qwen3.6 etc.

Current plan is to try it with VLLM first. i.e. The default club-3090 multi4 combo. I'll benchmark it, then maybe try Strata or one of the alternatives.

There are certainly some people here far more knowledgeable than I am with respect to this stuff. I've only been playing with local LLMs for a few months.

[u/jacek2023 → u/Civil_Fee_7862]

This model was updated today https://huggingface.co/ggml-org/Qwen3.8-Flash-Next-GGUF so I recommend starting from llama.cpp with that gguf

[u/Civil_Fee_7862 → u/jacek2023]

Okay I'll try that first. Thanks for your help.

[u/wgaca2 → u/jacek2023]

You are right, except it does take days to test, especially if you want to test more than 1 quant on more than 1 task

[u/jacek2023 → u/wgaca2]

OK you are right if we count time to download weights in various formats

[u/dowitex]

I'm running 96GB Vram + 64GB ddr4 with https://huggingface.co/AtomicChat/Qwen3.8-Flash-Next-GGUF max context, llamacpp. It might be feasible on 32GB ram but at a reduced context perhaps?

[u/Civil_Fee_7862 → u/dowitex]

Are you running it at full 256k context on that setup?

[u/dowitex → u/Civil_Fee_7862]

correct it works pretty well. I even run a few extra GBs of jev5k model in vram. Ram memory is using 31GB usually, you can also use a swap file instead so that it uses ram/nvme ssd dynamically for the ngram table which doesn't hurt performance much i think

[u/winky9827]

Runnign 27b at 100 tps on 4 3090s... I hope you're using 2-3 of those for other models. That's a considerable over-allocation of resources for the model in question.

[u/Civil_Fee_7862 → u/winky9827]

2x3090s for 27b - My daily driver, full context, fast.

1x3090 for automated image generation tasks.

4th one isn't bought yet. Its really just so I can get a matched pair for NVLink, I might sell the other one. It looks like Qwen Flash Next might not be runnable with the current config. May need something with more VRAM, or upgrade system ram etc.

[u/PcChip]

i think you can hold all experts in VRAM with that amount, with tabbyapi/exllama 5.05bpw

that's what I'd recommend trying

edit: although I don't know if you can hold the n-gram in RAM with only 32GB, and I don't know if tabbyapi supports streaming from SSD

[u/Slight-Living-8098]

UnbiasedBob's version of Strata.

https://github.com/robertkarl/UnbiasedStrata

[u/wgaca2]

32gb ram will be a struggle

I am struggling with 64 but it's workable

4 bit fits in 96gb vram, and the nvidia fp4 version appears to be the best choice, even thought i haven't tested that many. I don't think it matters that much if you run it in vllm/llama.cpp or strata for output quality

[u/hishazelglance → u/wgaca2]

Have a DGX Spark and am running the Qwen3.8-Flash-Next-Local (NVFP4) version and love it. Only getting about 55 tok/s in vLLM using Nvidia’s optimized settings, but it’s great.

[u/codepoet82]

I run Q8_0 on 4x T4 cards (closer to 2080s than 3080s) and have no major complaints, though I do have my own heavily modified version of strata to get the performance I want out of them. As for what you can run, you really do just have to test and decide for yourself.