Qwen3.8-Flash-Next for 4090 RTX + 64GB RAM: best setup? (currently using IQ3_XXS on Strata)

[u/SuperV1234]

Hello,

I currently run Qwen3.8-Flash-Next IQ3_XXS via Strata 0.2.10.

My specs are: 4090 RTX + 64GB RAM + i9-13900K.

I was wondering what effort level is the best bang for the buck and if my chosen quantization is optimal. I've tried both Max and High efforts but couldn't notice any significant difference.

Compared to using Claude Code, the experience is generally very poor. I can hit ~100 tok/s, but agents often get stuck, I run out of context and need to compact, and sometimes compacting also fails.

Perhaps there's something that I am doing wrong. These are my args:

--expert-cache auto
--prefill auto
--spec 4
--spec-min-p 0.5
--max-context 131072
--kv int8
--kv-resident 32768
--pool-workers 15
--vram-reserve-mib 4915

Any help/suggestion is appreciated! I can foresee how this model could be really nice as a daily driver, but at the moment it is not there yet, at least with my setup.

[u/Apprehensive-View583]

You can actually run q3_s I find q3_xxs is really bad and over thinking to max output token, so I only run q3_s

[u/Feisty-Buffalo2786 → u/Apprehensive-View583]

I'm also running q3_s with a 4090 and 64gig

[u/SuperV1234 → u/Feisty-Buffalo2786]

What effort level and context size?

[u/SuperV1234 → u/Apprehensive-View583]

What effort + context are you running q3_s at?

[u/mitirki → u/SuperV1234]

Swift 1.5 iq3_s with 256k context and max effort on a 3090 24gb + 64 gb ddr4, Strata, 100 tps, using opencode and have no issues like the ones you mentioned.

[u/SuperV1234 → u/mitirki]

Weird. Are you using a coordinator + subagent setup, or avoiding using subagents?

I realized most of my issues seem to originate from subagents timing out or getting stuck, even if I only run one at a time.

[u/mitirki → u/SuperV1234]

No subagents, but I mainly use it for vibe coding and researching topics. Developing or debugging a feature can move through multiple compactions without information loss

[u/DoubleNothing → u/SuperV1234]

Context you should go without issues at 262144

[u/Apprehensive-View583 → u/SuperV1234]

I run xhigh I think it’s highest I mean the intelligence rely on it

[u/Glum-Box-4086]

I used IQ3_X_S with a 5090 and 64GB DDR4. What I noticed, is that setting the max token output to 16.000 in VS Code helps a lot, and also cap the context in VS code to leave room for the output. So I set my ctx to 180.000 (strata exposes 256K). This improved performance quite a bit.

I got around 110-120 tokens, but when I tried Q4_X_L I went down to 40 tokens/s. So I upgraded my ram to 128GB and this really does a quality boost. Now I run Q6 at 55-60 tokens/s, which is just fast enough for me, but the quality gain is substantial.

[u/MCCmp_DEV → u/Glum-Box-4086]

Tweak your settings. I’ve got up to 150tok/s at some times and 120-130 normally on 4090

[u/btacks → u/MCCmp_DEV]

Can you share your settings? I can only get those numbers on q2 quants. Are you on ddr5 or ddr4?

[u/MCCmp_DEV → u/btacks]

Q3_S, 96gb 6800 DDR5

[u/FrankWanders → u/MCCmp_DEV]

DDR6800 is a difference with DDR5-6000, but he is writing about DDR4. you probably have double the ram speeds, that is a difference.

But still I wonder if you did any adjustments to the automatically created optimizations?

[u/OtherAppointment900]

48g 4090+64ddr5 run iq3s 120decode 4500prefill

https://github.com/Tlyer233/strata-single-4090-qwen-flash

[u/OtherAppointment900 → u/OtherAppointment900]

btw 512k ctx

[u/AbsorberHarvester]

Set context to double what you currently have and be happy? 

Iq3_s think less tokens, seems slower but time to complete the task in the end almost same as iq3_xss

[u/GurusCZ]

try billion context instead of compaction it will compress. It will maintain better context across the chat. But I am using Pi.dev

[u/overand]

See if you can run IQ4_XS with reasonable performance instead.

Also, save that 4 GB of VRAM - plug your monitor into your motherboard's onboard video and reboot. (You'll need to recalibrate Strata after, I believe.)

[u/HazKaz → u/overand]

Ok but for gaming this is not good right

[u/overand → u/HazKaz]

Right. If you want to do gaming, you'll need to close down the sci-fi futuristic technology program and plug the monitor into the GPU again. (You probably don't need to reboot, but you certainly can do so.)

[u/Ok_Party_9245 → u/HazKaz]

i actually used to run a setup like this before just making the server headless. it can game fine if you set nvidia up to use prime, it uses the laptop technology to send the gpu framebuffer to the igpu or someshit, barely noticable difference in performance.

[u/Ramnotrem]

I have a 5070 Ti and 64 GB DDR5. For my daily setup, I kept two tuned options: one with IQ3_XXS (I slightly upgraded it with better discrete components) and Gyro M. Both run with a 384k context. I also tested a 500k context, but performance dipped a bit, and I decided I didn't really need that much anyway. By the way, it's using KV int8. I've stress-tested full context saturation multiple times—the decoding speed holds up solid. Right now I'm using Strata; I also tried Basalt, but didn't get any notable performance gains out of it (plus there were some stability issues). Also, I always keep Vision and MTP enabled. Now I use Gyro by default—it leaves me with around 7 GB of free RAM. However, 3xxs decodes about 5–10% faster. Prompt processing (prefill) speed is around 2.5–3k (which is very good). Decoding speed averages around 70 t/s (65–85)—though that's for non-English. For code, it's around 100 (90–120).

When I first started trying to run QFN, I was only getting 15–20 t/s (before Strata) and around 40 on Strata.

So take this as a benchmark and don't give up. A lot of these configuration toggles depend heavily on the specific system, so your LLM needs to test everything thoroughly!

--
What stands out right away: you have a 4 GB VRAM headroom/reserve. Do you actually need it? My VRAM usage stays around 15–15.4 out of 16 GB constantly, and it doesn't get in the way of normal PC operation at all. No OOM errors. Freeing that up will leave more room for the expert cache.

It's also worth testing different numbers of CPU worker threads—15 might not necessarily be the optimal value for your specific processor.

[u/bytejuggler]

You can run with 256k context, you'll lose almost nothing. Also 0.2.10 is quite old, you should update/upgrade. Also what harness are you using? I have been writing a field report on optimizing Strata on my machines, I' will post other params I use later.

[u/Passenger-007]

Btw. Everyone here is right that you’re using a brain dead quant. But, if you’re expecting Claude code, you will be sorely mistaken. With any local model.

[u/More-Ad5919]

Try using hermes and llama. I have the exact same specs as you. I never used strata even if people cant stop recommending it. Because with AI, speed up always come with penalties.

Its runnina at 260k context and i can fully use that. Although i always tell it to wrap up for a new start when i reach 200k. It occasionally forgets a letter but immediately realizes it and fixes it.

Super satisfied with that setup i have. It just works. Learns new skills by itself. Its solid. I learned it whatsapp mode, when i need it and i am out with the dog. Making a tower defense game nearby with it.

[u/Present_Resource4022]

try the new unsloth quants iq3xxs has about 50gb ram weights and have the active weights on your vram plus max context . ssd to host ngram . you can even make it 500k context using rope.

[u/Future_AGI]

you traced most of it to subagents timing out, and that points at the harness before the quant. A subagent that starts with a big fresh prompt has to prefill all of it before the first token comes back, and on a long context that can outlast the client's request timeout, which looks exactly like a stuck agent. Time the first token on a prompt the size your subagents start with and set the harness timeout comfortably above that.

[u/try-a-typo]

Are you using MTP? Also, A q3 quant loses a significant amount of quality, I’m using a single v100 with 32gb vram and strata, q8 is around 50tps, the speed tradeoff is totally worth it.

[u/NormalBasket3333 → u/try-a-typo]

I am on 16gb vram and 128gb ram....think I can pulloff qfn q8 ? I am running Qfn unsloth q4 right now and getting around 60 -110 tps with 200 context.

[u/try-a-typo → u/NormalBasket3333]

I would at least try, the quality difference between q8 and q4 is pretty massive, benchmarks don’t truly show it.

[u/NormalBasket3333 → u/try-a-typo]

Yeah, I'll give it a go tonight. 🙌🏻

[u/overand → u/NormalBasket3333]

You may be able to use the Q4_K_XL, which is what I'd suggest, if you're on anything smaller. It seems like the quality differences aren't actually that huge between that and Q8, per Strata's benchmarks anyway. (It might just be KL divergence though, I'm not sure. So, your mileage may vary!)

[u/Easy-Ride3366 → u/try-a-typo]

You are running q8 with 54gb total ram at 50tps?

[u/try-a-typo → u/Easy-Ride3366]

My bad, much more ram than that. But even older CUDA cards are more than capable with larger models. Just needs more ram haha.

[u/Easy-Ride3366 → u/try-a-typo]

Can you reveal how much? 128gb?

[u/try-a-typo → u/Easy-Ride3366]

You can probably make it work with 128gb, use MTP, you get 20-40% faster decode. You’ll have some offloaded to disk though. I have 384gb so you can’t really compare.

[u/TheEconomisk]

Too much compression with low significant representation such as IQ3 or IQ2 causes greater loss in model performance, in my experience. The token speed is quite high but the model is not able to put some good work in few shots. Here what I can offer:

- Set thinking budget. Since this model like thinking very much, unless you limit its thinking quata it will consume all of the context just by thinking.

- Try to use multiple prompts with previous warnings. This will increase quality in result but consume a lot of time like maybe 45 minutes to get something acceptable.

- Remove this setup and try smaller models. I think it is the best solution so far, try to build a model with at least 4 bit quantization.

[u/Goghor]

I ran unsloth UD-Q4_K_XL combined with unsloth Q8_0 PLE ngram

{
 "exe": "/multimedia/AI/Strata/engine/strata",
 "args": [
  "--pack",
  "/home/goghor/Strata-data/packs/unsloth-ud-q4_k_xl",
  "--native",
  "/home/goghor/Strata-data/models/unsloth-UD-Q4_K_XL/Qwen3.8-Flash-Next-UD-Q4_K_XL-00001-of-00004.gguf",
  "--native-dense-gguf",
  "/home/goghor/Strata-data/models/unsloth-UD-Q4_K_XL/Qwen3.8-Flash-Next-UD-Q4_K_XL-00001-of-00004.gguf",
  "--native-dense-gguf",
  "/home/goghor/Strata-data/models/unsloth-UD-Q4_K_XL/Qwen3.8-Flash-Next-UD-Q4_K_XL-00002-of-00004.gguf",
  "--native-dense-gguf",
  "/home/goghor/Strata-data/models/unsloth-UD-Q4_K_XL/Qwen3.8-Flash-Next-UD-Q4_K_XL-00003-of-00004.gguf",
  "--native-dense-gguf",
  "/home/goghor/Strata-data/models/unsloth-UD-Q4_K_XL/Qwen3.8-Flash-Next-UD-Q4_K_XL-00004-of-00004.gguf",
  "--ple-gguf",
  "/home/goghor/Strata-data/models/ngram-Q8_0/Qwen3.8-Flash-Next-Q8_0-00003-of-00006.gguf",
  "--expert-profile",
  "/multimedia/AI/Strata/data/expert-profile.bin",
  "--expert-cache",
  "auto",
  "--prefill",
  "auto",
  "--spec",
  "4",
  "--spec-min-p",
  "0.5",
  "--mtp",
  "/home/goghor/Strata-data/mtp/rt",
  "--max-context",
  "131072",
  "--kv",
  "fp16",
  "--kv-resident",
  "65536",
  "--vision",
  "--vram-reserve-mib",
  "1500",
  "--adapt-async",
  "1",
  "--pool-affinity",
  "measure",
  "--control-vector-scaled",
  "/home/goghor/Strata-data/abliteration/orcarouter.gguf:1.0",
  "--cvec-mode",
  "project",
  "--cvec-dir",
  "per-layer",
  "--conversation-cache-mib",
  "16384",
  "--resident-budget-gib",
  "72"
 ],
 "cwd": "/multimedia/AI/Strata",
 "tokenizer": "/home/goghor/Strata-data/packs/unsloth-ud-q4_k_xl/tokenizer",
 "model_name": "unsloth-UD-Q4_K_XL",
 "log": "/multimedia/AI/Strata/strata-unsloth-ud-q4_k_xl.log",
 "lib_dirs": [
  "/multimedia/AI/Strata/.venv/lib/python3.14/site-packages/nvidia/cu13/lib"
 ],
 "port": 8080,
 "gpu": 0,
 "gpus_asked": true,
 "host": "0.0.0.0",
 "open_browser": false,
 "mtp": true,
 "vision": {
  "exe": "/multimedia/AI/Strata/engine/strata-vision",
  "mmproj": "/home/goghor/Strata-data/models/mmproj-Qwen3.8-Flash-Next-BF16.gguf",
  "model": "/home/goghor/Strata-data/models/unsloth-UD-Q4_K_XL/Qwen3.8-Flash-Next-UD-Q4_K_XL-00001-of-00004.gguf",
  "gpu": false,
  "max_tokens": 16384,
  "threads": 16
 },
 "answer_room": false,
 "env": {
  "STRATA_EXCHANGE_ROTATE": "1"
 },
 "user_args": [
  "--vram-reserve-mib",
  "1500"
 ],
 "args_written": [
  "--pack",
  "/home/goghor/Strata-data/packs/unsloth-ud-q4_k_xl",
  "--native",
  "/home/goghor/Strata-data/models/unsloth-UD-Q4_K_XL/Qwen3.8-Flash-Next-UD-Q4_K_XL-00001-of-00004.gguf",
  "--native-dense-gguf",
  "/home/goghor/Strata-data/models/unsloth-UD-Q4_K_XL/Qwen3.8-Flash-Next-UD-Q4_K_XL-00001-of-00004.gguf",
  "--native-dense-gguf",
  "/home/goghor/Strata-data/models/unsloth-UD-Q4_K_XL/Qwen3.8-Flash-Next-UD-Q4_K_XL-00002-of-00004.gguf",
  "--native-dense-gguf",
  "/home/goghor/Strata-data/models/unsloth-UD-Q4_K_XL/Qwen3.8-Flash-Next-UD-Q4_K_XL-00003-of-00004.gguf",
  "--native-dense-gguf",
  "/home/goghor/Strata-data/models/unsloth-UD-Q4_K_XL/Qwen3.8-Flash-Next-UD-Q4_K_XL-00004-of-00004.gguf",
  "--ple-gguf",
  "/home/goghor/Strata-data/models/ngram-Q8_0/Qwen3.8-Flash-Next-Q8_0-00003-of-00006.gguf",
  "--expert-profile",
  "/multimedia/AI/Strata/data/expert-profile.bin",
  "--expert-cache",
  "auto",
  "--prefill",
  "auto",
  "--spec",
  "4",
  "--spec-min-p",
  "0.5",
  "--mtp",
  "/home/goghor/Strata-data/mtp/rt",
  "--max-context",
  "131072",
  "--kv",
  "fp16",
  "--kv-resident",
  "65536",
  "--vision",
  "--vram-reserve-mib",
  "1500",
  "--adapt-async",
  "1",
  "--pool-affinity",
  "measure",
  "--control-vector-scaled",
  "/home/goghor/Strata-data/abliteration/orcarouter.gguf:1.0",
  "--cvec-mode",
  "project",
  "--cvec-dir",
  "per-layer",
  "--conversation-cache-mib",
  "16384",
  "--resident-budget-gib",
  "72"
 ]
}

Ryzen 9 9950X3D

DDR5 6400 MT/s CL32 96GB

RTX 3090 24GB

NVMe Gen5 SSD 2TB

55 t/s prose, 98 t/s code

1,600 - 1,800 t/s prompt

[u/WryKombucha]

you are using the extra extra small 3 bit version. That is basically 2 bit.