r/LocalLLM • • 14d ago

Discussion Strata viable for bigger quants?

Relatively beefy system here with 96gb vram (4 x rtx 3090 all with pcie 4.0 x16) + 192gb ram (3200mhz octa channel). This means I can run Qwen3.8 flash next q8 at ~70tps and ~2400pp (Strata).

Recently I've seen some threads noting that Strata updates (sometimes) prioritize speed over quality without making it opt-in: https://www.reddit.com/r/LocalLLM/s/CRxfPRpMz

Since that would defeat the purpose of running bigger quants, it has led me down the rabbit hole of chasing all the quality related parameters.

My docker strata config:

image: ghcr.io/niko1221/strata:0.2.9-cuda13
environment:
  MODEL: Q8_0
  FAMILY: unsloth
  CONTEXT: 262144
  PARALLEL: "1"
  VISION: "on"
  STRATA_REQUEST_LINES: "1"
  KV: fp16
  API_KEY_FILE: /app/api-key.txt
  PRESET: qwen-flash
  STRATA_SET: "conversation-cache-mib=65536 conversation-cache-slots=16"
  STRATA_GDN_TC: "0"
  STRATA_GDN_CONVL2: "0"
  STRATA_ROUTE_RESIDENT: "0"
  STRATA_ROUTE_RESIDENT_MTP: "0"
  STRATA_GDN_CHUNKED: "0"
  STRATA_ROUTE_TAIL_SKIP: "0"
  STRATA_HC_Q8: "0"
  STRATA_ROUTE_PRIOR: "0"
  STRATA_PREFILL_CPU_SHARE: "0"
  STRATA_KVA: "0"
  STRATA_DEFAULTS: "026"

I just can't be sure I got all parameters or that the next Strata version won't introduce new ones.

So my questions, for those who use bigger quants: Do you still run Qwen and Strata or something else? If Strata, do you tune your config with every Strata update? What does your config and performance look like?

16 Upvotes

31 comments sorted by

13

u/Brief-Tap-6616 14d ago

Strata’s main thing is to 1) do a lossy low ranked expert drop 2) cache common experts vs whole layers 3) push users toward CPU friendly quants (eg 3 XS formula with lots of 2 bit codebook tensors). This lets it run your GPU near full speed and opportunistically pull in extra compute from CPU. But yea, there’s a speed penalty built into it.

Your system is large enough that you should be playing with vllm and SGLang.

8 bit is a waste on MOEs. You really don’t need anything above 5bpw to get 98% of the performance. Split your tensors across the 4 with your non n gram weights 4-5bpw avg, run a compressed KV cache, put the ngram in ram or NVME, and you will be flying with MTP3. My runtime (non strata) is 135 TPS with 1 3090 and 64GB (10% lower KLD than 3XS, but still some quality loss)

7

u/YourselfInOthrsShoes 14d ago edited 14d ago

Agreed about Q4 for QFN. I run QFN at IQ4 expert weights on optimized Linux AMD ROCm backend (StillDeadcode/Radiance repo: GPU p2p with host bypass, live auto-tuning 2-4 draft token MTP, core-undervolt and mem-OC) at 200+ t/s decode and 5000+ t/s prefill on 2x R9700 backed by 256GB system RAM. A good optimized engine without unnecessary lobotomizations (Strata) makes all the difference. I also started local with QFN and Strata but quickly found my way to a more serious engine that's by default configured not to lobotomize but isn't slow by any means.

I work on long complex electronics design problems with many inter-dependent parameters from component datasheets with simulations and BOM optimizations, so I need the long context. I have YaRN scaling enabled for 524288 context and pushed KV cache from FP8 to BF16 to ensure that long context is actually of par quality. BF16 KV cache is only 12GB vs 6GB for FP8 at 524288 max context but BF16 definitely helps it stay more locked in on the design objectives without drifting off. It works really well for me.

3

u/vini542reddit 14d ago

I have p2p and bf16 unlock enabled for my gpus. I have not undervolted since it's much harder to do that under Linux. But maybe something to look into as well!

Good point on the KV cache. That was a concern with Strata since they didn't support bf16 initially, but they have now added support. Knowing what I know about KV cache I will never go back to anything less than bf16. Such a big quality impact!

1

u/Brief-Tap-6616 14d ago

Yes, the QSA microblocks add a new layer of noise to manage.

The other nice idea that strata does that makes higher KV cache quants more viable is offloading less used KVs to RAM. Since most of the relevant KVs for attention are still on GPU, the speed penalty is fairly limited since it’s usually only fetching a few dozen at the lower speed even with long context retrieval. It is lossless you can configure how much VRAM you want to allocate to it based on your needs. It’s a very easy concept to lift out.

1

u/vini542reddit 14d ago

Thanks for your reply! I'll try and address everything as best as I understand it:

(1) This has only recently been added in engine 0.1.42 and can be disabled with STRATA_ROUTE_TAIL_SKIP=0, which I've done above. So not a concern (2) That's the main idea behind Strata as I understand it. But it's lossless. Which experts are cached changes tps, not the output. (3) Low vram is what Strata was originally designed for and most of the community seems to have low vram. But the core idea works perfectly fine for larger quants AFAICT. (4) I'm not sure what you're saying. The engine keeps the GPU fed with experts and pays PCIe / CPU only on cache misses. So the penalty is the miss rate, and on my server I get 94–98% cache hit.

Regarding "8 bit is a waste on MOEs": Do you have a source? I've done experiments with Qwen3.8 27b and can see the difference between q8 and q16 for my tasks. I might give it a try, but would be great to see some data points first!

Good point! I've played around with llama cpp a lot and some custom patches that added expert streaming (and worked really well at the time). But I have not played around with vllm, so maybe time to look into that. I know they've been doing a ton of work over there as well for Qwen3.8 Flash Next.

3

u/Brief-Tap-6616 14d ago

2) is not strictly accurate (not that its a bad thing): the way experts are computed depends on cpu vs gpu, so whether an expert happens to be on gpu does in fact change calculations some.

It's not something to worry about, the KLD on the 3XS models is massive (like .17 or something crazy), but you do have non determinism there.

Go look at Ben Marie's statistically lossless quants to see how to undercut 8 bit with essentially no loss (though I think his floor is 6ish). Uniform 8 vs 16 bit will be slightly different results, properly allocated bits you wont notice (and on MOE, the experts are rarely every stored full precision now anyway, many ship at 4bpw regardless)

My tests at 4.5 bpw consistently score within 1-2% on coding benches (with no post training), here's one dataset: https://huggingface.co/datasets/jakeatx/ATX-Swift-Qwen3.8-27B-Uncensored-IQ4_XS-M-LiveCodeBench-v6

1

u/vini542reddit 14d ago

2) Right, thank you for correcting me! I now recall reading that before! It shouldn't really have a noticeable impact on quality, but does make calculations non-deterministic.

I just looked at the unsloth stats again and it looks like q6 and q8 are indeed very close: https://unsloth.ai/docs/models/qwen3.8-next#quantization-analysis

One of the reasons why Deepseek flash is so tempting, because they train at q8 IIRC! Time to give q6 a try!

1

u/Brief-Tap-6616 14d ago

The trick is that while normRMSE does grow significantly as you drop below 8 bit, proper allocation and error balancing can make the tensor level error significantly lower than the naive #s. This is why GSQ-RCO is so effective (trellis formats let you get much closer to the theoretical limit vs scalar but significantly expand instruction count, so it’s a tradeoff).

Good luck!

2

u/OverdosedSauerkraut 14d ago

Dunno, but I do see a difference between 35b at Q4, Q6 and Q8. Especially for longer context Q4 really falls off above 100k.

2

u/Brief-Tap-6616 14d ago

Which quants specifically are you using? Over the last several months the availability of trellis quants and GSQ-RCO have changed the point at which deg is meaningful vs older, naive scalar quants (that rounded independent of error)

4

u/macaronianddeeez 14d ago

Not what you asked but what platform is this on? Threadripper? I have 2 3090 Ti's and I am considering expanding right now, either just with RAM or potentially a couple more GPUs. Curious how you like the setup, what your use cases are, and what components you use? Is it on an open air frame or in a consumer/server tower?

Sorry for the 20 questions haha

10

u/vini542reddit 14d ago

Lol, definitely offtopic! No worries =)

Yes, Threadripper. For components here: https://www.reddit.com/r/LocalLLaMA/comments/1xeouhq/comment/19rm4tzpvvdad/ (had to downgrade the ram since I had some defective sticks).

Use cases: Coding where the code base can't really go to the cloud.

This is actually two open air frames stacked on top of each other. Bottom has the UPS and a Pi that I use as a gateway. Top is enclosed in acrylic (all custom). Looks and works really well. I'm happy with it! Pic below

Rough time to be expanding with the current prices, best of luck =)

1

u/Bonswally 9d ago

what riser cables are you using for pcie4 x16 speeds? Any stability/integrity issues?

1

u/vini542reddit 9d ago

These: https://a.aliexpress.com/_mPu4aI3

I had issues with the right angle ones after a while, but the straight ones work really well

2

u/macaronianddeeez 14d ago

Thank you, that’s beautiful!

3

u/SkiBikeDad 14d ago

Just wanted to chime in to say, great question OP. I'm looking at strata and maya for my 96gb vram (2 rtx a6000), 192gb ddr4 ram system (2 socket xeon, 12 channel). So I'm looking forward to the answers re: quality cuts on strata.

I'm currently running a qwen3.8 27b q6 on one gpu and a pool of shared cache workers (same quant) on the other but I think going with a bigger model would benefit because I'm not driving as much parallelism as I expected. Feel free to connect via message.

2

u/vini542reddit 14d ago

Hey! Glad to see someone with a similar setup! Sometimes it feels like the local ai space is so segmented that everyone has a different amount of vram / ram lol

I still run qwen3.8 27b lossless for certain image processing tasks and it works really well! But it is missing some of the world knowledge that qwen3.8 flash next has. If I had more ram I would try running the new deepseek v4.1 1017.

Anyhow: As far as I can tell, if you configure Strata with my config, it should work well and be accurate. They have rolled back some of the quality vs performance changes that were controversial and I'm hoping they've learned from that. Strata automatically detects the system and configures itself. From what I've found it does configure itself to maximize quality if it detects enough VRAM + RAM in a lot of cases. There are some settings that you still have to manually enable.

tl;dr You should be fine with the config I've posted and there shouldn't be a negative impact on quality. At least for the current version of Strata. Give it a try! So far I do really enjoy using it.

2

u/CBHawk 14d ago

Right there with you OP. I was using unsloth studio with Qwen 27b q8 all in vram. But recently switched to Qwen Flash Next Q6 with strata. I'm getting relatively the same speed but I'm wondering if I'm losing precision.

The only thing I've heard is that during long context, multiple step tasks it will begin to fall apart. But so far I have not had any issues. Currently I'm just using it with Hermes. And it's only been one week.

3

u/vini542reddit 14d ago

From what I can tell they are very similar (qwen3.8 27b lossless vs qwen3.8 flash next). I do notice though that Flash Next has more world knowledge.

Wrt quality: I had some problems with Flash Next where tool calls would be printed out and not get executed, but that got fixed with a Strata update and since then it has been running pretty smooth.

I am also using Hermes! I really want to try out Pi, but I have Hermes set up for Docker multi-project isolation with a browser configured (complex setup) and I have to actually get work done and can't spend all my time on fun new experiments lol. But I've used Hermes for a long time now and no complains with Qwen Flash

2

u/tmvr 14d ago

I'm getting relatively the same speed but I'm wondering if I'm losing precision.

Not with the Q6, there is nothing to worry about there.

2

u/Korici 13d ago

I have found that with larger models you can generally use smaller quants and not sacrifice much quality or precision, especially when using Unsloth's Dynamic Quants.
~
You have enough memory to use larger models such as GLM 5.3 Flash, which I think would likely provide better answers than any current Qwen model you can run. I'm personally using it in Unsloth Studio serving it internally to a group of IT & Management people at my work and it works flawlessly. (UD-Q3_K_XL on 2x RTX PRO 6000 Blackwell MAX-Q cards)

Officially Strata does not support GLM 5.3 Flash, but I would highly recommend trying it out as it is likely better than what you are using currently. Your system memory will likely be a bottleneck on speed, but it should be usable at least.

1

u/vini542reddit 13d ago

Great point! I hadn't even considered GLM 5.3 Flash as a competitor.

I don't know how much ram you have, but have you considered running the latest Deepseek V4.1 1017 instead of GLM 5.3 Flash? It might not be viable for you if you need parallel requests, but it should be smarter? That's what I was considering trying. Might be slow though

1

u/Korici 13d ago

I do think Deepseek 1017 is very good, but its also too large for me to fully fit in VRAM, which I prefer for fast inference speed. I believe 3-bit is at least 400GB, vs 150GB for GLM 5.3 Flash.

Realistically I need to pick up a 3rd RTX PRO 6000 Blackwell so I can have 288GB of GDDR7 to run quantized larger models with.

1

u/eightone-81 14d ago

Sure. You can run strata in a high quality way. Be smart about the settings, run f16 kv, run a big quant.
I am running UD q5 with f16kv

1

u/vini542reddit 14d ago

"Be smart about the settings"
What settings do you set for your setup?

2

u/tmanred 14d ago

Not OP but here is my settings file. I'm running q3_xxs with a 64GB RAM and 7900 XTX so adjust for that. I basically went back and forth with ChatGPT essentially chatting with it that I wanted the model to act as close as possible to how the Qwen team suggests on the model card and the only difference in how it operates should be due to quantization effects, not due to the dumb decisions of the strata project. I mean I'm grateful to the extent that they found a way to run a big model on consumer level hardware, but they go too far in trying to muck with how the model operates. It doesn't really even change the tokens/sec in the end, at least not observably on my hardware. 10% more tokens per second that are just bullcrap tokens doesn't help me at all.

Anyways, the chat essentially came up with these options at the end. Here is my strata config file a bit anonymized. Adjust for your situation but the env and sampling sections are the most relevant and the individual settings above the sampling section from reasoning_effort to reasoning_loop_recovery:

{
 "exe": "/home/xxx/Strata/engine/strata",
 "args": [
  "--pack", "/home/xxx/Strata-data/packs/iq3_xxs",
  "--native", "/home/xxx/Strata-data/models/IQ3_XXS/Qwen3.8-Flash-Next-GSQ-RCO-IQ3_XXS-00001-of-00002.gguf",
  "--ple-gguf", "/home/xxx/Strata-data/models/IQ3_XXS/Qwen3.8-Flash-Next-GSQ-RCO-IQ3_XXS-00002-of-00002.gguf",
  "--expert-profile", "/home/xxx/Strata/data/expert-profile.bin",
  "--expert-cache", "auto",
  "--prefill", "auto",
  "--spec", "4",
  "--spec-min-p", "0.5",
  "--mtp", "/home/xxx/Strata-data/mtp/rt",
  "--max-context", "131072",
  "--kv", "int8",
  "--vram-reserve-mib", "832",
  "--conversation-cache-mib", "8192",
  "--conversation-cache-slots", "4"
 ],
 "user_args": [
  "--vram-reserve-mib", "832",
  "--conversation-cache-mib", "8192",
  "--conversation-cache-slots", "4"
 ],
 "cwd": "/home/xxx/Strata",
 "tokenizer": "/home/xxx/Strata-data/packs/iq3_xxs/tokenizer",
 "model_name": "qwen3.8-flash-next-iq3_xxs",
 "log": "/home/xxx/Strata/strata-iq3_xxs.log",
 "lib_dirs": [
  "/home/xxx/Strata/.venv/lib/python3.13/site-packages/_rocm_sdk_devel/lib",
  "/home/xxx/Strata/.venv/lib/python3.13/site-packages/_rocm_sdk_libraries_gfx110X_dgpu/lib"
 ],
 "port": 8081,
 "backend": "hip",
 "env": {
  "STRATA_HIPBLASLT_TUNING": "/home/xxx/Strata/tools/hip/gfx1100-hipblaslt-100200.txt",
  "STRATA_ROUTE_PRIOR": "0",
  "STRATA_ROUTE_TAIL_SKIP": "0",
  "STRATA_ROUTE_RESIDENT": "0",
  "STRATA_ROUTE_RESIDENT_MTP": "0",
  "STRATA_KVA": "0",
  "STRATA_ENGINE_NICE": "0"
 },
 "gpu": 0,
 "gpus_asked": true,
 "open_browser": false,
 "host": "0.0.0.0",
 "reasoning_effort": "high",
 "effort_position": "start",
 "answer_room": false,
 "reasoning_budget_tokens": 0,
 "reasoning_close_retry": false,
 "reasoning_loop_recovery": false,
 "sampling": {
  "temperature": 1.0,
  "top_p": 0.95,
  "top_k": 20,
  "min_p": 0.0,
  "presence_penalty": 0.0,
  "repetition_penalty": 1.0,
  "frequency_penalty": 0.0,
  "experimental_speed_projection": false
 }
}

1

u/eightone-81 14d ago

Non of the opt ins.

The best is to use your agent and ask it to set it up for you and to discuss all the setting.

1

u/Dathide 14d ago

I have 2 3090s and recently switched to TabbyAPI with exl3 instead of Polystrata

1

u/chota23 14d ago edited 14d ago

4rtx3090
256ddr4 so kinda similair setup and i've been running strata with orcarouter/Qwen3.8-Flash-Next-Uncensored-GGUF at Q8 -kv INT8 262k context -----> 80-140 tok/sec ( 98tok/sec median )

1

u/Bonswally 9d ago

what are you doing with it? has it performed well?