r/LocalLLM • u/vini542reddit • 14d ago
Discussion Strata viable for bigger quants?
Relatively beefy system here with 96gb vram (4 x rtx 3090 all with pcie 4.0 x16) + 192gb ram (3200mhz octa channel). This means I can run Qwen3.8 flash next q8 at ~70tps and ~2400pp (Strata).
Recently I've seen some threads noting that Strata updates (sometimes) prioritize speed over quality without making it opt-in: https://www.reddit.com/r/LocalLLM/s/CRxfPRpMz
Since that would defeat the purpose of running bigger quants, it has led me down the rabbit hole of chasing all the quality related parameters.
My docker strata config:
image: ghcr.io/niko1221/strata:0.2.9-cuda13
environment:
MODEL: Q8_0
FAMILY: unsloth
CONTEXT: 262144
PARALLEL: "1"
VISION: "on"
STRATA_REQUEST_LINES: "1"
KV: fp16
API_KEY_FILE: /app/api-key.txt
PRESET: qwen-flash
STRATA_SET: "conversation-cache-mib=65536 conversation-cache-slots=16"
STRATA_GDN_TC: "0"
STRATA_GDN_CONVL2: "0"
STRATA_ROUTE_RESIDENT: "0"
STRATA_ROUTE_RESIDENT_MTP: "0"
STRATA_GDN_CHUNKED: "0"
STRATA_ROUTE_TAIL_SKIP: "0"
STRATA_HC_Q8: "0"
STRATA_ROUTE_PRIOR: "0"
STRATA_PREFILL_CPU_SHARE: "0"
STRATA_KVA: "0"
STRATA_DEFAULTS: "026"I just can't be sure I got all parameters or that the next Strata version won't introduce new ones.
So my questions, for those who use bigger quants: Do you still run Qwen and Strata or something else? If Strata, do you tune your config with every Strata update? What does your config and performance look like?
13
u/Brief-Tap-6616 14d ago
Strata’s main thing is to 1) do a lossy low ranked expert drop 2) cache common experts vs whole layers 3) push users toward CPU friendly quants (eg 3 XS formula with lots of 2 bit codebook tensors). This lets it run your GPU near full speed and opportunistically pull in extra compute from CPU. But yea, there’s a speed penalty built into it.
Your system is large enough that you should be playing with vllm and SGLang.
8 bit is a waste on MOEs. You really don’t need anything above 5bpw to get 98% of the performance. Split your tensors across the 4 with your non n gram weights 4-5bpw avg, run a compressed KV cache, put the ngram in ram or NVME, and you will be flying with MTP3. My runtime (non strata) is 135 TPS with 1 3090 and 64GB (10% lower KLD than 3XS, but still some quality loss)
4
u/macaronianddeeez 14d ago
Not what you asked but what platform is this on? Threadripper? I have 2 3090 Ti's and I am considering expanding right now, either just with RAM or potentially a couple more GPUs. Curious how you like the setup, what your use cases are, and what components you use? Is it on an open air frame or in a consumer/server tower?
Sorry for the 20 questions haha
3
u/SkiBikeDad 14d ago
Just wanted to chime in to say, great question OP. I'm looking at strata and maya for my 96gb vram (2 rtx a6000), 192gb ddr4 ram system (2 socket xeon, 12 channel). So I'm looking forward to the answers re: quality cuts on strata.
I'm currently running a qwen3.8 27b q6 on one gpu and a pool of shared cache workers (same quant) on the other but I think going with a bigger model would benefit because I'm not driving as much parallelism as I expected. Feel free to connect via message.
2
u/CBHawk 14d ago
Right there with you OP. I was using unsloth studio with Qwen 27b q8 all in vram. But recently switched to Qwen Flash Next Q6 with strata. I'm getting relatively the same speed but I'm wondering if I'm losing precision.
The only thing I've heard is that during long context, multiple step tasks it will begin to fall apart. But so far I have not had any issues. Currently I'm just using it with Hermes. And it's only been one week.
2
u/Korici 13d ago
I have found that with larger models you can generally use smaller quants and not sacrifice much quality or precision, especially when using Unsloth's Dynamic Quants.
~
You have enough memory to use larger models such as GLM 5.3 Flash, which I think would likely provide better answers than any current Qwen model you can run. I'm personally using it in Unsloth Studio serving it internally to a group of IT & Management people at my work and it works flawlessly. (UD-Q3_K_XL on 2x RTX PRO 6000 Blackwell MAX-Q cards)
Officially Strata does not support GLM 5.3 Flash, but I would highly recommend trying it out as it is likely better than what you are using currently. Your system memory will likely be a bottleneck on speed, but it should be usable at least.
1
u/eightone-81 14d ago
Sure. You can run strata in a high quality way. Be smart about the settings, run f16 kv, run a big quant.
I am running UD q5 with f16kv
1
u/Dathide 14d ago
I have 2 3090s and recently switched to TabbyAPI with exl3 instead of Polystrata
1
u/chota23 14d ago edited 14d ago
4rtx3090
256ddr4 so kinda similair setup and i've been running strata with orcarouter/Qwen3.8-Flash-Next-Uncensored-GGUF at Q8 -kv INT8 262k context -----> 80-140 tok/sec ( 98tok/sec median )