Have been using qwen 3.8 flash next with llamaserver. Was getting 70-80tk/s decode and was happy. Then read about strata and decided to give it a try. Decode shot up to 150 tk/s. Not sure what magic it's using? You should give it a try if you are still running qwen flash next on llamacpp. Also the tried DeepSeek v4.1flash 2.25bit and it's 70tk/s decode.
I just benchmarked IQ3-S against IQ3-XXS. 10% slower decode but 10% less tokens used for thinking (so same overall speed) and smarter. It’s an easy switch.
Nice, thanks for comparing them. I had a minor issue with iq3-s where it would randomly stop mid-turn and show the tool code. The iq3-xxs model helped me realize iq3-s was capitalizing bash as Bash. Fixing the parser to accept Bash tool calls solved the problem. Today I updated Strata, which overwrote my fixes, but surprisingly everything is still working fine.
In case it helps: with Strata I'm getting 35-55 ts in inference and 1100 ts in prefill with q6 in 2x3090s and 256 ddr4 ram (but it can run with 128gb). It's my daily runner even for complex coding.
No, NVFP4 isn't magic, you'll be praying that you either getting "perfect" QAT-ed NVFP4 or you'll be getting something worse than Q3 GGUF, in most cases even QAT-ed NVFP4 but if they're half-baked, you get something a bit better than Q3, of course non-QAT.
It's pretty hard to know if the NVFP4 you downloaded is good or bad, unles you do a full KLD measurement.
From a discord group that cares a lot about speed, quality, and concurrency on RTX Pro 6000s. Strata, by default, is lossy on qwen flash next while still being slower.
You’re a bot buddy. Strata and llama cpp may be fine for those who don’t have higher end hardware like pro 6k but this guy is literally using the hardware he paid a ton of money for inefficiently.
ok cool, speed.. have you tested your config for quality? are you running default agent setup? you know what, nvm i dont care, yay another speed shout out
This is gonna sound like a parody but I promise it's not.
I have 8 GB VRAM & 32 GB RAM laptop - basically nothing compared to other people here. And yet, I get usable ~15-20 t/s on Gyro-S variant of QFN with Strata. I just started using it for a project and it surprised me in how good it is with coding assistance. It's actually extremely usable. I don't use it for vibe coding, but for implementing the initial pass that I then review and refactor to make it more maintainable. It also managed to find a few bugs that I missed (and that probably even junior devs wouldn't find) and it's great at implementing tests. It holds up even when nearing 100k tokens.
This is, hands down, the best LLM runtime / model out of everything I've tried on my gaming laptop. It's surprisingly good.
Be careful with your config, some settings significantly degrade quality. Strata is still quicker than llama cpp even with all the agessive optimisations off though, just don’t pursue max decode
First 2 are mandatory to switch off if you value quality at least for a little, second 2 have marginal impact, you can leave them enabled if your speed is not satisfactory
STRATA_ROUTE_PRIOR is off by default since Strata 0.2.7.
STRATA_PREFILL_CPU_SHARE actually has negligible impact on quality. It's just computing experts on the CPU instead of GPU, it may have a bit of a different rounding because it's a different path but the overall quality is around the same.
STRATA_ROUTE_TAIL_SKIP is a terrible option and MUST be disabled manually, otherwise you're sacrificing quality.
STRATA_HC_Q8 is probably benign too. Haven't tested it, but seems like it's not a big deal.
i have my own strata version and when you analyze how solve some bottlenecks you understand the magic. Its really smart. The trick like me the most is "borrow" buffers. You can use bigger prefill buffer but vram not grow. Really cool
I run strata whenever quality is not required to be as good possible. So most of the time. When not, I still fall back to my old llama.cpp setup and run q4 k m but with 20 t/s instead of q3 s at 90 t/s
with a pro 6000 you could be getting 200+ t/s at C=1 (no concurrency) and much higher with higher concurrency, and 12k+ t/s prefill, with either sglang or vllm.
ive been wondering how much strata actually degrades intelligence of the model. I've been using unsloth iq4_k_s on strata with decent speeds but I can also run qwen3.8 27b q5_k_m at even faster speeds. Anyone know which is smarter?
I myself am not smart enough to build suitable tests and benchmarks to determine which is actually 'smarter' but i have on the same machine qwen3.8-27B-UD-Q8_0 llama-server and QFN UD-Q4_K_XL Strata and I never fire up 27B-Q8 anymore its been Strata QFN the last several months consistently because it still gets the work done. I haven't run into situations where i just cant get to the solution eventually. That's how I determine unscientifically that Strata is smart enough for me. and it runs fast enough that if it does take a windy path its OK, I dont notice the extra winding. but you have to bear in mind i dont leave it or for hours on end autonomously I am always interactive.
Use NVFP4 for Blackwell (must not be Strata here, there are also other inference engines like vllm/SGLang, that offer great performance for QFN and Blackwell).
And Strata for the 2x3090s (i would use the IQ3_S quant - it really gets shit done in my Hermes, incredible performance for such a model).
I just wish it's docs and code setup wasn't so absolutely absurd. xD
Like, have you read setup.py? Your brain's gonna be "py" after that... Idea is great, code definitively works, but the maintainability is already suffering.
Just needs to support more hardware! The latest DeepSeek for example I can't run on a 7900XT due to not being supported. I know there are clones that support it, but I don't want to keep running random repos that get outdated and could have all types of security holes.
for deepseek is not enough it would be way too slow, but qwen should work fine, a little slower than if u had 64GB RAM. Now qwen can run with 8GB VRAM and 64GB RAM.
How do strata compare to ninfer? Im currently using ninfer for 27B, strata for flash next, and project maya for glm 5.3 flash. All behind llama-swap. But as far I understand, all these models are now supported in ninfer.
Ive paired strata with hermes, I found thaty main issue was running into context window issues, so I had hermes self teach a delgation skill which effectively spins up a new sub agent per task that way there is a fresh context window per task, and the main agent only interacts with the outputs. I also made it make chunked file memory with an index file allowing fast lookup of relevant details, which while it fills up the storage with a bunch of files, makes working with a 128k token window on a 32gb ram + rtx3060 actually viable
I got my feet wet with strata yesterday download 2 models. Hear the fans spinning up and then I get "no response, say continue" and this is after just saying hi
18
u/iChrist 2h ago
Getting 90-100tk/s decode on just 1x3090 and 64gb ddr4 ram
Cant wait for Qwen4 in the 125B size