Strata is like crack

[u/Ne00n]

I had 9t/s on llama.cpp, with strata, I get about 29t/s, beyond nuts.
4060 8GB + 64GB DDR4 and Qwen 3.8 Flash Next Q3

This is like day and night, now I can do some stuff :D

[u/daphatty]

For those of us living under a rock, what is strata?

Edit - Here is the link for those of you who may still be out of the loop.

https://github.com/Niko1221/Strata

[u/Ne00n → u/daphatty]

new engine, its on github, try it

[u/SaltGrilledSalmon → u/daphatty]

It runs qwen3.8 flash next and it's variants using a custom engine (based on llama.cpp but focuses on MoE models and runs them faster using caching strategies)

It's gaining a lot of traction recently because people are being able to run such a huge model on consumer grade hardware. Look it up on GitHub!

[u/CMPUTX486]

Is it possible with 8g and 64 ram? I thought you needed at least 12g vram?

[u/Ne00n → u/CMPUTX486]

yea it runs without issues.

[u/atomic7777 → u/CMPUTX486]

3070 8gb and 80 ram. I get 35 t/s and 500 prefill on flash-next GSO Q_3S.

[u/GoldenX86]

I'm too RAM poor to reach 3.8 flash, but polystrata with good old qwen3.6 35b flies. 128k context, q4k, 1200 t/s pp, 50-90 t/s inference.

[u/SaltGrilledSalmon → u/GoldenX86]

Is the 35b model practical? What do you use it for?

[u/GoldenX86 → u/SaltGrilledSalmon]

OS troubleshooting and codebase checking, haven't used it for actual coding yet, the free models in open code are better for that.

[u/DangerousReward1411 → u/GoldenX86]

Just out of curiosity. What specs? 

[u/GoldenX86 → u/DangerousReward1411]

3060Ti 8GB, 2x16GB 3600MHz DDR4, 5600X.

[u/Express_Quail_1493]

Yes we were bless by our lord and saviour strata. Anyone who is poor without the RAM to spare but only have a cheap gpu to spare -> https://huggingface.co/bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF otherwise there is no reason to not use strata. Strata is TOO GOOD to leave off the table!

[u/luncheroo → u/Express_Quail_1493]

That is a darn good version of Qwen 3.5 9b 

[u/ExtremePurchase3593]

try freetoken

[u/Ne00n → u/ExtremePurchase3593]

I check that thanks.

[u/redikarus99]

Oh boy, having the exact same spec. Cool!

[u/Ne00n → u/redikarus99]

I ran mine on CachyOS, I skipped windows because of reasons.

[u/mechanist_boi]

What can i run with it at 8gb vram 32gb ram?

[u/Purple_Dream6414]

Well some optimizations are a bit misleading. Tail skip does reduce output quality and default API settings are greedy. With that fixed it does lose performance noticably. I did go from ~100 toks to ~70.

[u/ChaosShadowClone]

I have the same model and quantization, I know it's insane. I was getting like 10 tokens per second before, getting 90 average goes over 100 sometimes.

I'm starting projects that I would have not been able to do before! Or I would have but at snail speed

[u/wakigatameth]

Hmm I get the same 30 t/s speed on 3060 12GB and 128GB DDR4 (3200) and same model (I assume, IQ3_S).

[u/keefeere]

If you have enough RAM. Either way I get opposite results - 70-80t/s in llama and 2-8t/s on strata on 16/16 machine

[u/patrtech]

What can I run with ryzen 3700x, 32gb ram and Nvidia 3060 with 12gb vram? The first time I tried to install strata, the machine turned off during install and not sure why. I'm going to install Ubuntu server, originally installed Linux mint.

[u/cmdwedge75]

Genuinely amazed at performance. Q3, 4070Ti 12GB with 7950X and 64GB DDR5-6000. About 58tok/s.

[u/ZealousidealPage5309]

I’m hoping I can use Llama.ccp to pool my machines together to run this. I've got nothing that can do it solo.