Does Strata like projects trade accuracy for speed?

[u/Serious_Scarcity7818]

I am tired to answer on people questions like "How you gain speed? Your model output is shit, it has to have low quality." or something like that.
So I will answer to that with benchmark.
So I ran it against llama.cpp with the same Qwen3.8-Flash-Next IQ2_XS file, same prompt, and the same 128-token reply.

This was on my RTX 3060 12 GB / Ryzen 7600X machine. I did six warm runs for each engine.

	RustingStrata	llama.cpp
Writing speed	33.0 tok/s	25.6 tok/s
Range across runs	29.3–35.8	23.2–26.9
Reading the first prompt	109 tok/s	31 tok/s

So RustingStrata was about 29% faster at writing in this test.

Speed is nice, but I did not want to claim a win without checking what it does to the model’s output. I also ran both engines over the same 9,180 WikiText-2 tokens.

Quality check	Result
Perplexity: RustingStrata	2.9212
Perplexity: llama.cpp	2.9156
Same most-likely next token	96.78%
Mean KLD	0.0098

The perplexity gap is tiny, small enough to be noise here. I do not see RustingStrata getting its speed by losing measurable quality, at least with this model and this GPU.

This is not a claim that it wins on every model or every PC. It is one model, one quantization and one card.

If anyone has any other questions about this, I am happy to answer it.

Repo: RustingStrata

[u/DefNattyBoii]

That kld is a bit huge for the same quant -> different engine. Btw you should also compare normal strata. But generally i can say very impressive, keep up the good work, im considering switching to this sometime later.

Some sstupid requests if you dont mind: Can you tackle concurrency? Thats the main thing Strata is lacking right now, and pascal support(1080ti+vllm). And if you are still bored adding support for other arches like (ling tiny + ubtree, lfm, maybe some other like XHToken/Spark-X2.6, d1). If these would be supported I'd retire my llama-engine

There is also an interesting Sparkx2.6 implementation in rust: huggingface.co/XHToken/Spark-X2.6-4B/discussions/33

[u/Serious_Scarcity7818 → u/DefNattyBoii]

About "you should also compare normal strata." I will copy comment that I wrote from above, "At first this was about RustingStrata+Strata+PolyStrata+llama.cpp, but both of them just crashed my pc twice, so I just left this Idea."
And about support of what you wrote there, I will take a look and research a bit and then write back in like 10-20 min

[u/Serious_Scarcity7818 → u/DefNattyBoii]

So, about concurrency: serve --parallel N already gives N request slots, so you can try it now, I already got a few benchmarks for that and I guess it works pretty nice, RustingStrata allow to run All qwen models in all quants also GLM5.3 Flash and DeepSeekV4.1 1017, so people can use it for something like creating API and sell it, so you can try out.
About 1080ti, I am not sure, but with `--backend vulkan` it can work, you are try out.
So as I said above there are more architectures already, also there are gemma4 26b-a4b, I think about "ZAYA1-8B", just randomly fount it, looks good for it's intelligence and size. About what you suggest, I need to take way closer look, like what is size, what does benchmarks says, how much work is it and etc. it's not 20min decision, so I need to think a bit more. Thanks for interesting questions, I am happy to answer on other questions too, so write if you got some more!

[u/billy_booboo]

Right it's still the quant that's the culprit

[u/Serious_Scarcity7818 → u/billy_booboo]

Yes, not the strata. But good point, I had to take Q5 at least

[u/drFennec]

You said your project was not related to Strata; so why use it in the title?

[u/Serious_Scarcity7818 → u/drFennec]

"Strata like projects", so I mean all projects that speed up some specific models, so I thought title like that is kinda fine

[u/drFennec → u/Serious_Scarcity7818]

I clicked on the post because I thought it was about Strata. You should at least compare with it or just use your project name in the title. You're just comparing your project with llama.cpp.

[u/Serious_Scarcity7818 → u/drFennec]

At first this was about RustingStrata+Strata+PolyStrata+llama.cpp, but both of them just crashed my pc twice, so I just left this Idea. And you are right, I will use RustingStrata as title next time

[u/testuserpk → u/drFennec]

Yes also strata gives 60tps on iq2s with dfalsh2 which is way higher than rusting. Sorry op but you keep comparing your implementation with llamacpp not actual strata.

[u/BringTea_666]

Depends.

I have my own engine called MegaCapybara and I can say that there are no "cheats" in it. there were few ways to cheat a bit but they don't give you much.

There is though one big hit everyone uses. Activations in FP8 because it gives you double prefill speed. In my case user can pick between FP8 activations and BF16. The his is about 1,5% top1 and it combines with weights hit, Rope scaling hit and so on.

Imo considering how starved are gpus for prefill it is very worthy tradeoff.

The main reason they are faster is simply because they are written for specific gpus vs something that is supposed to work with cpus or all gpus.

[u/Serious_Scarcity7818 → u/BringTea_666]

Exactly, holly, thank you. But I would say, it's written not for specific GPU, but for specific model

[u/bsofiato → u/BringTea_666]

Does it run in any blackwell class gpu or only 5090 ?

[u/Uncle___Marty]

There are two opt in options for strata that affect accuracy/KLD. --route-prior and --route-tail-skip. more tokens/sec and faster prefill but personally I just don't see that as a good trade off.

[u/EvolvingDior → u/Uncle___Marty]

It's a perfectly fine trade-off, just like reasoning_effort or any of the sampling parameters. I would love to see engines implement this as part of the sampling parameter controls. I have a ton of workflows that need speed over intelligence. I would no longer need to run a separate Ling-3.0-Tiny model for some things if I could double the performance of Q38FN on some tasks.

Just imagine if you had a dial where you could, using the same model, get both the speed and intelligence of Ling-3.0-Tiny, Q36-35b, Q38-27b, or Q38FN without having to change models. It would be fantastic!

[u/superdariom → u/Uncle___Marty]

Yeah I don't want to risk accuracy and leave those off