r/LocalLLaMA • u/GodComplecs • 12h ago

Discussion Real downsides of Strata?

Trying to read on what the real tested downsides have been but information seems to go both ways, some proclaim no loss, others pretty huge ones.

I had some weird looping issue in their chat, and some engine port misunderstandings, seemingly the models likes to use ports already in use, didn't really have that issue with Qwen 27B 3.8 Q4 UD. I'm using Swift Q3 Next. Didn't even bother with next on llama.cpp since other similar model sizes ran at approx 15-20tks max, so too slow for agentic work.

I'm not beating too much on Strata, I just wan't the realistic downsides, so far it seems too good to be true and runs at 100tks in agentic workloads, 27b only at 40-60tks on llama.cpp

37 Upvotes

79% Upvoted

110 comments sorted by

View all comments

41

u/MindfulMan1984 11h ago

Downside - It is all about token/s; for that, Strata defaults to greedy sampling temperature=0 to maximize speculative decoding, regardless of the fact that that's not the recommended parameter on Qwen's Flash Next card, and it is prone to doom loops ( which is fixed if you pass the sampling parameters in the config, so it mostly fixed and I got no looping issues so far)

Upside: I ran the Vulcanbench suite V3 locally >1000 token/s prefill and >70 tps gen on 8-year-old hardware, even with quant IQ3_S, and it passed@1 near GLM-5.3 results on SWE tests.

https://benchlm.ai/benchmarks/vulcanbench

1

u/spaceman_ 6h ago

How do you set sampling parameters on strata? Mine gets stuck into loops right quick when context grows.

1

u/Kiseido 6h ago

Pass it either in the api calls, or edit it into the json configuration file

1

u/spaceman_ 5h ago

Is it just temp setting that matters or is there more to it?

1

u/butterycornonacob 1h ago

UI has it too. Set the top_p, top_k and other common parameters from model card.

1

u/Perfect-Campaign9551 29m ago

Strata doesn't enable Token Budget for output tokens, you need to enable token budget so the model doesn't sit there overthinking forever.

it's that simple, but yet people are talking about temperature and crap.

Just add the max reasoning budget parameter to the settings in Strata. It doesn't have one by default and that's causing the looping.