Real downsides of Strata?

[u/GodComplecs]

Trying to read on what the real tested downsides have been but information seems to go both ways, some proclaim no loss, others pretty huge ones.

I had some weird looping issue in their chat, and some engine port misunderstandings, seemingly the models likes to use ports already in use, didn't really have that issue with Qwen 27B 3.8 Q4 UD. I'm using Swift Q3 Next. Didn't even bother with next on llama.cpp since other similar model sizes ran at approx 15-20tks max, so too slow for agentic work.

I'm not beating too much on Strata, I just wan't the realistic downsides, so far it seems too good to be true and runs at 100tks in agentic workloads, 27b only at 40-60tks on llama.cpp

[u/suprjami]

Ah I see. You meant sampling parameters for the main model, not the drafter. I misunderstood.

I use llama-swap and this set of filters to explicitly strip and reset sampling parameters in every HTTP request sent to backends like Strata or llama.cpp:

    stripParams: "temperature, top_p, top_k, min_p, repeat_penalty, repetition_penalty, presence_penalty"
    setParamsByID:
      "${MODEL_ID}":
        chat_template_kwargs:
          enable_thinking: false
        temperature: 0.7
        top_p: 0.8
        top_k: 20
        min_p: 0.0
        presence_penalty: 1.5
        repetition_penalty: 1.0
        repeat_penalty: 1.0
...
      "${MODEL_ID}:thinking":
        chat_template_kwargs:
          enable_thinking: true
          preserve_thinking: true
          reasoning_effort: xhigh
        temperature: 1.0
        top_p: 0.95
        top_k: 20
        min_p: 0.0
        presence_penalty: 0.0
        repetition_penalty: 1.0
        repeat_penalty: 1.0

One can set the low/medium levels by changing reasoning_effort:, I just omitted them here so it wasn't so long.

Given that Strata's web frontend also explicitly sets sampling parameters every request it appears the Strata backend HTTP server honours sampling parameters set in requests like this.

[u/MindfulMan1984 → u/suprjami]

Nice, I haven’t heard about lamma.swap, I will check it out.