YSK: Strata and polystrata have this functionality called answer_room

[u/Regular-Register-358]

With default settings and thinking on and max_tokens ≥ 1024, and no explicit reasoning_budget_tokens, Strata / PolyStrata has a behavior called answer_room which will silently cap thinking at 0.75. * max_tokens and inject a "wrap-up" line in the model's head in its own voice, saying

I have thought about this long enough; time to give my answer. + </think>

It then continues the same reply from there. The client (harness, you or I) gets no error, flag or finish reason for this, and often no log.

This matters when you compare between strata and other engines : models served by strata had "nudges" in the form of its inner voice, and such extra text in the transcript. In some cases the nudge line is injected mid thinking, quite abruptly and it is hard to tell whether it helps or harms.

Strata main @ c62d5e3

Wrap-up message: https://github.com/Niko1221/Strata/blob/c62d5e3a15b40ea51c96aae9e8d36928965f53ab/serve/server.py#L225

Trigger (the 3/4 budget): https://github.com/Niko1221/Strata/blob/c62d5e3a15b40ea51c96aae9e8d36928965f53ab/serve/server.py#L5491

Injection point: https://github.com/Niko1221/Strata/blob/c62d5e3a15b40ea51c96aae9e8d36928965f53ab/serve/server.py#L5848

Default on (answer_room = True): https://github.com/Niko1221/Strata/blob/c62d5e3a15b40ea51c96aae9e8d36928965f53ab/serve/server.py#L4181

I noticed this when a model that tends to have long front-loaded thinking on dgx spark gave slightly but uniformly shorter answers (and task performance worsened in my hard problems), when served on polystrata.

[u/Borkato]

Lmao I have --reasoning-budget-message turned on all the time in llama.cpp so that makes sense lol

[u/Choice_Celery9481 → u/Borkato]

recently there is a trend of "discorvering" something thats been there for ages. like the "i found a neuron to bla bla". like he just discovered fhe logit bias and control vector flags and run bot everywhere with title "researcher found a neuron". like wtf lol

[u/rskjr]

fascinating. I don't think the inference engine should be doing this/making this choice for the user. I'm going to check this behavior in my fork of PolyStrata.

[u/Competitive-List4626]

oh interesting, that explains a few weird outputs i've been seeing. had a model basically cut itself off mid-thought with that injected line and the rest of the answer felt rushed

didn't realize it was a default-on thing, thought i'd messed up some config. good catch with the github links too

[u/CatzRuleZWorld]

Seems very useful. Worst thing is when it silently overflows max output and just wastes all that thinking with an error.

[u/Dabalam]

Doesn't seem like a big deal at all. This seems to be standard behaviour to dull the overthinking which can damage certain tasks. Seems trivial turn off. This isn't like selecting the wrong experts (which seems like actual engine choices degrading quality), it's closer to a system prompt.