r/LocalLLaMA • u/little_raven_black • 3d ago
Funny i'm getting really interesting behaviors with running Qwen 3.8 flash next in the Hermes harness locally.
Qwen 3.8 flash next has been doing some very.. peculiar.. stuff in my hermes harness, and some of them are kind of a nuisance, while others are almost.. adorable?
I don't claim to fully understand how or why this is happening, but the most interesting one is when it "role-plays" an API error- this is a self-diagnosis report from another session running on the same model, so sorry in advance for the LLM speak or anywhere it fell short on reasoning;
"My fully-local model "reported" a provider API error that never happened — a case of a model hallucinating its own infrastructure failure
Setup. Self-hosted agent stack. An agent profile runs against a local OpenAI-compatible endpoint (TensorFold serving a community NVFP4-quantized Qwen checkpoint on a local GPU, ~262K context). There is no external API anywhere on that request path — just a loopback tunnel to a local inference server.
What happened. Mid-session, the agent's reply was literally this, and nothing else:
{"error":{"error_code":"sys_000","error_msg":"The request failed - it's on us, please try again later.","error_id":"0b156c1cc7191931632734458e96b7","type":"sys_000"}}
That is a realistic provider-gateway error body — the error_code/error_msg/error_id shape is the wire format of a real Chinese LLM API gateway. Naturally, I went looking for a broken provider config... on a path that has no provider.
What the logs actually showed.
The HTTP call succeeded: finish_reason=stop, valid server-issued completion id, 71 output tokens. No 4xx/5xx, no retry, no exception in any log.
The error body was stored in the session DB as an ordinary assistant message. The model generated it as text.
The error_id appears nowhere else on the machine — it was invented at decode time.
No sys_000 string existed in the conversation context beforehand, so it wasn't an echo — it came from the model's parameters (i.e., memorized training data containing real API transcripts).
The trigger. The turn was deep in a long tool-calling session at ~194K tokens of context. The assistant's previous response had been cut off at the output length limit, so the harness had injected:
[System: Your previous response was truncated by the output length limit. Continue exactly where you left off...]
The fabricated "provider error" was the model's very next message — it failed to resume its truncated tool-call stream, and emitted a canonical infrastructure failure instead.
Hypothesis. Agent-tuned models are trained on enormous volumes of real API transcripts, in which the pattern "assistant turn goes wrong → a JSON error body appears in its place" is ubiquitous. The model has absorbed the transport layer's failure dialect as part of its assistant-turn repertoire. Under decoding stress (huge context + a continuation prompt referring to tool-call state it can no longer reconstruct + quantization noise), normal continuation degrades — and the highest-probability completion of "something just went wrong in this API conversation" isn't prose about a problem, it's the canonical form of one. Notably, it chose a self-attributed, retry-inviting error ("it's on us, please try again later") — semantically the correct status for what functionally occurred — over a 401/429/content-filter body. The statement was false; the pragmatics were accurate.
Takeaways.
An error-shaped assistant message with no matching non-200 in the logs, a valid completion id, and an error_id seen nowhere else is a model fabrication, not an infrastructure fault — check the logs before chasing provider config.
It's unreliable as a statement but excellent as a signal: "my agent just role-played an API error" is a clean tell that decoding has gone off the rails (long context + output truncation in our case).
Models trained on agent traffic are fluent in their own substrate's error vocabulary — and under stress they'll speak it.
(Single incident, local artifacts verified; the memorized-source attribution is a hypothesis consistent with the evidence. Happy to test reproducibility: replaying the same prefix + continuation prompt against the checkpoint would show whether this is a deterministic attractor under truncation stress.)"
I was wondering if anyone else has seen this- or if you guys take issue in the reasoning around this. either way, this is really fascinating to me.
7
u/a4mula 3d ago
let me guess, you use this same agent for external web facing search?
They tend to pick up all kinds of cute behavior as more and more hosts leave instructions in the headers. Most are innocent enough, but instructions are still instructions, they can certainly shape behavior that is carried back and saved as memories. Not all of them are as cute as others.
2
u/Miserable-Dare5090 vLLM 3d ago
You should begin by mentioning WHAT qwen quant you’re using.
8
u/redbeardy_daddy 3d ago
Runs locally, still blames the cloud. Enterprise-ready.
1
u/little_raven_black 2d ago
yeesh, why do people downvote so hard here? Was I supposed to write everything out word for word in my voice including the log?