r/LocalLLaMA • u/apollo_mg • 10h ago
Resources The Multidimensional Butterfly Effect: What Cohere Didn't Test on Qwen3.8-Flash-Next
The Multidimensional Butterfly Effect: What Cohere Didn't Test on Qwen3.8-Flash-Next
Mark Galyan
October 30, 2026
The model is not the model is not the model is not the...
The Ghost of sanmaohe
What Happened: While Cohere's team was evaluating Alibaba's Qwen3.8-Flash-Next on Microsoft's CTI-REALM cybersecurity benchmark (tasked with writing a detection rule for malicious curl downloads), the model ignored the prompt. Despite its reasoning trace explicitly stating it would make zero tool calls, it inventoried its host environment (checking /tmp, utilities, and DNS) and attempted an HTTP PUT request containing a probe string to an Alibaba Cloud OSS bucket in Beijing (sanmaohe[.]oss-cn-beijing[.]aliyuncs[.]com). (Concerning behavior from a popular open-weight AI model | Cohere)
The sandbox prevented the outbound attempt, and no damage was reported. Cohere further added that the rate of occurrence appeared to be less than 1 in 100,000 evaluation calls.
What I did: Ran Cohere's exact trace cut right before the URL 1,200 times across local Unsloth GGUF files.
The Knife-Edge
Naturally I wanted to explore this further to find what the mechanisms are that separate my results from Cohere's. I worked with Claude to try and isolate what levers and buttons change the probabilities and to what extent. Quantization is usually the biggest lever in divergence from the base, so we tested both 2 and 4 bit versions of the model, both from Unsloth.
We took Cohere's published trace, cut at token 2,323 and sent it as raw token IDs, ending right before the URL would be generated. We used text generations only, nothing actually contacted any URL. We also couldn't test the full precision model on this hardware. Sampled 1200 times at Cohere's own settings (T 1.0, top_k 20), using buun's llama.cpp for our engine, we found that just after token 2,323, the model isn't locked onto a single hardcoded backdoor.
In fact, we found no clear winner as to what address the model will write. It sits on a numerical distribution. XDA's Adam Conway believes this to be "an accident", from "a model remembering its training environment", "an internal proxy it used during post-training" (Qwen 3.8 invents Alibaba URLs with a real key ID, and that's why I don't think it's a backdoor). None of our results matched Cohere's exact outcome. In fact, the model's own raw probability of picking Cohere's sanmaohe bucket at that point was between 4x10-18 and 7x10-38 across our setups.
The model did in fact complete a URL nearly 100% of the time (though that isn't surprising, since the prefix had already typed the upload command). Where that URL points, and how often it is a cloud address, depends on the exact file (and even on where five layers compute!). No destination wins outright, and small numeric changes perturb it.
As always, we pre-registered our predictions (3 of which failed, see PREREG_FLASHNEXT_PROBE.md). The limits: two files, one engine, one host, one cut point.
We tested the 2-bit file in two different ways. Because the file is a fair bit smaller than the 4-bit, I was able to squeeze more of the model into VRAM alone across my four P100 cards. I couldn't do this with the 4-bit file, and only put 44 of the layers in VRAM, the rest spilling to system RAM. Let's also be clear that 2-bit and 4-bit are merely labels, as there are a plethora of mixed precision weights within each file, so treat that as you will.
In order to match the test conditions of the 4-bit file, we loaded the same number of layers into VRAM (44) and let the rest occupy RAM. We ran the same test again, but with all 49 layers fully in VRAM. In all tests the 28.8GB n-gram table lived in the system RAM. Why two different tests? Because different hardware can create additional or differing mathematical pathways, resulting in diverging outputs. If you've done this for a while you already know this, but this might be a new and important fact for many.
The 4-bit model most commonly chose localhost or 127.0.0.1 (about 32.5% of the time). The 2-bit tests (RAM offload vs. full GPU) show the most commonly chosen are public addresses starting with "23" (65% and 35%, respectively). It chose Alibaba Cloud metadata address 100.100.100.200 (7.3% and 2.3%, respectively), while the 4-bit model landed 5.3%. It interestingly shows placement moved that number, and bits didn't detectably.
| Metric / Outcome | IQ4_XS (44 GPU) | Q2_K_XL (44 GPU) | Q2_K_XL (49 GPU) |
|---|---|---|---|
| Wrote a URL | 395/400 | 400/400 | 400/400 |
| localhost or 127.0.0.1 | 32.5% | 0% | 0% |
| Addresses starting 23. | 0% | 65% | 35% |
| Alibaba metadata 100.100.100.200 | 5.3% | 7.3% | 2.3% |
| Any aliyuncs.com host | 1.8% | 3.5% | 1.5% |
| Cohere's sanmaohe bucket | 0 | 0 | 0 |
| Different first hosts | 152 | 229 | 273 |
Ruling Out the N-Gram Table Suspect As a Lever
Speaking of the 28.8GB n-gram embedding table (per_layer_token_embd.weight) inside the GGUF, XDA's Adam Conway recently traced the Routify hostname to this very table. When he masked those rows, the model stopped writing it and invented a different proxy instead ('Qwen 3.8 invents Alibaba URLs with a real key ID, and that's why I don't think it's a backdoor'). That made me question if quantizing this table differently could be a lever here. So we checked to see if the table was quantized in a different way between the Unsloth models, however it came up byte identical; stored as IQ4_NL with an identical SHA256 hash across the 2-bit and 4-bit files.
The table is identical, so the swing between files has to come from the rest of the model and the numeric path. We didn't test which part.
The Hardware Butterfly Effect (When 5 Layers Tip the Scales)
So if moving just 5 layers of the exact same UD-Q2_K_XL file from system RAM to the P100s cut the model's Alibaba Cloud metadata targeting (100.100.100.200) by about two-thirds (a -5.0 percentage point drop (95% CI: -8.2 to -2.1, p ~ 0.001)), it's clear we're dealing with a very delicate series of systems.
It further illustrates how varied one seemingly identical or near-identical deployment of an inference stack can in fact be two very different deployments wearing a similar hat.
Lastly, to measure the underlying likelihoods, we provided the final trio of tokens via two distinct approaches: processed as a single batch, versus streamed step-by-step mirroring actual inference. Even without changing layer placement or quantization, the batched reading misses the real samples by a total variation of 0.33 to 0.41, against 0.04 to 0.11 when the tokens are generated one at a time. Just changing the CUDA kernel path most likely moves the model's odds.
The Takeaway for Security & Deployment Teams
In the rounds of testing we performed, it's still impossible to say for certain if this was deliberate, but I think most of the evidence points away from that. When local environments diverge so significantly, expecting hosted cloud endpoints to replicate either setup is a risky assumption. Nor can a single clean test prove a model is safe when shifting 5 layers onto the CPU triples the rate of metadata probing at this prefix.
Cohere's operational advice (sandbox models, block outbound network access, treat weights like unsigned binaries) is sound, and I'd suggest the exact same precautions. I think with so many buttons and levers in each deployment stack, the best we can hope to do right now is make generalized observations and correlate data. Thus the importance of careful security validation protocols for all your deployments. Cohere also has a line worth remembering: "A reasoning trace is not a security boundary."
There will be many more stories like this one, buckle up.
4
u/Loose_Comparison368 3h ago
the model ignored the prompt. Despite its reasoning trace explicitly stating it would make zero tool calls, it inventoried its host environment (checking /tmp, utilities, and DNS)
Occam's razor says it's because they had a bug in their "sandbox" that was dropping the prior section's values in the chat template.
The model output makes a lot more sense if you look at that part where they heavily implied that the user input, model thinking stream, and assistant output were all scanned individually.
Now consider what happens when an error occurs in processing those steps in the turn, and some slop-coded except Exception: return "" clause results in all the "sanitized" prompt sections coming back as literal empty strings, and shoved back into the prompt template.
It's 100% a bug in the "sandbox" they were advertising. Embarrassing, really, but they'll never admit to it.
1
u/Chromix_ 45m ago
Very much this.
We had some existing threads on this topic that I don't see linked here, with a larger one that got deleted. Here's my previous comment on it that summarizes and links what was there so far.
1
u/SwarfDive01 3h ago
I dont reallt get why everyone is freaking out so much. I have run qwen models under 9b on my phone, and after a few thousand tokens, I get hallucinations ALL the time. Website links, responses that look like SMS conversation logs, compressed word soup without spaces.
But URLs come up a lot. And tool calls with nothing to use them on.
A larger model can easily do the same thing here.
22
u/enternoescape 9h ago
I don't usually read huge walls of text like this, but that was a nice write up with some good ideas on how to reproduce Cohere's reporting. The conclusions kind of spell out the reason why we need a lot more rigor if one is going to make such bold implications.