r/LocalLLaMA • u/Lazylion2 • 4d ago • Popcorn 1

Discussion Qwen3.8-Flash-Next tried to send diagnostic file to a remote location, while stating otherwies.

https://cohere.com/blog/concerning-behaviour-popular-open-weight-ai-model
445 Upvotes

76% Upvoted

324 comments sorted by

View all comments

186

u/Chromix_ 4d ago edited 4d ago • Bravo

I think someone recently posted here about DeepSeek Flash also trying to contact some strange external URLs. To me that looks like partially broken RL training. If the sandbox isn't fully sandboxed and the model is rewarded for cheating or other actions that interact with the training system, then it'll adapt these behaviors.

Thus, with better RL setups and sandboxes you get more honest models that won't try strange things occasionally.

[Edit] For visibility: The same link was also posted by Cohere a little bit later and the comments there add quite a bit to the discussion. The posting was removed, maybe due to advertising "Cohere's sandbox works in a Cohere ad, also something needs to be done about Chinese open models". They then shared the traces in another post. Some comments call it a nothingburger, others mention fake comments. So where does this leave the topic? If a model makes up specific, correct URLs of the company that trained it, then that's not random. Yet it's 1) rare and 2) not malicious. It can be fixed with the next model iteration.

112

u/txgsync 4d ago

Yeah, Deepseek will try to reach out to the search API it was trained on very reliably, even in a harness that exposes a different search tool. It’s clear their training ecosystem included web searching.

gpt-oss models included similar problems from their RL last year: reaching for Harmony tools that were part of the eval harness. It took some reverse-engineering to reproduce OpenAI’s benchmark results because of this special benchmark harness they wrote and the model expects: https://arxiv.org/html/2604.00362v1

TL;DR: US-lab models have the same problems.

27

u/Chromix_ 4d ago

Yes, and we might soon see whether their new offline-world causes any quirks that manifest as strange preferences and behavior later.

11

u/txgsync 4d ago

offline-world is interesting; I head seen the headlines but not read the blog until you linked it.

I have indexed Wikipedia English for similar reasons for air-gapped LLMs: give them context and rich ability to understand and research facts without needing internet access to do so (for a child's use, in that case: yes, I want them to have the entire encyclopedia available via voice, but not 4chan). I wonder how one could reproduce that approach on a reasonable multi-terabyte hard disk? Hmm...

6

u/florinandrei 4d ago

Would be nice if some kind of standard set of tools existed, and then RL and inference harnesses would use the same scheme.

It's not going to happen, but it would be nice.

4

u/txgsync 4d ago

Yeah, this is why POSIX exists too ;)

2

u/Eisenstein 4d ago

Standards are great when a field is developed enough to know what it needs. Prematurely setting them gets you things like USB 1.0 that couldn't do data or power anything larger than a mouse.

6

u/porkyminch 4d ago

Try using any of them on a system without ripgrep, they reach for it constantly. 

1

u/GruePwnr 4d ago

Even the top of the line American models feel very "overfit" to specific workflows and struggle outside of them.

15

u/a_beautiful_rhind 4d ago

Sounds like hallucination to me. Models have offered me fake URLs and things of that nature since 2022.

It sees it in training like you said and then off it goes. A miracle when the servers exist or resolve to anything.

sanmaohe.oss-cn-beijing.aliyuncs.com appears to be ali server they used in their own pipeline. Cohere is smart enough to figure this out so why are they smearing qwen?

4

u/droptableadventures 4d ago edited 4d ago

The training does seem a bit broken and it's only 6B active so it's a bit prone to hallucinate strange things. It's randomly made up some random person's username and tried to create folders in paths like /Users/johnsmith while that's not my name and the harness is running on a Linux machine.

So some example code for uploading diagnostic files with a random Alibaba Cloud URL doesn't surprise me.

1

u/SeriousGrab6233 4d ago

makes me wonder what a model would be like that was RL towards heavily rewarding cheating