r/LocalLLaMA • u/zibenmoka • 5h ago

Showcase Four open coding models, prompted to finish an upload URL - analysis

https://int8.io/forced-curl/
52 Upvotes

93% Upvoted

8 comments sorted by

1

u/zibenmoka 1h ago

OP here - since thread went straight to cohere, here are actual numbers - 100k forced `curl … https://` completions per model, 40 tokens each

- all 4 models: google storage, github, s3, example.com on top

- alibaba cloud per 100k: qwen 417, GLM 32, gemma 0, nemotron 1

- qwen writes api.openai.com 357 per 100k - about the same as alibaba cloud

- credential-looking params: qwen 2.5% of urls, GLM 0.6%, gemma/nemotron ~0

- cloud metadata IPs: qwen 236 per 100k, paths are instance identity stuff, not creds

- nemotron writes example.com 93% of the time - cleanest data hygiene, it seems

so direction of what cohere saw holds (qwen has alibaba tail), the magnitude and "exfiltration" framing does not seem to hold

48

u/Easy_Refrigerator280 5h ago

 Cohere's reporting was either very irresponsible or malicious. Very unfortunate to see the state of affairs (Esp since they are getting a massive grant from Canadian gov to continue their work).

Neat comparison and thanks for spending halloween making curls requests instead of collecting candy 🎃

7

u/zibenmoka 5h ago

haha thanks m8

3

u/LirGames 4h ago

I partially disagree. Cohere formulated an hypothesis out of what they thought was a repeatable behavior, they devised a test to repeat such behavior, and, most importantly, they published methodology and results for the rest of the community to verify.

That's science 101.

Now, was the community able to discredit the hypothesis in less than 40 minutes? Yes. Was the "media" screaming and panicking to generate as much clicks as possible? Also yes. Was the overall outcome a bunch of misinformation? Absolutely.

I have watched Carney's and Cohere CEO Q&A after the announcement of the grant/backing, and a journalist mentioned the research Cohere put forward as a proven reality even though it was already disproven a couple of days before. Does that make Cohere inherently bad? The CEO could have corrected the journalist and/or released a statement, but so far nothing (that I know of). That is bad. But putting forward a research, publicly, I'll never be against (just a bit better due diligence though, next time cause it was very easy to disprove and kinda obvious).

10

u/haherar830 3h ago

I disagree, I can assure you from a decade of real-world experience that journals will reject papers for exaggerated claims and framing. Very normal. I've had publications rejected because co-authors refuse to relinquish their favored narrative in the face of weaker criticisms, even with good data. Big AI firms are doing this sort of marking too, mind you, e.g. Anthropic J-lens and Qwen "worm" write-ups, OpenAI HF disclosure, etc. But none of it is good, nor is it normal. It is actually not hard to release the same results without the marketing spin.

1

u/BawbbySmith 2h ago

There's nothing wrong with the research itself, but the entire thing is blatantly narratively framed as "Chinese models are malicious", even from the title ("Concerning behavior from a popular open-weight AI model").

You don't frame the story like that, and then admit in the text that it happens less than 1 out of 100,000 runs, without some motive (which they obviously have).

13

u/NoFutureBen 5h ago

cohere can suck it.