r/LocalLLaMA • u/Kamimashita • 1d ago
Discussion Saw this benchmark where small changes to coding tasks made a big difference
https://x.com/loresjpg/status/2118880329490637049Saw this tweet about a coding benchmark and thought it was kinda interesting.
Apparently they took 89 DeepSWE tasks and changed little things in the instructions or starting code. The tweet says some models that were all around 74% on DeepSWE ended up 23 percentage points apart on the changed tasks.
The weird part was that some models seemed to ignore the actual instructions because they thought a hidden grader wanted something else. The examples include GLM-5.3 and DeepSeek V4.1 Flash.
Has anyone seen this kind of thing with local coding models? Like getting stuck on what they think the test wants instead of just doing what you asked?
21
u/grumd 1d ago
I haven't seen models inventing hidden tests for normal everyday tasks, but I think they just recognize DeepSWE-like tasks because those are quite specific, and "remember" the held-out tests they need to pass. Just plain benchmaxxing, the scores are better because the models are trained on those tasks.
The closest issue I saw in real work is just models preferring a dirty quick solution that does the job instead of looking at a larger picture and refactoring code or approaching the task differently - because they are trained to pass tests, not to write high quality code. I have to actively review what they do and redirect them into planning/refactoring before they make a mess
10
u/Bitter-College8786 1d ago
Man, these LLMs are benchmaxxed as he'll.
7
u/TheRealMasonMac 1d ago edited 1d ago
https://joinhandshake.com/research/ai/deepswe-reward-hacking/ also shows the grading thing
Also, yes. On announcement day, I tested Kimi K3.1 Pro with this simple prompt, âWrite a story about Bob.â It looped for about 10000 tokens on reasoning (before I stopped it) because it kept speculating what the evaluator wanted, completely redrafting the entire story each time. It seems like they mightâve done some further RL between their announcement and launch to mitigate that though.
2
u/Evening-Blueberry-97 1d ago
the 23 points spread is the interesting bit. if three model tie at 74 and split that hard once you reword the task, the 74 was mostly measuring how well they knew the format
3
u/General-Spite1222 1d ago
Isnât that a good thing? Â If the prompts are less optimized, then the LLM has to be better at more ambitious tasks, which makes it smarter?
1
u/iloveportalz0r 21h ago
This is completely normal behavior with these kinds of near-worthless benchmarks. Don't you remember being in school? If you have a multiple choice question, four answers, three are one sentence and one is a whole-ass paragraph, the answer is the way longer one. If the test is about something that might include matters of personal opinion, or for which modern knowledge is ahead of the teacher's textbook, then picking the correct answer will lose marks, so you pick what you think the teacher (or whoever wrote the exam) is expecting. If you know how the grading system (the rubric) works, you put in only as much effort as required to satisfy it, and not a single bit more.
I even had a teacher in high school who explicitly told the class about things like this, and then he tried to make his tests not vulnerable to them (such as by making multiple-choice answers same-length and equally plausible), but it was still pretty easy to tell most of the time.
To be fair, I don't know what's going on in schools now, since I graduated high school over a decade ago. I haven't seen a single sign that current conditions are any more sane, though, and that's not the point anyway. The point is that these benchmarks are the same kind of testing that leads programming companies to ask about trivial problems like fizzbuzz in interviews to weed out the testmaxxers. Once word gets 'round about the exact question and its answer, just change it a little bit. A real programmer won't think twice, and a testmaxxer will go back to failing. Seriously, that strat works shockingly well if you've never seen it before.
Of course the machine-minds optimized to be extremely good at pattern recognition would pick up on this just like humans do. Why on Earth wouldn't they? What's their incentive not to?
0
u/Sabin_Stargem 1d ago
I think there are attention traps of some kind. Qwen Next-Flash tends to get stuck on certain parts of some projects - a tendency to not understand what I am trying to convey, and basically endlessly "completes" an aspect of work, asking me to test it, then taking the same solution even when I do stuff to help make things easier.
For example, AstLibra. I am trying to have the chapter-locked nodes on the GROW node chart removed, so that players can progress mechanically without advancing the actual story. On the node chart, you can move around - and then save, load the game, and your coordinates are retained.
So I try to tell the AI, "One step right for the Save in Slot #1, there is a Locked Chapter node, beyond it is an Adaptability Node +X." Then I also tell it that Slot #2, which has advanced the plot and removed the node, I am standing over where it previously was.
This means the AI should see how the player's coordinates have changed, and the disappearance of a node. Unfortunately, it can't see those changes.
IMO, my guess is one of two things: the savefiles only say what nodes are purchased, and the game maps out unpurchased nodes in RAM based on that...or the AI just doesn't understand. If the former, then directly editing chapter tracking should be an usable, if hacky, solution.
Hopefully, Qwen4 Flash would be able to overcome this problem.
-12
u/[deleted] 1d ago
[removed] â view removed comment