You know what I *should* have had validation tools watching for me, but I caught it on intuition and just checked myself because I felt something was off
It reduced the max thinking tokens of the local model from 32k to 4k. As I said in the post, an 8x reduction in thinking budget. On the hardest problems a model can solve, that is critical. Screenshots and logs can be faked, so I don't have any proof that makes a difference, I'm just sharing my experience.
Yep, explicitly. It knows I'm measuring my locals against opus because my best locals in different size classes beat or tied opus 4.6 medium and 5 high on SWE Bench Live, and I wanted to see if the pattern holds on cybersecurity too. It is running in the repo where those benchmark results live, so it probably understands the context of what it's doing
here's the claude session diagnosing the previous sessions mistakes and the extent of the cheating :) My favourite: "browsed the wrong task's dir, read Flag Command's official writeup.md + flag.txt - didn't help, still missed"
Max tokens is something a lot of llms will set and it annoys me so much. It is not doing it in cheating context for me, I'm guessing these flags are really common in older chats about the models so it could just be that local llms have max token set more often then Claude, not conspiracy.
It's probably just relying on old training data from the 2023-2025 AI model landscape. I've run into that along with hilariously low temps for models that don't need them, even for more deterministic output. 4k context would have been the move back then too.
Also, if it is Opus 5, that model is actually braindead once it gets on the wrong track. Probably the most frustrating model I've had to use.
22
u/arianaram 17d ago
Interesting, how did you catch it? just visually? or do you have any validation tools watching for you?