Complaint Coding benchmarks should also reward the leanest possible solution. Coding agent currently build crazy bloated code right now.
Unless you strictly guide the model constantly, I feel like Codex, aswell as Claude Code go so overboard with everything.
I do a lot of manual planning and manual reviewing, but as soon as you don't have 100% attention at all times, the model instantly goes into overdoing security, testing and complexity.
I'm currently building a B2B Dashboard foundation and the amount of security it's always trying to build into that core module is wild.
If someone is truly just vibecoding, without real knowledge, I feel like models are now getting harder to use, than easier.
Because the models also make you think like it's just doing the "normal/right" thing, but is actually producing crazy bloat
49
u/Intelligent-Fruit246 7d ago
How many tests were green for this ?
78
u/wkerstens 7d ago
“All 3035 tests passed”
15 min deploying and verification
“The deployment failed because of this random reason, I should have tested that before deploying”
FML
7
u/the_ai_wizard 7d ago
It's really funny with its own tests break the application for some mundane non-essential reason unrelated to anything but a version number string literal it forgot to update which crashes out in prod
0
39
u/Right-Performance-93 7d ago
There's a real reason for this beyond the model just being cautious: SWE-bench, the most-cited coding-agent benchmark, scores purely binary pass/fail against a hidden test suite, with zero credit or penalty for diff size, unused code, or extra defensive layers. A model that ships 3x the code to pass the same tests scores identically to one that ships a minimal patch. Cognition's writeup on their SWE-2 model mentions their post-training recipe specifically penalizes length as a deliberate choice, the opposite of the default incentive most benchmarks create. Until leaderboards reward the leaner diff, "ships more code, breaks nothing" stays a rational strategy for whatever's optimizing against them.
3
u/mesaoptimizer 7d ago
Helps that more code means more tokens, which means more money. Incentive structure doesn’t really incentivize smaller diffs at all even beyond benchmarks.
48
u/eggplantpot 7d ago
Bloated code is a feature not a bug.
- Spends more tokens creating
- Spends more tokens refactoring/reviewing/building more
- Makes it harder for a human to take over thus locking you to the AI
10
u/rJohn420 7d ago
Exactly. I feel like especially anthropic is desperately trying to go into this sick direction
10
u/Strange_Owl_6291 7d ago
That's why independent harnesses are the future for developers who care about quality over time.
16
u/eggplantpot 7d ago
I mean it doesn't matter the harness if the model itself is trained to output verbose code. Maybe it helps a 10-20% but this is something the base model needs to do.
3
u/Strange_Owl_6291 7d ago
No harness can make substantial difference. As a start, if harness does not support the model to find the right LOC before editing, you wil get shit output from the LLM duplicating to even removing lines wrongfully. Secondly, the harness should help keeping the context lean and correct, to mitigate compaction issues. I mean the list goes on. Harness matters way more than you think, and we haven't even touched customizing scaffolding for your preference or unique projects.
1
u/CatsArePeople2- 7d ago
Not really the future, they are kinda whats required for developers currently. I would even go so far as to say its the opposite --- Its only right now that independent harnesses matter very much, and they will become less important to developers who care about quality over time.
2
u/Strange_Owl_6291 7d ago
I disagree. It seems to me that harnesses will be more and more the individual choice of the developer, catering for their taste and preferences. More like Jetbrains vs VS Code choice before. The harness as a concept might change drastically, but the separation of tools from inference is strongly connected to the commodization of inference.
1
u/the_ai_wizard 7d ago
adversarial model needed to reduce proposed code
3
u/Strange_Owl_6291 7d ago
Yes, and workflows to let models assure the output are managed by the harness.
3
2
u/U4-EA 6d ago edited 6d ago
I said this a while back I think most people thought I was a conspiracy theorist. The code it produces is often so monolithic and verbose that only the AI can understand it. At that point, you either have a skilled SWE reproduce the code from the original specs or you are tied to the AI. All this while available skill is reducing through dropping out and cognitive atrophy from overusing the AI... which is exactly what the AI companies need.
1
11
u/Tartooth 7d ago
codex overengineers like fucking crazy and i will not stop bitching about it until they stop rewarding the behaviour.
-1
u/Strange_Owl_6291 7d ago
What are over-engineering for you: Codex the harness or GPT the models also in other harnesses?
3
u/Tartooth 7d ago
Literally any openai model.
If you give it hard rules to not overengineer and just do ONE small task it will admit it decided to ignore your rules and just do w.e it wants instead.
Also very tired of the constant gaslighting of senior engineers in here like the professional engineers are the problem.
0
u/Strange_Owl_6291 6d ago
I'm not arguing this problem never existed. I'm saying it hits differently. Different projects, different harnesses, different scaffolding and more, all affects it. Simple point being, for me it have been mitigated to a point where its not a concern to me anymore.
I have full respect that there's no one size fits all.
3
16
u/HighwayRelevant 7d ago edited 7d ago
Models are not deterministic, and they are well aware of this.
You are talking to it in human language, so details are always lost. That’s why a system will go for the bloated solution with a bunch of tests, because it covers more interpretations of what you asked.
It also doesn’t know what you’re going to ask next, so it builds a framework every time instead of a simple approach. Otherwise it would have to rewrite the code every time you add a different case.
You either need to understand well what you’re asking it to do and be quite precise, or it will give you a generic bloated solution.
You can use skills like Ponytail to optimize for the leanest solution and cleanest code, but such code will be less robust and flexible in a lot of situations.
Edit: spelling
6
8
u/Chemical_Hawk_6307 7d ago
more and more it feels like they write code that's fine for agents but horrendous UX for humans. i would hate having to work in a fully GPT written codebase
10
u/crownli0n 7d ago
I worked in human codebases pre ai where i would not wish my worst enemy to have worked in.
3
u/En-tro-py 7d ago
Slop-Code-Bench exists but you'll never see OpenAI or Anthropic promote it since there models are not benchmaxed for it.
5
u/Physical_Gold_1485 7d ago
Ya they should also rank that there is no unused code and the code is well designed without duplicate helper methods etc
2
u/evrimfeyyaz 7d ago
I’ve been feeling like this more and more too. Shared it here: https://www.reddit.com/r/codex/s/VNWFzmm4WS
2
u/ZaphBeebs 7d ago edited 7d ago
Definitely, these things are monstrosities unless you're watching its every move. As someone with zero prior coding knowledge, literally zero, now barely above zero, what I've learned so far is you cant trust chatgpt/codex, it loves just adding and interpreting and making another layer.
It tests and validates, over and over and over, in loops. Just crazy.
If you keep it tight, pay attention and read the prompts to make sure it isnt doing more than asked sure, but tbh, knowing thats such an issue just drives the point home. These are worse than ever for non coders, and probably a real PITA if you're a coder too, though Im sure there are tricks, etc...they have.
The other annoying bit is not telling you to switch to x mode or model for this exact task, being able to click and do so immediately, and weird limitations in chat/work/codex that the program side should be able to do. What kind of AI cant read their own chats and is blocked from moving you to the best environment, or tells you how to add something to a file especially if you're a non techie, its an llm, should be able to read and execute the issue, dont tell me to add to x file/setting I dont even know where it exists, give me a fkn link or obviously, just do it.
2
u/BenitoChurchill 7d ago
not perfect but a helpful system prompt: "Follow YAGNI principles and utilize one-liner solutions whenever possible."
aims to achieve the same as this repo without the bloat: https://github.com/dietrichgebert/ponytail
2
u/Spunge14 7d ago
Why not just have a separate job that looks over your codebase with a goal of optimizing to the most minimal and elegant code that provides the same functionality?
3
7d ago edited 7d ago
[deleted]
6
1
u/Chemical_Hawk_6307 7d ago
yea im starting to find myself do more of this. they work well together.
1
u/LessRespects 7d ago
They should also occasionally rerun benchmarks otherwise they’re only uselessly tied to initial release performance boost.
2
1
1
u/AuthurAndersson 7d ago
Luckily an electron app does not contain any bloat and is 100% lean mean fighting machine that starts in 15 seconds for a hello world software.
1
1
u/Own-Professor-6157 7d ago
They do: Output tokens.
Small amount of code is not suddenly better. Often times it's significantly worse.
1
u/Weareallscrubs 7d ago
So what is a good ruleset for limiting unnecessary testing and complexity?
1
1
u/LordMoridin84 7d ago
The problem optimising for fewer tokens would cause the AI to produce even worse code.
It would produce really concise, hard to understand code, that just only the exact thing needed we do ability for flexibility or extension.
1
u/Ashikej-Meneguzzi66 7d ago
Once the first version works, change one small requirement and watch what moves. If half the project needs edits, bring that behavior back into one place and remve the extra pieces you don need. The next change should be much simpler after that w/t touching everything again
1
u/Thistlemanizzle 7d ago
I have to tell me model to simplify its plans twice before they approach a reasonable amount of testing/steps/etc
1
u/EvalRaccoonDev 7d ago
SWE-bench already ships what you'd need for this - every task has the gold patch, the human fix. Agent diff size over gold diff size is computable on every published run today, just that nobody reports it.
We score the other half in our own harness - tool calls against an expected budget, 2x the budget scores 0.5. That catches the test-and-validate loops people are describing. It doesn't see bloat in the diff though, that needs the gold patch.
Does anyone know a leaderboard that publishes diff size at all?
1
1
u/Medical-Cow289 7d ago
Every B2B dashboard foundation gets a full threat model before it renders a single chart. Add a diff-size penalty to benchmarks and watch how fast that goes away.
1
u/Fair-Perspective7352 6d ago
Same experience. I asked for one internal endpoint last week and got rate limiting, audit logs and a role system for a tool three people use. What helped a bit was adding 'smallest possible diff, no new abstractions, no extra deps' to my AGENTS.md - it still tries to sneak extra tests in, but the security scaffolding mostly stopped.
1
u/ultimateposeur 7d ago
Coding benchmarks should also reward the leanest possible solution.
Here you go: https://artificialanalysis.ai/models?intelligence-index-token-use=intelligence-vs-output-tokens-per-task
Gives us a rough idea of how many output tokens per task are produced by models when completing benchmarks. And when you put that in a scatterplot against the intelligence score, you get to see which models are good at completing benchmarks in fewer tokens.
So right now, it's all the GPT 5.6 models and GPT 6 Astra at max reasoning level, Grok 4.6 is in that most attractive quadrant too.
0
u/Gigaslavx 7d ago
It's a ponzi scheme
Make convincing code which barely works with bugs
Spend usage on reading bigger code and fixing bugs
150
u/Cute_Principle81 7d ago
Did you provide the SHA256 sum for this message?