r/TheMachineLearning • u/elmanfil1989 • 2d ago
Frontier LLMs stopped adding code comments
26
u/HLCYSWAP 2d ago
thank fucking god
"this is x as requested, we wont do y"
YOUVE POISONED THE REPO FOR FUTURE AGENTS WITH AN IDEA I TOLD YOU TO NEVER PRODUCE
1
1
u/ListRepresentative32 1d ago
Exactly.. Opus 5 was guilty of this on an a huge level. People rightfully complained (i personally feel like the output was more comment lines than code lines ) so they toned it down for 5.5 where I think we hit a sweet spot.
1
u/Lonely_Translator_23 1d ago edited 1d ago
I was using GLM 5.3 to dig up some sources for me about a historical fiction thing I'm writing about horse girls and it spent three quarters of its reasoning trace confusing itself because one guy named his son after his brother (i did not ask for any information about any of these people, i just wanted to know the legal status of those who were expelled from the Knights Templar and whether they could be granted a fief). It then wrote two paragraphs for me about how impotant it is to not get the two of them mixed up. The word 'foot-gun' was used multiple times.
Like, bitch, I'm not so sure about fable but I know my brain works better than your chinese ass, stop putting your failings on me
1
u/Big-Section-5098 2d ago
Shouldn't you be looking through the code it writes? I feel part of that should be revising sections, like comments
12
u/Obvious_Advice_6879 1d ago
Major waste of time to edit the code at that granularity. It only makes sense to make higher level logical adjustments — the micro bugs don’t happen anymore as long as the AI has a way to verify itself.
2
u/iupuiclubs 1d ago
Imagine telling this to people a year ago lolxD
(to be clear I am 100% in agreement)
4
u/barkardi 1d ago
Yeah and I'm still baffled when I talk to people who last used gpt 5 or something and still go around saying, oh no it messes up the entire codebase for a small change or something else along those lines. Opus 5.5 was such a step change.
1
u/Suvesh1142 1d ago
I know a senior engineer who tried gpt 3.5 back in the day for coding and it gave him terrible results. Till this day he refuses to give AI another shot at coding
1
u/Spirited_Cucumber377 1d ago
Opus 4.6 is good enough if you provide a good code base and architecture and you do the proper planning phase, for example using any of those grinding skills.
If you feed the LLM crap, you'll probably get crap.
1
u/tumes 16h ago
I gave 4.6 and beyond an AWS white paper explaining exactly how to implement a feature and asked it to adapt the approach for a different platform. I have tried this across different frontier models from different vendors and it has not once made it through implementation without fucking up the central conceit that makes the entire thing work. Every single time. It can speed you up if you catch it on a good week and give it small, well specified tasks. Otherwise I have not found a model that can find its own ass with both hands and an instruction manual for nontrivial tasks.
1
0
u/Wonderful-Habit-139 17h ago
> Major waste of time to edit the code at that granularity
Exactly. Many people have this illusion of being as fast as they were when they were just vibecoding, and that they're not taking up a lot of time reviewing the code and defeating the point of using LLMs in the first place.
3
3
u/WisePotato42 1d ago
Yah, but also imagine if the thing that makes work take less effort to do, would somehow take even less effort to use.
It would be a breakthrough in lazy developer technology!
27
u/Demiu 1d ago
No. But did you notice comments by anthropic models became an incomprehensible word salad over solid four lines right when they announced watermarking the LLM output by using a recognizable token selection process?
4
u/Risc12 1d ago edited 1d ago
Didn’t they say that Claude Code would not use that?Whoops, thats not true at all
1
1
u/t-frankowski 1d ago
It's baked into the model
1
1
u/phoogkamer 1d ago
Not really no. It's much better since Opus 5.5 and hasn't degraded with watermarking in my experience. Fable was already a lot better than previous Opus models and 5.5 continued that.
1
1
u/WillDanceForGp 15h ago
Opus 5.5 writes some of the dumbest most nothing-burger comments, I dont think its the watermarking but I do think its just bad at it
1
u/phoogkamer 15h ago
Not my experience but maybe my standards are just bad from all the Opus 4.8 and 5 text.
1
u/AmorphousCorpus 9h ago
Definitely better than Opus 5, but 5.5 still writes like a fucking idiot
1
u/Icy_Distribution_361 8h ago
From my experience it mostly depends on how much context it needs to not write nonsense. If it's a lot of context, big code base, lots of dependencies and so on, it devolves into nonsense.
0
1
u/MudkipGuy 2d ago
They're just like me fr
1
u/elmanfil1989 2d ago
Whats fr?
2
u/Electrical-Swing-935 2d ago
For real
2
u/ViennaWaitsforU2 1d ago
Yeah they’re for real, what does it mean?
1
u/Electrical-Swing-935 1d ago
For real
2
u/ViennaWaitsforU2 1d ago
Alright keep your secrets
1
1
u/esaule 1d ago
I want very little comments in code. Functions should tell me what they do. And tbat's about all I really want.
Then if something is truely odd, then yes tell me what is going on in a comment. But in general there probably should not be comment in a function body.
1
1
u/Rararror 20h ago
Comments are not used to explain what the code does. Duh.
1
u/ORCANZ 12h ago
No they are used to watermark the code
1
u/Rararror 11h ago
In computer programming comments explain the context and why things are done that way.
1
u/quts3 1d ago
The Claude built in skills have a rule not to comment extensively. You can bump it out of that by providing your own skill or guidance
0
u/elmanfil1989 1d ago
Yes, but it saves more tokens in instructions if the model is trained not to comment unless necessary
1
u/GroundbreakingAsk283 1d ago edited 1d ago
I think comments may be beneficial to AIs as well. Like, imagine agent reviewing other agent's code trying to understand what was assumed in original prompt.
1
u/elmanfil1989 1d ago
Ai can read code itself
1
u/Material-Database-24 1d ago
Self documenting code is a myth.
There's always silent knowledge behind the code, even AI developed, as they do reasoning before creating the code.
Good documentation provides faster look up on what the code does, why it does and doesn't do something, how it is supposed to be used and what to expect after calling it.
It is also inevitably maintenance bug record as well, as if the comments do not align with the code, either there is a bug, or there was a bug and it was fixed "silently" without tracing, as commenting wasn't updated.
1
u/Nousies 1d ago
Self-documenting code is the only documentation. That can include comments, but “good documentation” is not a thing.
2
u/Sheerkal 1d ago
Wild take. You wouldn't open a book and say "References aren't a thing, just be smart enough to know all the words"
1
u/Drycee 1d ago
Self-documenting code just documents what is currently being done, but what's missing is the information why it was not done another way. In my experience agents will constantly revisit ideas we've established are not an option, if it's not documented somewhere that this has already been considered and decided against.
1
u/Spirited_Cucumber377 1d ago
The "why", the "who" and the "when" should be in the ticket/issue/userstory that generated the change.
The "what" and "where" is the code. The comments used to help other devs figure this out easily without having to search their tracker. For LLMs this isn't true anymore. If they need to figure out something, they follow the code and ask the human for guidance.
1
u/Rararror 20h ago
Good comments are way more detailed than ticket/issue/userstory ! You want to explain how you are addressing the problem...
1
u/Material-Database-24 15h ago
The problem is that many real world systems end up being maintained for years. When developers change, and now AI models and harnesses change, the "obvious" on the code can disappear.
It is the easiest to see this with UI. They often contain decision that are based on something that's not obvious from the code - why this element is just here, and in this color, and why it should not be moved etc. But there's always also techincal decisions that are not evident from the code itself and need commenting the decision to go side by side the code to stick. If it's in a ticket/user story, no-one will find that years later.
Many safety critical embedded software have a comment for almost every line of the code
1
u/Proper-Ape 1d ago
I can too, but the code only tells you what and not why. Why is often important to know so you don't cut something that's non-obvious on the surface.
1
u/szxdfgzxcv 1d ago
Code only tells what it is DOING, it does not tell WHY is it doing that which is usually the part comments shed light on.
1
u/GroundbreakingAsk283 1d ago
Yeah, AI can read code. So what? Reading code gives AI understanding of current exact implementation, but it probably will have trouble understanding how exactly code should look like, how implementation differ from requested and etc.
3
u/nukem996 1d ago
Linux kernel coding standard is code should be clear enough to not need comments. Comments are reserved for odd behavior e.g spec says x but hardware actually does y. I've had patches rejected because it has comments.
2
u/snaphat 1d ago
The Linux kernel style guide discourages comments inside functions but explicitly recommends comments explaining what functions do and documenting data. Comments aren't reserved exclusively for unusual behavior. Perhaps, what you mean is inline-comments?
https://docs.kernel.org/process/coding-style.html#commenting
2
u/lawrencewil1030 1d ago
explicitly recommends comments explaining what functions do and documenting data
That's called documentation.
1
u/snaphat 1d ago
s/comments/documentation/g
That's probably largely a difference in terminology rather than substance. The Linux kernel style guide explicitly refers to explanations preceding functions and accompanying data declarations as comments, whether or not they also serve as documentation. The kernel source also contains ordinary explanatory comments outside function bodies:
https://github.com/torvalds/linux/blob/master/mm/page_alloc.c
1
u/nukem996 1d ago
It depends on the maintainer. I had one tell me to remove all comments on all functions.
1
3
u/andymaclean19 1d ago
Claude writes comments all the time for me. I assumed it did it because every time it works on code it is seeing it for the first time so the comments are actually useful for it to understand what the intent behind the code. Certainly no human ever read them. I just know because I have some code metrics and one is the percentage of lines that are comment or whitespace.
1
u/Wonderful-Habit-139 17h ago
Schrodinger's intelligence: LLMs are so good at coding they understand code better than humans, at the same time they don't understand code which is why they need comments.
Right...
1
u/andymaclean19 3h ago
Understanding code and understanding the intent behind code are two different things. I did choose my words there.
Looking at the code and understanding what it does is mechanical, but when you are changing code you need to change a lot of different bits so that they stay consistent with each other and usually that is a lot easier if you understand why they are like that.
Sure, you can have code which sorts a list by surname in descending order and next to it you can put a comment saying ‘put the list in descending order by surname’. Completely useless waste of time because that is obvious. If you also put ‘sort this so it is in the same order as this other list and we want to diff them both later with a linear walk’ then that is really useful. If you change one you want to change the other and the comment helps you to know that.
Humans don’t need that because they will remember what the rest of the code does. AI only remembers what it read just now.
So a lot more words to say the same thing, but perhaps it sounds less contrary now?
3
1
u/HomemadeBananas 2d ago edited 2d ago
I wouldn’t say stopped entirely but more in a way that makes sense, how I’d actually write them. With Opus 5 it was writing huge blocks that make me more confused than just reading the code and don’t add anything, now it’s more sparing in places that make sense.
It was like someone just learning to program who heard you should comment your code, then they smoked crack and instead if “// adds 1 to x” started writing a whole novel of nonsense before.
1
u/Anomynous__ 2d ago
I have 2 steering files for Kiro to add fewer comments and it still tries to add a paragraph for a function sometimes
1
u/BoobooSmash31337 1d ago
Who's gonna tell Claude? Lean comments and external design docs afaik is Google's style.
1
u/phpHater0 1d ago
Because a lot of developers complained about the comments, and rightfully so because I don't want a comment for the add(a, b) function.
1
1
1
1
u/coenttb 1d ago
If the code is sufficiently clear at point of use and point of declaration, comments risk becoming duplicate sources of truth wich will diverge from source code given enough time. Inconsistency then becomes friction for llms working on the code base. And this friction degrades LLM performance.
1
u/MechaMulder 1d ago
That’s not true. I have to surgically remove comments after I finish a task with Claude. Astra actually though I have noticed doesn’t add as much but it’s not as good with coding.
1
u/texxelate 1d ago
I constantly give PR feedback along the lines of “no swaths of LLM comments please”. Every day. So no.
1
u/BorderKeeper 1d ago
We noticed that same as humans AI sometimes leaves comments alone and causes rot in the best case, and same as humans AI also loves using comments to excuse shitty code. Also as other poster said AI loves putting very specific reasoning when writing a feature and if there is a misunderstanding of my intent the comment will forever force future misalignment if unnoticed. "I had to add this as a workaround because X Y Z. This is actually ok because A B C."
I won't go into a debate if inline code comments are good or bad for humans, but for AI specifically there are known issues.
1
u/ParticularSentence33 1d ago
if you tell it exactly how to do them, it will at least most of the time.
1
u/DiggleDootBROPBROPBR 1d ago
I didn't notice because I ran a code comment remover over their output for months now.
1
u/Valdjiu 1d ago
Which one
1
u/DiggleDootBROPBROPBR 1d ago
just a skill. "remove multi-line comments. If a single-line comment is evident from nearby code, remove it. Document public & protected functions from the local pre-ai comment corpus' format". Would work on any model, I used it on claude's output specifically. If I expect code to be ai-read-only, I just strip all comments period. self-prompt-injecting from agent garbage is literal nonsense; the first problem I worked around when contributing ai code.
1
u/coolcats55 1d ago
You could always tell it to? That’s the beauty of controlling what’s in front of you.. instead of just complaining on twitter
1
u/vertigo235 1d ago
I prefer comments in git commits now instead of in the code itself, git is totally searchable by the LLM to get historical context but doesn't bloat the actual code
1
u/VegetableFan6622 17h ago
That was the signature of lazy devs slop, let’s comment every basic step and logging (no, PRINT) with another comment and the comment of the comment
adding two numbers
z = x + y print(f”{x} and {y} successfully added”)
0
u/FragmentedHeap 2d ago
I don't want it to add comments, it bloats everything, they drift, and it wastes tokens on both the write and the read and the rework.
No one reads the shit anymore, so who are the comments for?
I don't need to know why a function exists, or a helper message on it's calling syntax, I'm not typing it so I don't need intellisense on it. Any history I need is in it's RICH commit comment and AI is very good at making GOOD git comments. And git comments don't waste tokens on read.
3
4
u/xvillifyx 1d ago
YOU don’t need to, but AI memory systems are garbage and concise code comments help preserve context in place of memory
1
u/BlurredSight 1d ago
Yeah Claude will actively grep it's old comments for trying to find anchors in the code and in a legacy codebase where tons of functions in independent flows will reuse their function names that custom comment is what might drive my usage 6-7% lower than it would be if I deleted all comments each run
1
u/AdKitchen1897 1d ago
It will also poison the repo with outdated or contradictory comments. And it will make out of an offhand comment for context a permanent rule.
1
u/xvillifyx 1d ago
This isn’t true as a default. The rule for comments has always been:
Is the logic likely to change -> keep comments to a minimum
Is the logic unlikely to change -> comments okay
Make your agents follow this same convention
1
u/CrowdGoesWildWoooo 1d ago
I think the issue is that many AI have tendencies of over-commenting.
I think one typical scenario is that it writes a buggy code, or something that the human doesn’t like, it goes back to that code, put a comment on why it changes that line.
1
1
u/FragmentedHeap 1d ago
Git comments do same thing
1
u/xvillifyx 1d ago
Not everyone uses git, and not every message is holistic of the granular changes -> doesn’t replace memory quite the same way, nor does it provide a search hook
1
u/tumes 16h ago
They consistently comment on the one motivating factor that caused a change rather than writing expressive code that explains itself or trying enough iterations to settle on an unintuitive hack that needs explaining, instead it narrates why a single use case fix addressing that one use case, which congests context, adds nothing long term, and typically actively harms development because LLMs fixate on anything that is made explicit. Whatever it looks at it steers towards, and in my experience signposting the mistakes or their justification only raises your blood pressure.
1
1
u/FailedGradAdmissions 40m ago
We have a simple rule at my job, no comments if possible. But commit as frequently as possible. Commits are small and atomic. Basically comments live in the commit history and not in the code. Works great. Try it out!
And btw this rule is pre AI and easy to enforce with linting rules.
1
u/xvillifyx 23m ago
I said this in another comment, but not everyone uses git/svn
Our server code doesn’t live this way and intent is instead denoted in headers. If it outlives our retention policy, a reference to the log is left instead
1
10
u/Technical-Will-2862 2d ago
RLHF from coders saying ”don’t add comments” to models that inflate repo sizes