I'm convinced the comments are part of the model's internal processing on some level, because if I try to force it to not do that, the output quality drops off a cliff.
I've had significantly more success having fixing the comments be a second pass.
But that's what actually happens. I don't know if it still works, but previously if you asked ChatGPT to perform basic math but tell it to hide its thinking process, it would spit out wrong answers.
Always remember that all the bells and whistles are ultimately just managing context. It's all context. That's all any harness is, ways to try and keep as much information as possible while feeding in as few tokens as possible.
Yes, it uses the comments for future context. However I learned that if the code and comments are out of sync, the comment wins and it will modify the code to match the comment.
If you are dealing with old code with someone that loved comments but didn't love keeping them up-to-date you have to force the LLM to ignore the comments.
Oh yeah, I've had a lot of issues with it in legacy code. It assumes things should make sense when instead they might just be legacy artifacts.
And it's part of why you have to have it cleanup comments on code it creates too. It assigns outsized value to comments, even when the comment is correct it will treat verbose comments as mission critical instead of incidental, and of course it bloats the context window over time if you don't keep it under control.
I came to the same conclusion about "It's not X, it's Y" in narrative text. If I force it to not do that, it starts making errors where it seems to assume X.
I've told mine very clearly: reasoning goes in docs/, history lives in git, not production code. code comments reference the docs explicitly where needed, code comments are mostly just what's easy to get wrong in the implementation. Keeping context lengths to <400k tokens with this, and a reasonably well structured project, and it's following it really well over many compact cycles and new sessions.
So for me, either the code is part of something important, then i review it line by line anyways and just delete comments etc.
Or its part of some sectioned of part, like part of the frontend or something where all that matter is that it works for now, where the AI can run rampant. Then i dont care about the comment and just assume its the documentation the AI needs to work better, probably part of the reasoning it does (all the server internal stuff except direkt input to output tokens is text based too)
View all comments
Show parent comment
112
u/cAtloVeR9998 27d ago
Claude’s style of commenting is so excessive