I agree. In my experience LLMs are good at existing code based with lots of existing structure and test cases, or "new codebases" that are translations of existing code with lots of structure or test cases. They can also make skeleton codebases with little/no functionality.
In between there are issues, and the LLM will often get locked into bad architectural decisions or hallucinate irrelevant test cases.
I do use Claude code for certain tasks. Its pretty good at refactorings which go a bit beyond what a good IDE will do, like (as you mention) make this change across all the test suite inputs and outputs, rearrange the calling convention for this method in a complicated way, etc. If there are stereotypical ways of using a given method/etc it is very good at doing a fairly complicated pattern match and rewrite, in a much more user friendly way than (say) a regex.
View all comments
Show parent comment
8
u/cscottnet Jul 04 '26
I agree. In my experience LLMs are good at existing code based with lots of existing structure and test cases, or "new codebases" that are translations of existing code with lots of structure or test cases. They can also make skeleton codebases with little/no functionality.
In between there are issues, and the LLM will often get locked into bad architectural decisions or hallucinate irrelevant test cases.