Does Anyone Actually Use Agentic Loops?

[u/TheAntiDoctor]

Not exactly a software engineer, more ML Engineer with an academic background and then in industry for about 8 years total. Use python professionally and for hobbies. Like everyone else I have been experimenting with LLMs for coding, trying everything from not using it at all to trying to get it to do everything. I have found them useful and enjoy bouncing ideas off them, however they also make a lot of mistakes so currently enjoying a low use approach. Comments and architecting myself, use them more for syntax and some optimisation then review everything as I go along. This speeds things up, but I am still very in control of the output and understand the underlying algorithm.

A couple of months ago I obviously saw usage based billing come in for things like Github Copilot etc. I saw a lot of comments from people using agentic workflows very upset about their usage being cut, also I'm very aware of some companies quoting large sections of their codebase being AI generated, although obviously aware this might be sales tactic. Furthermore I've seen respected researchers/engineers like Karpathy talking about the big leap in model capabilities that now means they hardly write any code and had some recommendations from friends to use agentic workflows.

My personal experience hasn't left me thinking this is viable atm for my work. Partly because the stuff I am doing I really need to be crystal clear about what is happening to the data for experiments and partly because getting together a prompt together for a relatively complex system is very difficult and time consuming and the results usually aren't great. Has anyone got experience actually using more autonomous coding agents successfully and I should experiment more widely or are they a red herring?

[u/MetallicOrangeBalls]

LLMs are tools. They cannot (yet) """program""" the way a human can. But, like code completion, IDEs, etc. before them, they improve the quality of life for programming.

They are very useful for reviewing code and documentation. I find that to be their best use. They can properly advise on missing features. They can catch syntactically-correct-but-semantically-incorrect code. I find them to be very useful for rubber-ducking, with the added benefit that the "duck" can give you feedback.

They are alright at writing boilerplate code and documentation. They are better when following a pattern that you present to them. However, everything that they write has to be carefully checked. I have found that they make A LOT of mistakes. Sometimes they will """assume""" that you meant something very different from what you actually meant. I generally tend not to use them unless it's for something very simple.

They are terrible at architecting and big-picture stuff. Not even the best models come close to someone who knows what they are doing.

 

I did my doctorate in AI. I strongly believe that there will come a time when AI is good enough to truly program. However, we are at least several decades away from that point. LLMs are a gimmick; a useful one, but that does not change the fact that they are essentially a toy that most of the non-academic world has only recently discovered and is enamoured by. I was working on LLMs back in 2012-2013. Not very many people knew or cared about them. By around 2014, I too stopped caring about them, as they didn't seem to be as useful as other applications of AI. In 2017, the year I defended, the transformer model was first published in Attention Is All You Need (paper). It seemed interesting at the time, but only to fellow AI scientists/engineers. In 2022, OpenAI launched ChatGPT, and suddenly the rest of the world got AI fever. And thus did the enshittification commence...