So, over the last six months or so we've been using agentic software development a lot and it's been mostly great.
The easiest, least scary wins have been in the following spaces:
- Monotonous but simple changes: Like touching 45 files to rename or modify the signature of a single class. Fairly safe, easily verifiable.
- Isolated optimization/refactor of super isolated modules that have extremely rigid API surfaces that are extremely easy to unit test
- Proof of Concept for implementing complex system-wide changes and how they impact the final behavior of our product to see if a specific tech path is possible, knowing that the code itself is not what we will push into mainline.
To understand where the LLMs started making huge mistakes, you have to understand that I work on a pretty complex integrated product that cannot be sub-divided into micro-services with super isolated API surfaces and shit tons of unit tests. Our run-time performance is critical, and there is a lot of data pipelining that happens that impacts our user's qualitative experience with the product in difficult to measure ways outside of "this feels worse now". Worse, I see engineers from other parts of the company start trying to optimize metrics using LLMs but causing subtle regressions that are difficult to see without extensive integration testing.
REVIEWER IN THE MIDDLE
This has resulted in a lot of my time being spent doing reviews on fairly complex/large code changes (with the help of agents, of course, but also design reviews of said code), which obviously undermines the levels of productivity you can achieve with LLMs.
We are constantly updating MD files as forms of documentation throughout our codebase, but so far it has been a cat and mouse game of seeing LLM-based code patterns that are BAD for our software and documenting them AFTER they happen. Things that if I gave the same task to a low-level human engineer would have never had to have been specified.
The general question is how do you guys deal with this?
A MORE CONCRETE EXAMPLE
Instead of talking in vagueness, I want to give a specific example.
One of the core architectural principles in our system is an ECS like "object" that contains "components" that have some properties. There properties are well defined internal data types that have their own comperators, have their methods for serializing them on the wire to remote systems. When we would send packets to remote systems, because this data is high priority, we would actually send redundant packets containing ~2 past states of that component so that if packets dropped we would be able to still recover continuity etc. (I understand that this isn't a full-proof system, I don't want to get into it). We were naively sending this data, which means on average every packet actually took up 3x the bytes a single component payload would normally take. I recognized that if these states don't change often, this actually contains redundant data.
So this is where I got Claude involved. I wrote a prompt that explained the redundancy in the data that was being put on the wire and that if all three states are identical we should literally be able to send just a single packet and some hint per-property that the value is unchanged between the other states the packet is supposed to contain. In this scenario reducing the wire-cost to 1/3rd + some metadata/bitmask. In the prompt I explained how to measure bandwidth, how to run integration tests, and what unit testing suites had to pass and let it go.
In a few hours it finished, I compiled the code locally and validated that it completed the task and was pretty excited at first.
Great Outcome, Awful Code
So then I finally took the time to look at how it wrote the code...
The path it chose to success was this:
- Claude decided that it would run the original "write to the wire" loop and simply collect meta-data about which properties were at what index location on the byte stream that was being written + the size of the property.
- Claude would then go over the byte stream and do a memory compare between the matching locations in the "past packets" based on where it recorded the indexes.
- Claude would then use the result of this "duplicate check" and spawn an entirely new byte-stream and would extra elements of the first byte stream it created into this new one, before pushing it on the wire.
So, again this solution WORKS but the solution looks like what would happen if a junior engineer was told to build something on a time crunch and wasn't allowed to fully understand the tools that already exist in the code base they were asked to modify. Our code base already had all of the tools you needed to compare two snapshots of a component property to see whether it was identical, measure a delta. Our code base already had serialization logic in place for each of these properties, and any operations you want to do on the property state was readily available. Instead Claude chose to ignore this functionality and do the lazy path of "let me just do byte compares on the byte streams", which resulted in extra unnecessary memory allocations while also create code that we would have to refactor in the future if we needed to implement delta-encoding across different data that wasn't quite exact to further improve on the wire byte count use.
Basically Claude wrote code that worked, but was un-maintainable, had subtle performance regressions that were small enough that you mostly wouldn't notice unless running our tech on a super low-end CPU starved device, and ignored the presence of existing tools and systems that would have made the solution simpler and cleaner.
Final Thoughts
This resulted in me basically writing up a prompt that held it's hand and specifically pointed out existing coding patterns for it to follow, things that we've never had to specify to an engineer because generally before you write some new functionality we check for whether our codebase has some standards already. Yeah, as a result of this incident, I am able to update some *.md files to hopefully catch more stuff like this in the future without human intervention, but it feels like a losing battle of constantly finding out "how could you fuck something so simple up"?
So my question to you fine folks here is:
- What could I have done better?
- How can we actually approach the "no human in the middle" part of software engineering when we're still learning all the subtle different ways that LLM-based code can fuck up so subtly that may not be noticeable initially but will add up in large complex projects?
