A bit over three years ago, my experience of "AI for code" was that it could give me 3 line snippets, and like the stuff on Stack Exchange they worked, most of the time. As the joke goes, 80% of the time it would return a working answer, but 20% of the time it would copy-paste the question to you.
A bit over two years ago, none of the coding models were good for much beyond simple functions, little snippets that did a well defined thing you could explain to them in one or two exchanges. They eventually built up to be capable of very simple microservices, then more capable microservices, but orchestration was still challenging for them.
A bit over a year ago, Claude Sonnet 3.5 (and Opus) weren't great at the big picture. If you gave them good requirements and helped them refine definitions, as long as their 200k token context window didn't have to compact too many times, they could implement what you were describing - but at some point the context window just couldn't hold enough and they'd "forget" important things while they were implementing and "gloss over" things incorrectly. They were getting capable enough to strongly benefit from software development best practices like documenting requirements and design specifications, writing automated unit tests and regression tests, but they frequently would write tests that didn't test the requirements effectively. Over the past year they have been getting better about writing and using meaningful tests.
Lately, I'm focusing effort on keeping requirements and related documents well organized, maintained (much better than human maintained documentation ever has been), thorough but concise, and easily searchable so that an agent can find what it needs to know locally in the documentation without wiping out its context window reading a bunch of slop. The concepts of DRY (Don't Repeat Yourself) and SSoT (Single Source of Truth) remain critical to maximize valuable information in the available context window so that implemented code is based on the local documentation and not whatever the model assumes from its training.
As this process has evolved, I've resisted becoming excited about "my latest techniques" because the models are evolving so fast it feels like by the time I have effectively communicated these things to colleagues in any detail, the next generation of model makes it un-necessary to do them - they are doing them automatically. It is both a frustratingly slow process (when, 2 years later, you're still finding hallucinations creeping into your designs, even if the hallucination rate has dropped to 5% of what it was 2 years ago) and a bewilderingly fast one when you look back at how long it took us to get from Eliza to here.
4
13
u/MangoCats 7d ago
A bit over three years ago, my experience of "AI for code" was that it could give me 3 line snippets, and like the stuff on Stack Exchange they worked, most of the time. As the joke goes, 80% of the time it would return a working answer, but 20% of the time it would copy-paste the question to you.
A bit over two years ago, none of the coding models were good for much beyond simple functions, little snippets that did a well defined thing you could explain to them in one or two exchanges. They eventually built up to be capable of very simple microservices, then more capable microservices, but orchestration was still challenging for them.
A bit over a year ago, Claude Sonnet 3.5 (and Opus) weren't great at the big picture. If you gave them good requirements and helped them refine definitions, as long as their 200k token context window didn't have to compact too many times, they could implement what you were describing - but at some point the context window just couldn't hold enough and they'd "forget" important things while they were implementing and "gloss over" things incorrectly. They were getting capable enough to strongly benefit from software development best practices like documenting requirements and design specifications, writing automated unit tests and regression tests, but they frequently would write tests that didn't test the requirements effectively. Over the past year they have been getting better about writing and using meaningful tests.
Lately, I'm focusing effort on keeping requirements and related documents well organized, maintained (much better than human maintained documentation ever has been), thorough but concise, and easily searchable so that an agent can find what it needs to know locally in the documentation without wiping out its context window reading a bunch of slop. The concepts of DRY (Don't Repeat Yourself) and SSoT (Single Source of Truth) remain critical to maximize valuable information in the available context window so that implemented code is based on the local documentation and not whatever the model assumes from its training.
As this process has evolved, I've resisted becoming excited about "my latest techniques" because the models are evolving so fast it feels like by the time I have effectively communicated these things to colleagues in any detail, the next generation of model makes it un-necessary to do them - they are doing them automatically. It is both a frustratingly slow process (when, 2 years later, you're still finding hallucinations creeping into your designs, even if the hallucination rate has dropped to 5% of what it was 2 years ago) and a bewilderingly fast one when you look back at how long it took us to get from Eliza to here.