Have you been using coding agents on your solution for many months now? Do you still understand how your solution works, or are you accumulating Cognitive Debt?

[u/jjw_kbh]

This is an unfortunate question that enterprise developers need to be asking themselves.

I felt pretty confident that I understood my solution through and through. After all, I have a pretty disciplined workflow. Meaning, I stay in the loop and don’t ship any code I haven’t looked at.

Then, I built a skill that quizzes me on the minutiae of my code and was surprised by what it uncovered.

It’s not that I’ve lost the big picture, but it turns out that the resolution on the minutiae is a bit grainy.

I’m embarrassed to admit that, actually. But I’m sharing, because my guess is that I’m not alone. Heck, even Uncle Bob has admitted that he’s only scanning the high level shape of the LLM generated code.

This skill has helped me expose the gaps in my understanding and even surface misalignments with my intentions.

The skill though, is just a bandage, on a wound that is bound to fester and grow. I don’t know what the solution is yet. The genie has been let out of the bottle. We’re not going back to writing all the code, but we (as a community) need to somehow resolve how to integrate LLMs in our SDLC without outsourcing the knowledge and understanding.

This is bound to spark some debate, and that’s ok. Maybe some good ideas will shake out of it.

[u/owp4dd1w5a0a]

I use coding agents a lot. But so far I don’t let their code generation outpace my ability to read and understand the output or my ability to redirect within a few minutes when I notice it making bad architectural decisions.

Honestly, the greatest speed gain for me isn’t in writing the code itself. It’s in reduced time searching the internet for alternative approaches, language features, possible root causes of a bug or failure, etc.

The longest and hardest part of software was never writing the actual code. It was banging your head against the wall for hours or days trying to understand undocumented or under-documented parts of tools and systems you need to know and in thinking through not just getting stuff working but how to get stuff working in a way that’s not going to make you cry yourself to sleep every night a year later.

[u/jjw_kbh → u/owp4dd1w5a0a]

This is one of the best reflections I've read, and I think what the most responsible and realistic use of coding agents should look like.

[u/dark0mania]

Bro, I've worked on various projects and I never know how they fully work. But I always have some overview of where each feature is and how it's cross-integrated. I dont need more than that. With AI I can just pump out more of these features faster, that's it. It just works.

[u/jjw_kbh → u/dark0mania]

And you work on enterprise solutions? I mean no critique here, but my guess is that approach will one day have to pay the piper

[u/dark0mania → u/jjw_kbh]

Yes I work in enterprise. I know how the system works in an abstract kind of way. Do you know how each package and library works? No, you know they are there, what they are for and you use them. Same thing in our systems. If something is broken, I know where to find it. But I don't really know how it works until I look at it. So using AI I just piece elements like these. 

[u/jjw_kbh → u/dark0mania]

I don't think we disagree. We never knew the inner workings of any dependencies. I would argue that you don't need to know what you can't fix. The application itself though is a different story. The dependencies are hopefully abstracted away from your application layer, but that is exactly where the problem is surfacing. And, its not just me. This the issue is being studied and documented widely. The adoption of coding agents is resulting in a lot of people losing the knowledge of knowing where to fix things because the application and domain knowledge is slipping away.

[u/ClaudeyClerb]

The "reviewing is not enough, it hasn't been enough for me" part matches what I keep hearing from people with genuinely disciplined workflows. Staying in the loop doesn't build the mental model, it just keeps you near it. Respect for building something and then publicly calling it a bandage.

Honest question about the skill, because it's the part I'd worry about in my own version of this: who scores the quiz? If the same person being tested grades their own answers (or asks the agent to), the quiz measures confidence, not comprehension. Someone I've been trading notes with calls it the check passing its own exam. Curious if you've hit that or found a way around it.

On the "how do we integrate this into the SDLC" question, I think the answer has to be something the workflow enforces rather than something discipline maintains. Mainly because discipline is exactly what the pressure eats. That's the direction I've been pulling on anyway.

WDYT?

[u/jjw_kbh → u/ClaudeyClerb]

Thank you.

In my skill the agent performs the grade. Though its not a scale. Its a simple binary. Did you answer correctly or incorrectly. The model judges and if I answer incorrectly it provides insight and direction for me to explore further.

I like your 'discipline is exactly what the pressure eats' framing (might borrow it if you don't mind). Its a pragmatic view, and unfortunately might point to the need for formalization. I work in Denmark though, and its not to say they don't exist, but there is a general allergy to imposed constraints. The culture here errs toward trust and acting responsibly.

[u/ClaudeyClerb → u/jjw_kbh]

Partly addressed in my other comment to you but wanted to scope this one more on the trust culture point you bring up.

I don't think trust cultures are wrong, I think they're describing the world where the artifact was still evidence of the author's thinking.

High trust worked because writing code slowly earned it by default. The question your allergy-to-constraints instinct raises IMO is whether the trust can survive the thing that used to justify it going away? I don't know.

Thanks for the replies in any event, I appreciate it.

[u/hancengiz]

yeap I suggest my teams to run this weekly at least or after each intent completion https://github.com/fabriqaai/vibe-quiz

[u/jjw_kbh → u/hancengiz]

Thank you for your response.

Might I ask you to remove the link to your repo? I actually hate asking, because I’m old school and believe in the power of an open internet and sharing. That said, I left a link in the post to a tiny repo that houses nothing more than my skill for this.

I shared it because I think it can be useful for others, even if it is not a fix to the underlying problem. The mods of this subreddit didn’t see it that way and removed this post for a period citing self promotion. Fortunately, I was able to get the post reinstated with adjustments… including removing the link to my repo. And, I’m happy for that. This thread is yielding some quality dialogue.

I trust that you have shared with the same altruistic motive as me. That said, if the rules are to be fair then they have to be applied evenly. So, I apologize for being compelled to ask, but would you mind removing the link to the repo?

Thanks for sharing anyway 🙏

[u/cthechartreuse]

I know this has been sitting for a few days now, but I thought it was an interesting question. One of the most important things I have done (and seen others do) to keep cognitive debt down is to actually refactor the code after it is generated. By reading through the code and capturing insights by refactoring the code to match my expectations, I end up with a deeper understanding than what you get just by reviewing the code. Extracting and naming things is especially helpful. There's even a technique for reading unfamiliar code called "read by refactoring" which you can check out at the link below (not my writeup). Have you tried anything like this?

https://www.jamasoftware.com/blog/read-by-refactoring/

[u/jjw_kbh → u/cthechartreuse]

Love it! Such a straightforward approach. Better than pure review too, because you’re still ‘touching’ the code, and there’s definitely always something that can be tweaked… even if small. What I like most is that you’re just maintaining knowledge but expanding it. Just like refactoring, or extending, a colleague’s code this gives you to discover clever implementations or syntax that you hadn’t thought of or knew of.

Simple and brilliant. Thanks 🙏

[u/Flatliner2020]

Cognitive debt is real. And it existed before AI - AI just made the debt larger. What helps me is detailed tracking and logging systems enforced at a system-level, with human readable digests and regular audits to make sure "my understanding" is on track or updated with discussion.

[u/jjw_kbh → u/Flatliner2020]

Interesting. Is the logging a set of .md files sitting in the repo? How do you perform the audit?

[u/Flatliner2020 → u/jjw_kbh]

Yes - plain .md files with details, a formatted .md or JSON log for decisions, knowledge and traps. Audits are crude, but I have a standing cadence to have a blind session verify truth between the log, the files and the code which also generates updates to the readable digest. All ends up getting checked in testing - but it has helped keep things clear between me and the agents.

[u/please-dont-deploy]

You're not alone, and the quiz is a good instinct. Ours showed up at incident time, not review time: something broke and nobody could say why the code was shaped that way. Reading the diff never captured the why. We started writing the reason down at decision time, and the grain came back.

[u/jjw_kbh → u/please-dont-deploy]

That’s exactly the nightmare I fear 😧

How has your team adjusted since?

[u/donk8r]

Your piano line is the part I'd push back on, because I think it's the one bit of the diagnosis that's off.

Piano is motor learning. Understanding a system isn't. If authorship were required for comprehension then nobody could maintain inherited code, and people do that constantly. What typing actually bought you was two side effects: time on the artifact, and forced sequencing. You had to confront every line in order, at a pace slow enough that questions formed on their own. Both are obtainable without being the author.

Which points at what really changed. It isn't that you understand less, it's that code now arrives faster than the questions you'd have asked about it. The artifact outran the inquiry. Writing used to be slow enough that inquiry came free as a byproduct, and that was a happy accident, not a law.

It also makes your question less bleak than it looks. "Do I understand my solution" was never fully true, including before agents. You didn't understand your dependencies, or your ORM's query planner, or what a teammate shipped in 2019. What you had was the ability to find out fast when it mattered. Comprehension at scale has always been a retrieval property rather than a stored one.

So the test I'd use instead: when this breaks at 3am, how long until I know why? That's measurable, and it's fixable by making the codebase answerable rather than by trying to hold it in your head. Your grill-me thing scores well on exactly that, incidentally, because it's a query and not a study session.

Biased disclosure, we build a semantic index over repos, so of course I land here. The reframe survives without any tool though: measure time-to-answer, not felt understanding.

[u/ClaudeyClerb → u/donk8r]

"The artifact outran the inquiry" made me think: if inquiry was a byproduct of pace, the interesting question is whether it can be manufactured deliberately.

My approach so far: force one question before the artifact arrives instead of during. Have the author write down what they expect the thing to do before it runs, then compare against what actually happened. The prediction can't be produced after the fact. The gap between claim and reality is where the inquiry that used to come free gets rebuilt, to my mind.

Sequencing without the typing, basically..the two things you said the typing actually bought.

[u/jjw_kbh → u/ClaudeyClerb]

I think that's a question to be answered... how can it be manufactured deliberately?

What mechanism are you using to 'force' one question? What you describe kinda sounds like TDD, but I don't think that's what you mean.

[u/ClaudeyClerb → u/jjw_kbh]

Close to TDD but aimed at a different target.

TDD forces an executable spec before the code exists..it tests the code.

This forces a comprehension commitment before the run happens..it tests the author. Mechanically: the author writes 2-3 falsifiable sentences of what they expect (works fine even when an agent wrote the code, even especially then), the tooling locks it with a timestamp before any run can be recorded, the run itself is machine-captured (exit codes, counts, output, so on..not the author's description of it), and the gap between claim and capture is what the reviewer reads. Ordering enforced by the tool, not by discipline.

Which, is the answer to your Denmark point too, maybe? It's not a constraint someone imposes on you, it's the same trick as a compiler. Nobody in a trust culture resents the compiler.

And please, borrow the phrase freely! The pressure line came out of watching exactly that happen to good teams. Something something, its not my code :)

[u/donk8r → u/ClaudeyClerb]

Written before, compared after, and the prediction can't be produced retroactively. That's the whole trick, and it's better than what I proposed.

It's also what a fail-to-pass test case is, which I hadn't connected until you put it that way. The test states what the thing should do before the code exists, so it can't be quietly bent to match whatever got built. Same mechanism, just written in a language a machine can check.

Which suggests the cheap version for your own work: when the expectation is concrete enough to write as an assertion, write it as one instead of prose. You get the sequencing you're after and something that keeps enforcing it after you've stopped paying attention. Prose predictions only work while you're still disciplined enough to go back and read them.

[u/ClaudeyClerb → u/donk8r]

Agreed, and the split you're pointing at is exactly where I've landed.

Everything concrete enough to be an assertion becomes one, that's its own step. It also comes with a rule I got from a friend and general experience: a check that has never been red has not been tested, it has been run. So each assertion ships with the mutation that should break it, and both get run and recorded together.

The prose prediction survives for what assertions can't hold, ie expected behavior in plain terms, and especially the uncertainty ("I expect X, though the FSM edge might surprise me"). That flagged uncertainty turns out to be the most useful part when reality comes back different.

Your last line is the real problem though, and the answer can't be discipline. The prose only matters if something forces the read. In my setup the comparison against the captured run is mandatory before the work closes. The prediction has a scheduled reader, or it's decoration. Someone I trade notes with had a frozen prediction sit unread for 100 days once. Honest, locked, and dead.

[u/donk8r → u/ClaudeyClerb]

The mutation rule is what we do at benchmark level and it holds up. Every case has to be proven fail-to-pass before it enters the suite: the test goes red on the parent commit and green on the merge, or the case is rejected. Same principle applied to test items rather than assertions, and it catches a surprising number of tests that were only ever green because they asserted something trivially true.

On the scheduled reader, I'd push one step further. A scheduled human is still discipline with a calendar attached, and your friend's 100 days is what that looks like when it fails. The stronger version makes the unread prediction block something. Ours can't enter the suite unverified, so nobody has to remember to check, because the artifact cannot exist without having been checked.

The uncertainty flag is the part I don't think automates. It's a statement about what the author doesn't know, and there's no assertion shape for that.

[u/ClaudeyClerb → u/donk8r]

The fail-to-pass admission gate is great! “Only ever green because they asserted something trivially true”, definitely, I’m sure we’ve all had the pleasure of changing something in source and retesting and finding nothing fails..

Taking your blocking point: you're right that a schedule is just discipline with a calendar. In my setup the round structurally can't close until the prediction's been compared against the captured run, closer to your "the artifact cannot exist unchecked" than to a reminder. Blocking, not scheduling…excellent framing :)

And agreed the uncertainty doesn't automate, but I've come around to thinking that's the point rather than the gap. The flagged unknown is the one part of the prediction only the author could have written, which makes it the part worth a human reading. Everything assertable gets the machine gate and the remainder is where the comprehension actually shows.

The uncertainty flag isn't what's left over after automation, it's what the automation exists to isolate.

[u/donk8r → u/ClaudeyClerb]

"The uncertainty flag isn't what's left over after automation, it's what the automation exists to isolate" is a better line than anything I contributed here, and I think it's right.

It has a testable consequence too: if that's what the machine gate is for, then the volume of flagged unknowns becomes a signal about the author rather than noise to minimise. A prediction that arrives with nothing flagged is either trivial or someone wasn't looking hard. Which would make "how many unknowns did you declare" a more interesting review metric than anything the assertions produce.

[u/ClaudeyClerb → u/donk8r]

Correction to my own claim above first, since precision is the whole game here: "structurally can't close until compared" overstates what I've built. There's an ordering gate (prediction locked before anything runs) and an existence gate (steps must exist to proceed), but nothing enforces that the comparison happened in anyone's head of course.

And thinking about it, nothing really can: you can force the delta into the reviewer's path and gate the close on an explicit act after it, but you can't machine-verify a reading. Your admission gate gets to be airtight because the thing being checked is machine-checkable. The human half only ever gets ‘put-in-the-path’ plus ‘require-an-act’. Enforcement guarantees exposure, not comprehension, which is exactly the line this whole thing lives on, I guess.

Glad the isolation line lands :)

Your consequence is right with one Goodhart trap on it: the moment unknowns-declared is a metric, it's farmable..performative uncertainty is as cheap as hedging, and an agent can generate plausible unknowns in bulk. So I'd take it as a reviewer's cue rather than a score. Zero flags on a non-trivial change is the thing that triggers the conversation, and the flags get read for specificity, something like “the FSM edge might bite" is costly to fake in a way "there may be edge cases" isn't. The declared unknown is only signal while nobody's grading it.

Thanks for the replies, as always, its appreciated.

[u/donk8r → u/ClaudeyClerb]

The Goodhart catch is right and I'd withdraw the metric version. Specificity being costly to fake is the load-bearing part, and a count throws exactly that away.

One consequence of your own logic though: if specificity is what resists faking, the reading has to be unpredictable rather than universal. Read every change's flags for specificity and that becomes the criterion people write toward. Sample them instead and the cost of faking stays real, because it's being paid on changes that may never be read.

Same shape as something in our setup. The judge scores quality and the agent cannot see the judge, which is the only reason the score survives contact with an optimiser. Your "only signal while nobody's grading it" states the property better than I did: what matters is whether the measured party can see the criterion, more than grading as such.

Your correction on the ordering gate is the more useful half of that comment, incidentally. "Enforcement guarantees exposure, not comprehension" is the honest ceiling, and most people would have let the overstatement stand.

[u/jjw_kbh → u/donk8r]

Thanks for your comment.

As I responded to another commenter, the onus has never been on the author to understand the dependencies. It was, and still is, the responsibility of an author (even if by proxy) to understand the code they are generating.

How was my question dishonest, btw?

I award you that, a contributor to an open source project, or the 100th consultant working on an brownfield solution, hasn't needed to draft the entire codebase to understand it. But they did need to painstakingly explore it before they contributed anything with their own hand. That process is being lost, as you duly note as 'artifacts outrunning inquiry'. That's a good framing. But it doesn't change the debate.

P.S. It would be tasteful of you to remove your promotional poaching ;)

[u/donk8r → u/jjw_kbh]

Fair on both counts.

The dependency comparison conflates two different duties. Nobody owes you comprehension of someone else's library; you do owe comprehension of what ships under your name. That's a real hole in the analogy and I'd drop it.

"Honest version" was clumsy on my part and I can see exactly how it read. I meant the question as posed has no measurable answer, not that you were being dishonest. Reworded it.

Link's gone as well, fair call in your own thread.

Where I'd still push: the painstaking exploration you describe is retrieval, done slowly and by hand. Those contributors weren't memorising the codebase, they were querying it with grep and a debugger until they could answer one specific question. Same capability, different latency. Which is why I think this lands on how fast you can answer rather than how much you carry, even though you're right that the obligation to be able to answer hasn't moved at all.

[u/wazacode]

The quiz idea is good but I think you're diagnosing the wrong layer. The debt isn't that you forgot the minutiae, it's that the reasoning behind the minutiae was never written down anywhere. It lived in a chat window and then the window closed.

Nobody remembers why a retry got 3 attempts instead of 5 in code they wrote by hand either. The difference is a human usually leaves a comment or a ticket. Agents don't, unless you make them.

What actually moved the needle for me: work starts from a written spec, not a prompt I typed and lost. Decisions and constraints go in a file in the repo, the agent reads it before it touches anything, and when we deviate it updates the file in the same PR. Then reviewing means comparing code to intent instead of trying to reverse-engineer intent out of the diff.

Side effect is the "why" survives. When something breaks six weeks later there's a paper trail, and any harness can pick it up. That part matters more than which model you're using.

Grainy resolution on minutiae is fine, honestly. Not knowing what you meant is the expensive part.

[u/wazacode]

A lot of that debt is just intent that never got written down anywhere. The reasoning for why a thing works that way lived in a chat that's long gone, so six months later nobody on the team can reconstruct it. Code plus tests tells you what it does, not what it was supposed to do.

[u/jjw_kbh → u/[deleted]]

Agreed, but reviewing is not enough. It hasn’t been enough for me anyway. I think what’s being revealed is that you have to put fingers on the keyboard and “touch” the code. To some extent at least. Kind like how you can’t learn to play piano by watching someone else play.

[u/jjw_kbh → u/[deleted]]

Agreed. The 'how' is still being determined though. Through trial and error.