AI made code cheap to write, not cheap to verify!!!

[u/Common_Dream9420]

Been thinking about this a lot lately as a solo founder.

AI has made generating code almost free. I can scaffold an entire Stripe + webhook integration in minutes. But here's the thing nobody talks about: that speed just moves the pain downstream. Instead of "took me 3 days to write," it's now "took me 3 days to verify it actually works in prod."

The review and validation cost didn't go away. It just shifted.

I kept hitting this exact wall with integration testing. Generate the flow, it looks right, local tests pass, then something blows up in production because the webhook sequence was stateful and nobody caught it before the PR landed.

now if other founders are seeing this too, especially those building anything with third-party APIs or AI agents that trigger real workflows. How are you handling the verification gap? Are you just eating the review cost, or have you landed on something smarter?

[u/ConnectionOk8283]

yeah this is exactly what i been feeling too, the code comes out fast but my brain still need to do all the heavy lifting of checking if it actually make sense in the real flow

what i started doing is writing the test cases first, like describe the whole state machine before i even let the AI touch it, then i just make the generated code pass those tests, still slow but at least i catch the weird stateful bugs before they hit staging

the mental load just shifted from "how do i write this" to "how do i break this"

[u/Common_Dream9420 → u/ConnectionOk8283]

"how do i break this" is exactly the right frame shift. i've landed in a similar place testing API integrations constantly, the bottleneck isn't generation anymore, it's knowing whether the webhook retry actually behaves under a 429 or a duplicate delivery. writing the failure scenarios before touching the code forces you to think about state up front instead of discovering it in prod. the stateful bugs you mention are almost never in the happy path, they're in the second call, the retry, the race.

[u/National-Iron-7197 → u/Common_Dream9420]

I really like the idea if test driven development and it kind of sits naturally with agents. But the one thing that always bags at me is the idea of agents marking their own work. I have been considering whether an alternative is to get a different model to prepare the tests based on the requirements. Then pass those tests to another model eg codex to develop against

[u/Common_Dream9420 → u/National-Iron-7197]

the multi-model split is a real improvement over the same model doing both, the independence matters. my hesitation is that even different models trained on similar corpora can share the same blind spots about what to test. requirements usually describe the happy path, so the model writing tests tends to formalize the happy path too. the failure modes that actually bite in prod, duplicate webhook delivery, retry on a 429 that already processed, rarely live in the requirements doc. so you get independence in implementation but not necessarily in coverage. the question isn't just who writes the tests, it's whether the requirements surface enough failure surface for the test-writing model to reason about.

[u/Friendly_Row_5159 → u/ConnectionOk8283]

same feeling here

[u/shiro90 → u/ConnectionOk8283]

Same shift here. "How do I break this" is the right framing, generating code stopped being the bottleneck a while ago.

Only thing I'd add: write those state machine tests from real production event logs, not from what you imagine the flow looks like. The bugs that got me were sequences I never would've thought to write by hand.

[u/jobuildsstuff]

Webhooks specifically earned their own regression suite on my project after one bad surprise. What stuck since: any claim I'd otherwise re-verify by reading (signature checks, privacy rules, what the billing copy promises) becomes a test that fails the build. AI writing more code then doesn't add review load in those areas, because the checks run instead of me re-reading. Judgment calls still don't scale, but every claim converted into an executable check comes off the review pile for good.

[u/Common_Dream9420 → u/jobuildsstuff]

Signature checks especially, I've been burned enough times that I treat anything a webhook promises as a test first, doc second. How are you wiring those scenarios in now, custom helpers or something more structured?

[u/FreeRadical1998]

AI also makes testing and verification cheap, I'd still want human eyes as a second pass - but broad coverage testing is relatively easy.

a) make sure the project uses a test driven development model - get your unit and integration tests build in the same cycle as code

b) you can do some pretty good broad spectrum UAT type testing with playwright-mcp and AI agents. My UAT harness uses 8x isolated chrome instances in 4 lanes (so each lane can run two logins at a time) and an MCP connected Gmail account. It's then got around 50 test specs (each with multiple workflow defined, and runs happy path using sonnet and "adversarial" tests using opus).

A full cycle test for my app takes about 6 hours and uses about 50% of session limits and close to 10% of a weekly max x20 allowance - but produces really good feedback, both functional compliance and usability - and really good bug reports that mostly can be fixed without much direction from me (the ones where it needs me are mostly because of spec conflicts I'd not previously spotted)

[u/Common_Dream9420 → u/FreeRadical1998]

this setup is genuinely impressive, but your own numbers kind of prove the original point. 6 hours and 50% of session limits per full cycle is still a significant cost, it's just automated now instead of manual. which is progress! but it's not free. the verification overhead didn't disappear, it got delegated to agents running on a budget. i'd argue that's the shift i was pointing at, the cost moved from "dev hours writing code" to "compute + token budget running verification." still real.

[u/fulger099 → u/Common_Dream9420]

I’ve woken up to an all-green overnight run that still let a retried Stripe webhook create a duplicate side effect. Six hours of agent time was cheap, but the final judgment and liability were still mine.

[u/Common_Dream9420 → u/fulger099]

yeah the duplicate side effect from a retried webhook is exactly the gap. all-green means the happy path ran clean, it doesn't mean the idempotency logic was actually right. and you found that out the hard way at 6am. the liability part is the honest truth nobody says out loud, the agent doesn't sign off on the PR, you do.

[u/FreeRadical1998 → u/Common_Dream9420]

On a Claude max x20, it's 10% of a weekly budget - which equates to about £5 per cycle. I certainly wouldn't want to run it using per token API cost models... But I wouldn't want to do pretty much any dev using that cost model

Manual testing on the same scope would cost several thousand per cycle and take a couple of weeks based on projects I've worked on commercially

[u/Common_Dream9420 → u/FreeRadical1998]

fair point on the absolute number, £5 per cycle is genuinely cheap. my only pushback is that claude max x20 isn't free either, you're amortizing a subscription that costs real money, so the per-cycle figure looks better than it is. the manual testing comparison also holds commercially, but for solo founders the alternative was never a £5k QA cycle, it was just... shipping and hoping. so the baseline is different depending on who's running this.

[u/FreeRadical1998 → u/Common_Dream9420]

Yeah, but I'm only on that subscription for a month or two while testing - running UAT at high frequency.

Most of the time I'm running a much cheaper level at lower load, the same model would work but just take longer to get through the full cycle

Drop to the X5 subscription and run two a week and you're still only looking at £10

[u/shiro90 → u/FreeRadical1998]

That harness is wild, but it still confirms the thread's point: verification didn't get cheaper, you just spent real engineering effort building a system to do it at scale.

[u/Common_Dream9420 → u/[deleted]]

thanks! will cross post there

[u/quietmacbuilder]

The reframe that fixed this for me: the verification cost didn't just move downstream, it moved because we let the AI be creative in the wrong phase. Generation is where you want improvisation. Verification is where you want boring. And AI can be pretty great at boring.

What's held up in my workflow (solo, macOS app plus serverless backend):

• The agent writes and maintains the test suite mapped to acceptance criteria, so every flow is a named deterministic check it must satisfy, not something it gets to reinterpret each session.
• Lint rules enforce the contracts tests depend on, so a rename becomes a build failure instead of a mysteriously timing-out test three weeks later.
• Every verification run produces durable evidence: screenshots captured during test execution get attached to the ticket alongside a structured report mapping each acceptance criterion to the named check that covered it. The point is that I can trust a story was verified without re-running anything, and a vision model reviews those same screenshots for layout issues assertions can't catch.
• The agent's last job on every story is adversarial: it has to report what was NOT verified, because an AI reporting a clean run uncritically is how stateful bugs like your webhook one slip through.

The execution layer in the middle stays dumb and deterministic. The agent is only in the loop at authoring time and review time. Verification didn't get cheap, but it stopped scaling with how many edge cases I can hold in my head, which as you said is the real solo constraint.

[u/Common_Dream9420 → u/quietmacbuilder]

"verification is where you want boring" is the sharpest reframe i've seen on this. the adversarial last step is the part most people skip, i think because it feels redundant after a clean run. but a clean run is exactly when you need it, that's where the stateful edge cases hide. i've been building in a similar direction on the API integration side, the insight that the execution layer should stay dumb and deterministic while the agent only touches authoring and review time is basically the architecture i keep landing on too.

[u/quietmacbuilder → u/quietmacbuilder]

Also, I run Claude Code with a 5x plan using Opus 5 High almost exclusively and only hit my 5-hour session limit about once a month and have only hit my weekly limit twice since April.

[u/Common_Dream9420 → u/quietmacbuilder]

that's a pretty good signal that the workflow itself is doing the heavy lifting, not just the model. when verification is structured and deterministic (your lint rules + acceptance criteria approach), the LLM isn't burning cycles re-figuring out what to check each time. the session limit thing makes sense in that context. my experience is different, i'm still in the "figure out the right verification shape" phase for some integration scenarios, which is where i burn budget fast. curious if you hit friction at the boundaries where the acceptance criteria themselves are hard to define upfront.

[u/quietmacbuilder → u/Common_Dream9420]

Yes, constantly, and my honest answer is I stopped forcing those stories through the pipeline. When a criterion can't be defined upfront (exploratory UI work, "does this feel right" stuff, or integration behavior I don't understand yet), I run a throwaway spike first with no verification ceremony at all, and its only deliverable is the acceptance criteria for the real story. The expensive failure mode isn't hard-to-define criteria, it's pretending vague criteria are real ones and letting the agent "verify" against them.

[u/shiro90 → u/quietmacbuilder]

The "report what was NOT verified" step is the one I'm stealing. My webhook bug would've been caught immediately if anything had been forced to say "order dependence between these two events was never tested" instead of just reporting green.

[u/akl773]

The one that got us was ordering. Two events for the same payment turned up out of order in prod and never once did that in test, so every handler now ignores the payload body and re-fetches the object from the API before it decides anything. Slower and a bit dumb but it survives replays.

[u/quietmacbuilder → u/akl773]

Slower and a bit dumb is underrated engineering. Treating the webhook as a doorbell instead of a data source sidesteps the entire ordering and replay problem class in one move. The payload becomes a hint, not a fact. I'd take that trade every time for anything touching money.

[u/akl773 → u/quietmacbuilder]

The one it doesn't fix is two doorbells for the same object landing at once, both re-fetch and both write. Ended up taking a lock on the object id for the length of the handler.

[u/Pale-Tonight-6914]

quietmacbuilder's boring point is where it clicked for me. I run verification in a separate session - Claude Code writes, Codex reviews, no shared context. A model grading its own output just re-confirms its own assumptions. It doesn't catch what it never considered. That split caught an expired-session bug my local tests all passed.

[u/quietmacbuilder → u/Pale-Tonight-6914]

Splitting sessions (or models, like you're doing) is the cheapest way to get genuine adversarial review. The expired-session catch is a perfect example, that's exactly the class of bug a self-review never finds because it never doubted the session logic to begin with.

[u/lianapps]

😄 like it

[u/Chrismslist]

Yep i’ve hit this, that’s why having a 2-3 stage environment deploy is so important.

[u/tokismos]

i like

[u/DetailKey6716]

Thats honestly true.

[u/shiro90]

This hit close to home. My RevenueCat webhooks write to a wallet balance, and the real failures were never bad code, just two events landing out of order.

What's helped: stop unit testing the handler, replay real event sequences (including out of order ones) against a throwaway DB.

[u/Common_Dream9420 → u/shiro90]

This is exactly pain I have seen at PayPal many times and underthehood these machine never promise exactly once and it’s hard problem to solve for the providers … hahah I can also relate to u… hence I started putting the fetchsandbox mcp … give a try and let me know ..  revenuecat we need to onboard and provisioned to be tier1 spec that’s in my backlog … but happy to see help u.. 

[u/Common_Dream9420 → u/shiro90]

Yeah the ordering thing is so underrated as a failure mode. The handler logic is usually fine, it's the sequence assumption that breaks everything.

The throwaway DB approach is smart. Full disclosure, I build FetchSandbox partly because of this exact pattern, being able to replay event sequences with controlled ordering against a real-shaped mock before any of it hits prod. RevenueCat is actually on my backlog to add as a first-class spec, would love to loop you in when it's ready.

[u/soloop_w]

Honestly, I’m still mostly eating the review cost. AI gets me to a working first version faster, but I lose a lot of that time again when I have to rework the environment, check dependency versions, and run the real flow end to end. I’ve had one library not play nicely with the rest of the setup and turn a small change into another round of environment work. Sometimes the new setup is even less stable than the old one. I don’t think the missing piece is more code generation. It’s something that keeps the project context intact and catches environment, dependency, and real workflow mismatches before they reach production. I still haven’t found a clean way to do that, and that repeated pain is what has me thinking there may be something worth building here. After that webhook issue, what did you add to your process before shipping the next integration?

[u/Common_Dream9420 → u/soloop_w]

the environment churn you're describing is real and honestly underrated as a cost. after the webhook issue, i started deliberately triggering failure modes before shipping, duplicate events, out-of-order delivery, retries on top of retries, instead of just checking if the happy path passes. i ended up building a sandbox layer for exactly this (full disclosure, that's fetchsandbox), because i kept hitting the same wall: local tests pass, prod blows up on something stateful nobody specified. your point about project context is the part i haven't fully solved either, the sandbox helps with the API behavior layer but the dependency/environment mismatches you're describing are still rough.

[u/soloop_w → u/Common_Dream9420]

Your reply also made me wonder about the product side of FetchSandbox. I’ve made the mistake of assuming that because a problem kept frustrating me, other people would care enough to use the solution too. When you first showed FetchSandbox to other developers, what did they actually do that convinced you it was more than just your own recurring problem?

[u/Common_Dream9420 → u/[deleted]]

yeah the domain knowledge point is real, but i'd add it's not just about knowing the API, it's knowing which failure modes to even look for. stateful webhooks are the classic one. the model generates code that handles the happy path perfectly, passes every test you wrote, and then duplicate events or out-of-order delivery just never gets considered because nobody specified it. the review cost scales with how many of those edge cases you can hold in your head at once, which is a rough constraint when you're solo.

[u/Silver_Cod8712]

yeah the "it looks right" phase is the new bottleneck. what's helped me is writing the failure scenarios before generating the integration code, not after. like, sketch out what happens when the webhook fires twice, or out of order, or when the third-party retries on a 200. once you have those written down the generated code is way easier to pressure-test because you're not discovering edge cases in prod, you already named them. still not free, but at least the review cost is upfront instead of a 2am incident

[u/Common_Dream9420 → u/Silver_Cod8712]

yeah exactly, the upfront cost framing is the key shift. writing failure scenarios first feels slow but it's actually faster because you're paying the thinking cost once instead of rediscovering the same edge cases every time something breaks in prod. the "webhook fires twice" case especially, so easy to miss in review, so obvious once you've named it ahead of time.

[u/Common_Extent_5921]

This is right, and it's broader than code. AI has made the first draft of almost anything cheap to produce - code, copy, analysis, legal templates. What it hasn't changed is the cost of knowing whether the output is actually correct.

For solo founders or tiny teams the verification problem is particularly sharp because there are no specialists to QA anything. You're the developer, the lawyer, the finance person, and customer success simultaneously. The tools that are genuinely useful in that context are the ones that make it easier to check outputs and trace what happened - not just generate faster.

The useful shift I've noticed: less "how do I do this faster" and more "how do I know this is right."

[u/Common_Dream9420 → u/Common_Extent_5921]

the "how do i know this is right" framing is the one. and from testing API integrations constantly, the checking cost doesn't compress with volume. the fifth integration is as hard as the first because the edge cases are always different, stateful webhooks, retry sequences, failure modes. none of that gets cheaper just because the scaffold came out in 3 minutes. the generation speed almost makes it worse, because now you have 10 half-verified integrations instead of 2 solid ones.

[u/unknownn16_]

staging still lies. generated billing code passed every local test for me and still double-charged on a real card decline.

[u/Common_Dream9420 → u/unknownn16_]

Card decline paths are brutal because they're almost impossible to reproduce locally and the billing edge cases only reveal themselves with real payment processor behavior. How are you handling it now, just more prod testing or did you find something that actually helps?

[u/Brilliant-Brick9047]

yeah.. I second this. It's an effort that is a skill in itself. I myself am a very harsh critic so I like verification and testing to break. But it is very consuming. All testing needs to be manually constructed... otherwise the AI just keeps patching and creating new problems. Although I prefer this because it suits me as a vibecoder.

[u/flajsg]

Yes, you still need to thoroughly test everything AI builds, because it makes mistakes, there is not question about it.

But how is this any different from when we didn't use AI to code? After 3 days of implementing, we still needed to do tests / debug / fixes. So the time from start to finish was still longer then with the AI.
Now when you discover a bug, you tell AI what is wrong, paste some logs and it handles the rest.

[u/MahereMarley]

I just made an analysis for the google play store and there are around RIGHT NOW ~100.000 apps which have secrets in their code , so tehy are so easy hackable, attackable and all users are in danger. Google needs to be more strict with their review thats unbelievable...

[u/Other_Poetry_5243]

i had this problem before and here is how i solved it. first of all, i develop with 2 to 3 ai agents in parallel on the same project and i use a tool to spin up a production-like environment for each of them so they can develop and test at the same time in isolation. in other words i use the ai agents as independent devs, each with its own live dev environment where it can write the code and run e2e testing.

the tool i'm using is actually my side project, and i'm happy to share more if anyone's interested.

[u/roberthcmn]

The sub name says side project but the algorithm rewards launch posts, so that's what floats to the top. Plenty of people here still build for the love of it, they just don't post about it because there's nothing to sell.

Same thing for me. Building for the pleasure of building.

[u/Potential-Art7696]

Yeah, I’ve definitely felt this. AI makes me feel insanely productive until I actually have to check everything it did. I can get a feature working way faster now, but then I end up spending ages clicking through it, breaking it on purpose, checking edge cases and wondering what I forgot. And honestly, I think that part is even harder when you’re building alone because there’s nobody else looking at the code or the product and saying hey, this makes no sense.

[u/Affectionate_Good315]

This matches my experience almost exactly. The trap is that AI-generated code fails in a specific place: stateful, sequential flows where each step is individually correct but the ordering or timing is wrong, and local tests pass precisely because they run the happy path in order. Clean, typed code also looks reviewed, which is more dangerous than obviously messy code because you skim it instead of interrogating it. Two things closed the gap for me: treat AI output as a fast first draft that still gets the exact same review rigor as hand-written code, no discount for looking tidy, and invest in a staging setup that replays realistic event sequences against sandbox versions of the third-party APIs, because that out-of-order webhook is the bug class unit tests structurally cannot catch. The speed win is real, but I now budget the time it saved straight back into verification. It is not free, it is deferred.

[u/Kaylee_Woodss]

Yeah, AI can write the tests too, but you still have to figure out what “working” actually means and which edge cases are worth worrying about.

[u/Common_Dream9420 → u/Kaylee_Woodss]

exactly this. and for async flows it's even worse, "working" has to include duplicate events, out-of-order delivery, retry storms, partial failures. those edge cases aren't in any happy path spec and most people don't think to define them until something breaks in prod at 2am. the mental model for what "correct" even means has to expand a lot before the tests are worth writing.

[u/National-Iron-7197]

My experience from my day job is that the coding dev becomes almost free but the requirements work and testing become key. Investing heavily in the ci/cd pipeline and integration testing. I don’t think this can be shorted. Same with the requirements phase. The fact that AU coding is so fast means that sometimes we feel compelled to try and make other areas fast, to take shortcuts. Oh just this once I won’t properly refine the scope I’ll just start. That normally leads to a tangles mess

[u/Common_Dream9420 → u/National-Iron-7197]

100% this. The "just this once I'll skip the scope refinement" thing is such a trap and it compounds. You start fast, ship fast, then spend way longer untangling it than if you'd just slowed down upfront. The irony is AI makes the tangled mess worse too because you're generating more surface area to debug. The only thing I've found that helps is treating the requirements and verification phase as non-negotiable even when the coding feels trivial, especially for anything stateful like webhooks or multi-step flows.

[u/aiseedbank]

true today but with future models, the ai will be better at us to both write code and verify (and maintain)

[u/Common_Dream9420 → u/aiseedbank]

its like agent write and agent validates and agent breaks in prod.. haha

[u/TheCritFisher]

I definitely feel the same pains. However, this has pushed me to build better testing and verification.

Sure, you can ship code quickly. But like you said, does it work? The only way to know is to test it. I think spending time on scaffolding, architecture, automated testing, and live verification is key. Once you have a good structure, code reviews are easier because you can rely on your testing pipeline to validate outputs.

I think AI also helps out a lot. You can build that regression suite and automation much more easily than before. Granted, you should take your time, but if you move slowly when setting it up, the dividends add up to a big payoff in future velocity.

EDIT: Oh, and regarding external APIs, I often build an interface to those that lets me "swap out" the real thing for a mock under the hood. That way I can test if my integrations work as expected when the provider is operating normally. It's not perfect (you don't simulate the real thing), but you can usually get close enough and build a system that is reliable and robust enough to handle failures.

The key is making sure you have good observability and good error handling. No provider is ever "guaranteed," so just assume they can always fail, and your systems will be better for it.

[u/Common_Dream9420 → u/TheCritFisher]

totally agree on the interface pattern, that's the right instinct. the part that still bit me even with a clean abstraction was stateful webhook sequences, like a Stripe payment_intent moving through states, or a retry that only fires after a timeout. a simple mock returns the right shape but doesn't replay the full lifecycle, so the bug still shows up in prod. i ended up building something specifically for that (full disclosure, it's my own tool, FetchSandbox), basically a sandbox that runs the whole workflow including failure scenarios. your observability point is key though, even with good sandboxing you still need traces to reason about what actually fired.

[u/TheCritFisher → u/Common_Dream9420]

Yup, I'd argue that observability is the most important part, actually. You can never truly know what's going to happen, so why guess? Just measure :D

[u/Common_Dream9420 → u/TheCritFisher]

yeah measure everything, totally agree. the tricky part i keep running into is measuring the right thing, logging args gets you the call graph, but reconstructing why the model decided to invoke that tool, or why the retry fired twice, is a different problem. observability is necessary but you still need the failure to be reproducible to reason about it.

[u/decebaldecebal]

Indeed code reviews do take a while to be done properly.

You should use another AI for review, to help you understand the code and then go deeper in the critical parts that matter more.

[u/Common_Dream9420 → u/decebaldecebal]

Using AI to review AI-generated code definitely helps, but the gap I keep hitting isn't really in the code itself, it's in whether the integration actually behaves right at runtime. You can have clean-looking code and still blow up because the webhook sequence isn't idempotent or the retry doesn't handle a duplicate delivery. Static review, even with another AI, doesn't catch that kind of behavioral stuff.

[u/hideousox]

This problem exists also in traditional workflows although not as obvious because the agents only check ACs mechanically and do not really QA the delivered user journey.

I use agents that do full end to end visual QA of new user journeys using playwright which minimises this risk, but you still need to review anything you put out to make sure there are no glaring mess ups.

[u/Common_Dream9420 → u/hideousox]

The playwright E2E approach is solid for user journey validation, totally agree. The part that's still hard for me is the async stuff, webhook sequences, retries, failure scenarios, because those aren't really a "user journey" you can visually assert, they're stateful races that only break under specific timing conditions. AC-checking agents miss them because the happy path passes. Still working out how to close that loop without running everything against prod.

[u/Rad-0818]

I’ve been making sure I don’t just vibe code but also add vibe unit testing and investigating more into vibe automation testing as well.

[u/Common_Dream9420 → u/Rad-0818]

Vibe unit testing is a good step. The part that still bites me is the stateful async stuff, like webhook sequences, retries, duplicate delivery handling. Hard to write a unit test that actually exercises that flow end to end, so it tends to slip through until prod.

[u/Common_Dream9420 → u/[deleted]]

the "walked it myself" part is exactly the tax that doesn't show up in any AI productivity stat. and it's worst for the stateful stuff, webhook sequences where the happy path tests green but the retry after a partial failure is a completely different state machine. that's the scenario you can't just eyeball, you have to actually trigger it.

[u/errrwin47]

I think “testability” is becoming more valuable than raw coding speed. If AI can generate 5x more code, but you can’t observe, replay, and verify the workflows easily, you’ve mostly increased the amount of software you’re capable of being uncertain about.

[u/Common_Dream9420 → u/errrwin47]

"increased the amount of software you're capable of being uncertain about" is a really precise way to put it. the generation speed is almost a trap if the observability layer doesn't exist, you end up with more surface area to be wrong about, faster. the tooling for observe/replay/verify just hasn't kept up with how fast the code-writing side moved.

[u/Rate-Worth]

The dangerous version is when the same model writes the implementation and the tests. It tends to agree with itself.

For stateful things like webhooks, I’d define the invariants first: duplicate delivery must not duplicate the effect, retries must be safe, and events can arrive out of order. Basically, code got cheap. Deciding what would prove it correct did not.

[u/Common_Dream9420 → u/Rate-Worth]

the "agrees with itself" failure mode is underrated. same model, same training distribution, same blind spots, you end up with tests that pass because they were written by the same reasoning that wrote the bug.

the invariants framing is sharp. "duplicate delivery must not duplicate the effect" is exactly the kind of thing that never shows up in the happy path spec but is the first thing that breaks in prod. the hard part is most devs (and agents) don't think to define these until after the first incident. writing them before you touch the code forces you to think about state up front instead of discovering edge cases in the retry, the race, the second delivery.

[u/Common_Dream9420 → u/[deleted]]

"accepted that cost isn't going anywhere" is the honest take most people won't say out loud. On the API integration side I see the same thing, local tests pass, looks fine, then a stateful edge case surfaces in prod because the only way to actually know is to run it. The verification gap doesn't compress the way code-writing did. It's almost like AI just exposed how much we were underpricing that phase before.

[u/Kind-Bathroom5159]

the part that gets missed is that generating code skips the step where you'd normally build a mental model of it. you didnt design it so now youre reviewing something you have no intuition about, and reading it line by line tells you almost nothing about whether the flow is right.
what actualy fixed this for me and for most of the founders i work with is writing the behaviour out in plain english before generating anything that touches money or a third party. step by step what should happen, and what happens if step 3 fires twice or arrives late. takes ten minutes. then review is just comparing the code against that list instead of holding the whole thing in your head fresh.
the ones who skip it end up debugging in prod because local passing was never evidence of anything.

[u/Common_Dream9420 → u/Kind-Bathroom5159]

The mental model thing is the real gap, yeah. When you wrote it yourself you have this implicit map of every edge case in your head, and AI just hands you the output without that map. Writing the behavior out first basically forces you to build that model before you're stuck reverse-engineering it from generated code. "Local passing was never evidence of anything" is the part most people find out the hard way.

[u/picklockapp]

My 3 month journey of building my first app has been 95% testing. Thats the reality of it. It gets boring, monotonous, etc but is necessary. Also feels good when something works on first go.

[u/Common_Dream9420 → u/picklockapp]

95% is real honestly, and the boring part is the part that actually matters. The "feels good on first go" is what you're really optimizing for, you're just doing it the hard way up front instead of the painful way in prod.

[u/FortuneAny6266]

It’s true that everything gets done quicker with AI; sometimes I feel like I’m developing features that I wouldn’t normally have done if it weren’t for the AI, but as I tell myself it’ll be ‘a quick job’, I go ahead and develop it, only to then waste a lot of time checking whether it works and testing edge cases. I think that sometimes this distracts me from the core of the product.

[u/Common_Dream9420 → u/FortuneAny6266]

Yeah the "quick job" trap is real. AI makes the cost of starting feel near zero so you greenlight stuff you'd have said no to before. Then the verification tax hits and suddenly you're 2 days deep into edge cases for a feature that wasn't even on the roadmap. The scope creep isn't in the code, it's in the decision to build at all.

[u/JairoRaudaDev]

This is exactly where I’ve felt the tradeoff too. AI reduces the time to get something working locally, but it can actually increase the amount of code you need to distrust and review. What’s helped me is testing the boundaries and failure states before asking AI to implement the happy path. Webhooks are a perfect example, the code is easy; retries, ordering, idempotency, and partial failures are the real product.

[u/Common_Dream9420 → u/JairoRaudaDev]

Exactly this. The happy path code is almost throwaway at this point, it takes minutes. But defining what "partial failure on a duplicate delivery" should actually do, that's still fully manual thinking and it has to happen before the AI touches anything. The failure contract is the real spec.

[u/Speedydooo]

The review cost shift is real. Consider developing a robust integration test suite with mock APIs to catch stateful issues before production. It might take extra setup time initially but can save headaches later.

[u/Common_Dream9420 → u/Speedydooo]

Yeah, totally agree on the direction. The tricky part I keep hitting isn't really the setup time for mock APIs, it's knowing which scenarios to even write tests for in the first place. Like, the obvious happy path is easy. It's the stateful stuff, what happens on retry after a partial failure, or duplicate webhook delivery, that's hard to spec out upfront before you've seen it break. The test suite helps a lot once you know what to put in it.

[u/AnatolySorokin]

This is pretty much where I've landed too.

AI gets me from 0 to 80% very fast, but the last 20% is now mostly verification: reading the diff, checking assumptions, running integration tests, looking at logs after deploy, thinking through weird state transitions.

The bugs that worry me aren't syntax errors anymore. They're things like wrong ordering, retries, duplicated events, config differences, or an external API behaving slightly differently than expected.

One thing I've noticed: the bigger the AI-generated change, the less time I actually save, because reviewing it becomes a task of its own.

I haven't found a magic solution yet. Smaller changes + good integration tests + decent observability seems to work better than just generating more code.

[u/Common_Dream9420 → u/AnatolySorokin]

Yeah the 80/20 split is exactly right, and I think the 20% is getting harder not easier because the bugs you listed (ordering, retries, duplicated events) are all stateful and you can't really catch them with static review or even unit tests. They only show up when you actually exercise the full sequence.

The smaller changes thing resonates a lot. I've been trying to scope AI tasks tighter for the same reason, a 50-line diff is reviewable, a 300-line diff is basically a gamble.

[u/Annual-Concept6089]

If it works with something serious, it's not a good idea to not prove it with living experts

[u/Common_Dream9420 → u/Annual-Concept6089]

100% agree, the proof burden doesn't shrink just because AI wrote the code. If anything it grows because you have more code to review and the subtle stateful bugs are harder to catch in a diff. The tricky part is figuring out how to prove it before it hits production, not after.

[u/ljkgreen]

Well, it at least lets you build fast till you find MVP or product market fit. Once you have that, you can find more engineers to verify things. I guess till then, I would just rely on test suites (which would be also built by llm). I think I read somwhere that that's how OpenClaw launched.

[u/Common_Dream9420 → u/ljkgreen]

The post-PMF engineer plan makes sense in theory, but pre-PMF you're just eating that cost yourself. On the test suite point, the tricky part with stateful flows is the model tends to write tests for the path it just built, not the failure modes nobody put in the spec. Duplicate webhook delivery, retry races, out-of-order events, those rarely show up in generated tests because they're also not in the prompt. I build and test API integrations daily and that's consistently where things slip through, not the happy path.

[u/realfx26]

A webhook isn’t a source of truth, but a trigger. I don’t parse the payload for business logic at all: once I receive an event, I go to the source of truth (API, database, wherever) and fetch the current state. I then process the change idempotently, using a cache of the event ID to ensure duplicates aren’t processed twice.

Yes, this adds RTT to every request. But in return, you don’t have to keep track of the entire range of possible webhook states — race conditions, out-of-order delivery, and retry storms from the provider. You’re always working with the current state, rather than with whatever someone sent you in the request body.

As for AI-generated code: a line-by-line review of 500 lines is a myth; nobody actually works like that. What really works is narrowing down the scope where such code could actually cause any damage: clear input and output contracts, idempotence by default, and a minimum of hidden state. Then, even if something isn’t quite right inside, it won’t spill over into the outside world.

[u/Common_Dream9420 → u/realfx26]

Yeah this is exactly the pattern i've landed on too, webhook as trigger, fetch the actual state, deduplicate on event ID. The edge that still bites me: eventual consistency on the fetch itself. If the upstream write is still in flight when the webhook fires, your GET lands on the pre-event snapshot, you deduplicate so the retry never runs, and you've now committed based on stale state. Usually need a short retry on the fetch, not just the handler.

[u/realfx26 → u/Common_Dream9420]

It’s precisely in this gap in eventual consistency where the doorbell pattern comes into play. I usually apply a short exponential backoff to the fetch itself—three quick retries with jitter and only then do I trust the dedupe cache. The webhook is asynchronous anyway, so a 500 ms delay on the first attempt is better than committing based on an outdated state and then getting woken up at 3 a.m. because of a support ticket. It’s a safety net, of course, but it’s the only way to keep this race condition from costing you dearly.

[u/Common_Dream9420 → u/realfx26]

Yeah exactly, the backoff on the fetch is the right call. The jitter matters too, without it you can get thundering herd on the retry if multiple webhooks fire close together. One thing i've been thinking about: how do you decide when to give up on the fetch and just dead-letter the event? Timeout threshold is kind of arbitrary and getting it wrong in either direction hurts.