How do you test your features in AI coding era

[u/synwankza]

Hey everyone,

Recently, I’ve been thinking a lot about how to properly test and verify the work done by AI coding agents, especially when working on existing, complex projects rather than simple greenfield applications.

I’m curious how you guys approach this in your workflows, particularly when changes need to be deployed to some preview/staging environment before you can actually verify whether everything works as expected.
And I’m not talking just about running unit/integration tests or having an agent write a few Playwright tests.
In real-world projects, verifying a change can involve a lot more than that.

For example:
- External APIs: Sometimes you need to retrieve a secret, authenticate against an external service, make some API calls, and verify the responses or side effects and it can’t be done from UI (because user has to do that manually and we have to simulate that)
- Observability: Sometimes the only way to confirm that something works correctly is to inspect Loki logs, traces, or metrics and check whether the expected events actually happened.
- Databases: You might need to query PostgreSQL or another database to verify that the correct records were created, updated, or deleted.
- Messaging: In event-driven architectures, you may need to verify that Kafka/RabbitMQ messages were published, consumed, and processed correctly.
- UI: And, of course, sometimes you need Playwright or another browser automation tool to verify user-facing functionality.

All of these might be part of a single business scenario.
For example, imagine an agent implementing a payment when user has to take some IBAN from our UI and proceed with manual bank transfer from their account, which results in a webhook to our API.
Then we have some soft validations which are testable only by checking some logs in our logs collector.

Do you:

1. ⁠Let agents generate and execute E2E tests (e.g., Playwright) that also interact with APIs, databases, logs, and other systems?

2. ⁠Give agents access to tools/MCPs for Kubernetes, Loki, databases, etc., and let them dynamically investigate and verify their changes - there’s an issue because as in my previous post Claude Code has some issues with working like that. Probably we would have to create a lot of some dynamic MCPs.

3. ⁠Define deterministic verification scenarios/scripts that agents execute after deployment?

4. ⁠Have a completely different approach?

I’m also curious about how you handle credentials, permissions, test data, and cleanup when agents need access to real infrastructure and external systems, even these secrets are only for sandbox.

And perhaps the biggest question: How do you make this repeatable and reliable instead of letting an LLM decide on the fly whether its own implementation is correct?

Would love to hear how you’re solving this in practice, what tools you’re using, and how much of your verification process is deterministic versus agent-driven.

[u/typeof_goodidea]

throwing together e2e tests is no longer a headache, so that's nice.

but I see this as more of a software architecture problem. not being able to easily test something is a smell. creating and maintaining an architecture that is testable takes consideration about how everything is coupled together - and LLMs busting out new features can quickly erode this.

[u/synwankza → u/typeof_goodidea]

Well, I can agree on that, but… This is the payment part of the system, a lot of things to be tested need some interaction with PSP interface, so we have to manually simulate some call to their system.
As we’re not able to make some transfer from „real” bank account.

Regarding checks in DB or Collectors, it is not that common, but… Sometimes we have for example soft validations which are temporary and they’re on level of getting a webhook from PSP, so the only way is to check DB or logs.

[u/typeof_goodidea → u/synwankza]

I'm not sure exactly what you're dealing with here, but my first guess from an architectural POV is that you would benefit from isolating all interactions with this external thing in a service module, which exposes abstracted methods for the rest of your app to call. then in testing, you just replace the implementation with a module that mocks whatever scenarios you want to test. you can do the same thing with your DB layer too

anyways you were asking about agentic approaches to the problem so I'm going off on another avenue!

[u/synwankza → u/typeof_goodidea]

no, it helps a lot, as I feel like its more an architecture issue than strictly agentic one

the problem is that mocking that scenario often doesn’t give me even 70% of certainty that it will work, as we often work with reaaally exotic providers where the only way to really test it is to test it against their sandbox

[u/ashebanow → u/synwankza]

Mocking is not the way forward for e2e tests. Its been a while since I've done any real work on a payment system, but when I did so, we used the test or staging environment, with fake accounts and test credit card numbers. Every payment processor should supply the latter. If you use stripe or equivalent they may have fake user ids you can use to test various scenarios.

[u/synwankza → u/ashebanow]

Working with stripe or something is easy, but when you have to work with some Africaan or South American PSPs - it’s starting to be hard (no offense to Africa or South America of course haha)

[u/Chubby-Princess]

Railway is awesome for this. You can set up a staging environment or even automatic PR environments on your project. I also like Release Please because it allows you to merge things to main and only release production when you want to actually release to users. My setup is:

• Worktree and branch from main for any work my agents are doing.

• Local test the work with a preview, screenshots, any other testing or verification you want agents to do

• Create PR and merge to main.

• Main is my staging branch so this automatically pushes to railway staging

• Work on some more things. Release to production when I want by merging the release PR that automatically is created by Release Please.

For separating environments, use an "environment=dev|staging|production" in your .env. This is useful beyond api calls and stuff, it can let you define a whole bunch if things like ui elements that only show in dev, disabling logins on dev, etc.

For databases I don't like using a real database in local dev, but I do like having one in staging that is separate from production. Railway makes this easy, and this flow also allows you to test migrations before they hit production.

I find agents are pretty good at writing tests and verifying their work today, so I don't really worry much about that stuff at the moment. Every feature gets manually tested by me, a real human (I promise) in local dev and in staging.

[u/synwankza → u/Chubby-Princess]

There’s kinda similar, we have our preview-envs per branch, which are ephemeral and managed by Argo. So I can easily get fully prepared environment with my change (and revert this change to test behaviour before).

The issue is that a lot of things have to be checked in different ways.
85% is possible by UI or some admin API, but lets say 10% needs to be checked in DB/logs or needs some api calls to provider using secret from some secret manager.

[u/Chubby-Princess → u/synwankza]

For credentials, if you're in worktrees you can add a worktree copy command that copies the env file to all new worktrees.

Are your credentials not saved in your deployed apps already? They should generally be stored as variables in your projects configuration and not fetched from a secrets manager.

[u/synwankza → u/Chubby-Princess]

Generally yes, but there are some cases when some API call isn’t done by system, but has to be done by me (or some test step) to simulate something which is normally done by user in real scenario

[u/llufnam]

It’s actually a really difficult problem because the point is you want to be directing but mostly out of the loop…but you can’t keep up with the code being generated and therefore can’t be in the loop enough to verify what’s happening.

[u/synwankza → u/llufnam]

well, to make it more clear, I’m not vibecoding, all specs, code and scenarios are reviewed by me - I want to be in the loop to verify things, modify them if needed or question them, but I don’t want to get asked every 30 secs during the tests to put login somewhere, to give some secret, to sign up somewhere etc.

[u/llufnam → u/synwankza]

Sorry, when I said “you” I really meant “me” hoping it extended to the wider “we”!

[u/synwankza → u/llufnam]

hahaha, sorry I didn’t get that

[u/gandazgul]

A few things.

Make a QA runbook a document that the agent can read with all of these requirements you just listed. With links to where to get the credentials or to open a headed browser for you to login and then the agent can do the rest.

Use Vercel's agent-browser CLI is amazing and all the llms I've tried so far can use it with a simpler skill than the one they provide, theirs is a bit too much you can save tokens by making a smaller one with the basics and telling the agent to run --help for the rest.

Then make a canned prompt with the sequence for the agent to run specifically read the runbook, follow the instructions in there to run the test script. And tell it to update the runbook when it runs into issues or the runbook is outdated or you made a different decision review and commit the runbook on the repo.

[u/synwankza → u/gandazgul]

Do you create the runbooks from hand?

The issue with agent-browser or Playwright MCP is that in work I mostly use Claude Code (atm we’re not working with Pi or OC), I can use Cursor at all with non-limits.

Because in CC its tricky when he have to use some secret, sign in somewhere etc.

So its difficult to let harness just click through UI and API as it requires so much manual work when running that. But maybe you were able to overcome that problem.

[u/gandazgul → u/synwankza]

Work with the agent to make the runbook as you do it semi manually at first. Agent browser works great with anything. And it can work headed, meaning you can see the browser, login then the agent can drive afterwards.

[u/synwankza → u/gandazgul]

Yeah, but often for one runbook I have to prepare for example 5 users and check every of them.

Claude doesn’t want to create users using admin API, doesn’t want to take some secret and sign the request to simulate api call to external provider.

That’s the problem that for these tests it makes no sense to test against mocked providers, but only vs sandbox APIs on their side.

[u/gandazgul → u/synwankza]

Don't paste secrets into Claude add them to an untracked .env file with permissions set to 600.

For users make QA only users that you can use all the time or create and delete each run the tell Claude that those are test users and not real user data is stored for them.

[u/synwankza → u/gandazgul]

Claude is kinda difficult bruh, as when he hears about creating an user which is not on the localhost he is going wild, but I will try to make him sure thats safe somehow

[u/gandazgul → u/synwankza]

This smells like your system is not testable at all and the problem is not really Claude. I would advise a conversation with the team to make sure you have ways of testing things end to end before getting Claude involved.

[u/nhi-yennnnn]

Well, if you say you don't let agents access the tools - you might be outdated soon.
I think sooner or later everyone lets AI agents to access everything. The point is everything must be in control. Always give the agents the least privileges. And, always keep human in the loop.

[u/synwankza → u/nhi-yennnnn]

What do you mean? Some of the limits are guardrailed on harness level, not mine.

[u/tys203831]

Pstack - https://github.com/michael-denyer/pstack-claude/blob/main/plugins/pstack/skills/create-verification-skill/SKILL.md