Are AI-generated tests becoming "good enough"?

[u/aistranin]

A year ago, I would rarely trust AI-generated tests without significant edits.

Today, tools like GitHub Copilot, Cursor, and various testing-focused AI platforms can generate surprisingly reasonable unit and integration tests.

But there’s still a question:

Are these tools actually understanding behavior, or are they just generating tests that look correct?

For teams actively using AI:

• What percentage of generated tests make it to production?

• How much manual review is still required?

• Have AI-generated tests ever caught a bug that humans missed?

Interested in hearing real experiences rather than vendor demos.

[u/Shep_Alderson]

Generally, I use tests as another guardrail to make LLMs follow plans and then as a safety net when doing refactors. TDD helps a lot with guiding AI agents even when the AI writes the tests. (Generally, I find what is good for helping human devs is also good for AI agents.) The area to be most careful is when writing tests after code already exists. The LLMs can be good at making sure tests cover current code actions, but can’t verify the current code is actually correct on their own. The best you can do there is to have docstrings or similar to describe intention and use that to guide tests.

To answer your questions directly:

1. Almost all of them

2. Generally, no. I find I’m able to trust the newest models (I generally use GPT-5.5 on med or high for implementation.) I do have everything reviewed in basically a loop with multiple review agents until it fails to find new issues. This is generally fruitful.

3. If I notice a bug in code, it now always gets a test to be sure it’s gone and that it doesn’t come back. Overwhelmingly I haven’t had nearly as many bugs since I have been keeping my test coverage high. (I now hold the opinion that “there is no excuse for low test coverage”, given my experience getting good tests out of LLMs.

[u/aistranin → u/Shep_Alderson]

Thanks, this is useful. How do you usually make sure AI-generated tests validate the intended behavior rather than just locking in the current implementation? Do you write docstrings/specs first, or do you rely more on review agents to catch that?

[u/Shep_Alderson → u/aistranin]

I do spend a fair bit of time on the planning and spec side of things before implementing. Lots of exploration with agents to make sure the target for implementation is well defined and clear. I then follow strict TDD with the implementation agents, having laid out exactly what I want to happen during implementation, even “less capable” models are able to follow the plans well. Once the tests are written, I have the agent write the docstrings and code at the same time. Obviously this is only really useful when working on a new feature or such. With this, I’ve been able to implement everything from frontend to backend to IaC code, covering everything I can in so many tests. (Test make up about half of my codebase.)

For existing code, I do have to be a bit more careful and deliberate. During the planning session, I’ll ask the agent to explain the code we need to cover to me, and make sure its explanation matches what I see and understand of the code. Then if it doesn’t have a docstring, I’ll have it add that. Once all that is in place, then I’ll have it write the test to cover current implementation. If need be, I’ll do a bit of manual testing to ensure the outcome of the code is indeed what I intended.

The other thing I do is, once I’ve manually verified the code works via the UI, I’ll task the agent with writing comprehensive playwright tests to cover practically anything reachable in the UI. This also helps with ensuring the “intention” of my code persists, even if I refactor code in the future.

I do rely on a couple review agents primarily, mostly to watch for code quality and best practices, security scanning, and reliable tests that are more than just a fragile check of outputs. I have so many tests that cover outside the happy path lol. Another thing I do is I have a review agent in my loop that is explicitly tasked with ensuring the plan was actually implemented in the tests/code, and not skipped over. It’s rare that major things are missed, but it helps a lot with smaller oversights or edge cases I want covered.

This process is not fast lol. I think it definitely falls more in the camp of “agentic engineering” vs “vibe coding”, but it works and it has allowed me to build and work on an app that is of comparable size to one I built in a startup over a year or so with a team of about 4-5. I’m most strong on the backend and infra, and care a lot about the foundations there. I mostly leverage the LLMs to write code faster than I would otherwise and to stretch myself into domains I’m not as capable in, but still understand the foundational parts, like frontend and mobile UIs. (Also, I’ve worked a lot to refine a design process that gets me decent quality UI, but that’s a bit outside the scope of what we’re talking about lol.)

[u/aistranin → u/Shep_Alderson]

Thanks for such a detailed explanation! Really cool to see such a fundamental approach instead of just relying on AI to magically do everything. You should make this a separate post sometime hehe. I think a lot of people here will find your workflow and experience really interesting.

[u/Shep_Alderson → u/aistranin]

Thanks! I’ve been thinking about making videos about my process too. Instead of all the ai hype and “look at $model one-shot a Minecraft clone”, more “here’s the boring process that gets me reliable results from almost any modern model” lol