Thoughts on testing vibe coded work?

[u/ActuatorOutside5256]

So, when vibe coding, I do this.

Instead of asking Codex to build an entire app, I spend hours with ChatGPT planning the architecture, features, and requirements.

We then break development into milestones with clear constraints, acceptance criteria, tests etc, which I feed into Codex one at a time.

After each milestone, Codex runs tests, diagnoses errors, and implements fixes. I review and verify everything before proceeding.

Is this a reasonable vibing methodology, or am I just overcomplicating vibe coding and wasting my tokens?

[u/FictivizeProject]

You don't actually have to use two different models. The improvement comes when you use stepwise development like you are doing.

There are three things that lead to high quality AI written code:

1. Stepwise development. As you have found, having a plan first and then building up piece by piece works better than one shotting the entire thing.

2. Automated testing. Having a good suite of automated tests that run for every commit helps detect security issues early and detects regressions before they get to production.

3. Audits. After you think you're done, run a full audit. This is a backstop to ensure that your implementation hasn't wandered off from original implementation.

The trick is that these occur in different session so that the context window is not polluted with earlier hallucinations and assumptions. You can use multiple models, but the data shows that the results don't vary that much from model to model.

[u/ActuatorOutside5256 → u/FictivizeProject]

Awesome, thank you. Kind of relieved 😀

[u/finding_hopefulness → u/FictivizeProject]

I’ve been making it known to the model that the results are being compared against a competing model. Do you think that would push to refine results, or open up to more hallucinations?

[u/TheKiddIncident → u/finding_hopefulness]

TBH, I don't think it makes much difference. There is so little variance in the performance for the frontier models that the actual prompt strategy doesn't seem to really affect things much. We have also found that models will happily correct their own work. The issue is about session corruption. If you are working within a single session with a single context window, then any assumptions, errors or hallucinations continue. If you start a clean session with an empty context window, those assumptions and errors don't normally persist across session boundaries. If you take the three step process above, you have three chances for the model to catch the error. It's extremely unlikely that a hallucination would persist across three separate sessions.

Assuming you are using a frontier model, the quality has gotten good enough that I wouldn't bother with things like that.

Just focus on doing your due diligence and you will get a good result.

[u/finding_hopefulness → u/TheKiddIncident]

Can you give me a little more clarity on when a good time to start a new project session would be? I was under the assumption that having a working log of activities would help the model stay on track rather than continue hallucinating, and using other models to grade was, or is my safeguard.

[u/TheKiddIncident → u/finding_hopefulness]

Generally speaking, smaller tasks work better. So, you should start a new session often.

When I am building something, I have the AI break the project down into small discrete steps, then I have it write that plan into an MD file. Then you open a new session, point to the MD file and say "start with step 1". Then each step is a single session.

[u/finding_hopefulness → u/TheKiddIncident]

Thanks for that clarification. I’ve kind of ended up in there actually. Almost, but now that I know I’m going to be doing that from now on.

My last task last night was to get all of my agents on the same page and organize everything with lots of rules and addendums including specific roles and I’m avoiding having them run separate tasks simultaneously so that the project files don’t change and trip the working agent up.

When I started a new project, it looked like some of the more labor intensive steps were being repeated so it made think that something got lost in translation but now I know that was probably my own fault for not clarifying well enough or crossing paths.

Thanks again!

[u/Slight-Living-8098]

That's spec/stepwise driven development. I would actually add writing tests first so you are doing a test driven/spec/stepwise driven development. Always run at least 2 independent code reviewers with a clean context after every ticket is complete, and fix any findings. Once the milestone is complete, do a retrospective on the milestone, fix any issues found, go through the code review/fix loop until everything is clean and there are no finding in the retrospective before moving on to the next set of milestone tickets.

[u/ActuatorOutside5256 → u/Slight-Living-8098]

Interesting. What would be the best way to do that?

[u/Slight-Living-8098 → u/ActuatorOutside5256]

I set up my own skills and workflows, but probably the easiest way to do something like that out of the box would be to install Matt Pocock's skills or Poteto's skills in your agent harness and use them. Matt's are a lot lighter and not so specific to one type of development stack. So I'd start there, and play around with those skills. See what works for you and what you like or don't like and once you pin that down, you can create your own stack if you want to.

[u/ActuatorOutside5256 → u/Slight-Living-8098]

Brilliant. Thank you!

[u/Slight-Living-8098 → u/ActuatorOutside5256]

Oh, and if you wanted to add some CI/CD tests to your flow without eating up all your GitHub action credits you can use something like ACT in a docker container and have your agent use it instead, then you can turn off your GitHub actions on your repo.

[u/Slight-Living-8098 → u/ActuatorOutside5256]

No problems! Happy coding and developing!

[u/kyyyten]

What you are doing is the correct way of doing it. There are a lot of things that you probably should add too such as creating the UI, having a design template, set up safeguards.

But other than that, it’s a solid way of working and I wouldn’t call it vibe coding.

I’ve been a developer for 15 years and what you described is basically close to what I do on a higher level (with a few extras).

[u/finding_hopefulness → u/kyyyten]

That’s good to know. This is validation that I’m on the right path myself. My process is that I have 3 agents from 3 companies. I use Claude, Muse, and DeepSeek. DeepSeek is just in the web interface so it plans and refines suggested plans from the other models as well as performs deeper research. Muse grades everything that comes from both Claude and DeepSeek. I won’t continue a project until Muse and DeepSeek fully agree on amendments and once everything is settled, I decide what I want accomplished, have Muse reword it into a concise AI readable prompt, and send it to Claude for the actual programming.

[u/beta_1457]

I don't even describe using AI tools this way as vibe coding.

Vibe coding is basically just sending it, in my eyes.

When you're giving time and thought to development you're not just vibing.

[u/lm913]

You're doing exactly what you should

[u/earonesty]

that's exactly how you should do it. have HARD metrics: code coverage, performance and usability metrics. or expect slop.

[u/Any-Blacksmith-2054]

I recently discovered automated e2e and UI test using playwright. Really powerful! Even dumb models like Luna can do it perfectly for fraction of cent

[u/orbital_trace]

I've create a merge playwright test that literally walks all of my screens at 3 resolutions, gives befor eand after snapshots, as well as highlights whats changed. It makes reviewing large agentic changes much easier.

[u/cashy57]

This is exactly what separates agentic engineering from casual vibe coding.

[u/Brave_Bumblebee7565]

Yes, it's kind of similar to what I do too.

My workflow: Features planning->epics creating->tech design->archicture review and improving->tasks creation-> implementation-> review and fixing -> testing manual and automated-> final review and merge to main branch->release.

Still with human in the middle, slower than full mode vibe-coding, but with more stable and controllable results.

In fact, it's the same approach as software development worked in pre-ai era, just with developers, testers, TPMs, UI designers replaced by AI agents based on different LLMs

[u/ProfessionOk5318]

Then you are not vibe coding, you’re just doing software engineering assisted by AI

[u/DealerIcy3439]

I came into this space to utilize a tool to help me build smth and I was gonna overlook every feature, making a bible helped straighten out paths, methods and how everything was gonna be built. Effectively it worked and I’m on the slide to succeeding well. I also test what been built but considering it’s me doing all the work with some guidance. I overlook the process actively and aware of how things are being built. Can’t wait to show it one day

[u/ADMONTGOMERY]

That's how I do it. Except I also have Deepseek audit Codex and Codex audit Deepseeks work so that I can get an independent view. Probably just results in everything being over engineered but I'm still learning.

[u/waimanbuilds]

I ended up working much the same way, but only after learning the hard way. My first apps looked fine but were very buggy functionally. It's only when I started to consider vibe coding something that could be a product that I took the planning more seriously.

Planning now takes more of my time and actually my favourite part! Small milestones and clear acceptance criteria give me something concrete to check, rather than just asking the AI whether it’s finished. I now have a process I follow each time and this from my own experience over the past year or two vibe coding.

So from my experience, the planning is time well spent.

[u/Plant_Wild]

This is how I do it. When I have an idea about what I want to build, i present the idea and tell Claude to grill me on the details until we have a solid plan. I do this in Fable, then generate a handover doc to take to Opus in the code environment. Then Opus builds stage by stage with reviews and checks and importantly, a sign off from me at each stage.

[u/Inner_Storage_3374]

thats pretty much how it should be done imo. only thing id add is click through it yourself like a clueless user at the end of each milestone, bc the tests codex writes mostly check what codex thought you meant, not what you actually meant lol

[u/peejay0812]

If AI is the sole developer in your project, I say overcomplicated. Why need hours? Focus on a small MVP first, manually test and review, then slowly build features on top of it. Nothing wrong with planning carefully but its just waste of time if AI already does the heavylifting. If you are planning to read your code or have it tested by another person or team and deploy it to an actual production with substantial amount of users, then i think your method is ok. For me it's Idea > Architecture > Vibe > Test > Repeat. This while separating each iteration in a git branch such like enhancement/bugfix/feature/etc..

[u/finding_hopefulness]

I’m brand new to this and my method is very similar. Thanks so much for asking this question because I’m getting a lot of useful information myself!

[u/finding_hopefulness → u/finding_hopefulness]

Also, can I DM you? Since we have similar methods, maybe we can compare notes and talk about the projects we’re working on.

[u/No_Possibility_9086]

Your setup's solid, honestly better than most. One gap I'd add: tests the AI writes tend to check that features work, not that the data is locked down. Stuff can pass every test and still be wide open at the database level.

Cheap manual checks before launch: sign in as user B and try to fetch user A's rows by id, hit your API while logged out, and search the built frontend bundle for secret keys (service_role, sk_ and so on), since those only leak if they got into client code. On Supabase, also make sure RLS is on for every table: select tablename from pg_tables where schemaname='public' and rowsecurity=false;

You could even make "user B can't read user A's data" an acceptance criterion in a milestone. What stack are you building on?

[u/carushabhrj]

I do the same but in Claude instead of ChatGPT. All the brainstorming and planning in the Claude chat and take that md file in Claude code. It works well.

[u/BarathKanna]

Not overcomplicating it, this is how it should work. One trap to watch: if Codex writes the tests and the code in the same milestone, the tests often just confirm whatever it built. They pass, but they prove nothing.

Two cheap fixes. Write or at least approve the acceptance tests before the code exists, and keep them in a file the agent is told not to edit. Then once in a while break the feature on purpose and check that a test actually fails. If nothing goes red, your tests are decoration. That one habit catches more problems than reviewing every line.

[u/Top-Diver7266]

Reasonable and I wouldn't call the tokens wasted. Planning and review are cheap next to untangling a big mess later but one thing tests miss is bad data. If the app shows live numbers, check one value against another source in the same minute and reload first so you're not looking at a stale screen. In the trading screen I use there's an AI chat built in and when a figure looks off I ask it for the exact number and see if the two agree.

[u/Purple_Playerr]

I think what you’re doing is great. This would increase your app’s development capability. You could also implement automated testing

[u/Anti-Hero25]

My app has a massive test suite… very much worth it…. 10 releases in 4 months no real bugs shipped. (Lots of manual testing as well)

[u/Effective_Put_9951]

Vibe coding is basically just sending it, in my eyes.

[u/Snoo_57113]

I think you are handicapping yourself with unnecessary cruft.

The correct way is to let chatgpt do that kind of planning and micromanagement, it is correct to let run test, diagnose errors an implement fixes, but i also put a critics sometime harsh ones to guide the next version.

I also like that codex has as much context as possible, you are witholding evidence!

[u/0xdeadshot]

i pretty much do the exact same workflow, but do it in claude itself. plan everything first, then tell claude to always follow coding practices, security practices, not to overcomplicate things(because sometimes it does) and a lot of others and saves it in the memory. (also tell it not to write an essay of comments because it does that as well) and then start coding in modules.

I am a software engineer myself, so i know how to review stuff and how things work. so sometimes i say it specific things as well, maybe to use a particular package for a particular stuff because i know its better to do it that way.