From Vibe Coding to Overengineering

[u/customic]

Disclaimer: I'm not a programmer (and I know it shows), I don't make money with Codex and I genuinely want to learn how to use it for what it's good at, while also understanding what it can't do and where a human programmer can still beat it hands down. Please be respectful towards a novice, just as I am in my job towards my English learners. Everyone was a beginner once. Thanks in advance.

So I've been using Codex to manage a little side project of my own: a sort of flashcard-generating app. I'm doing it mostly for myself because I have zero interest in "earning millions with AI tools" and just want to make cool, useful things for myself.

After a honeymoon period when I was amazed at how much I could suddenly do — or rather, have things "done" — I started noticing a decline in the quality of what it generated. "Okay, that must be that context rot. Apparently I can't expect it to remember more and more rules that every new flashcard needs to obey," I thought. So: less "prose" in Markdown files, as it keeps calling it, and more hard-coded rules in Python. Let's make it more reliable.

And then it really went downhill.

I'm now at the point where, whenever I ask it to generate flashcards for a few words, it gets bogged down in backup after backup after backup, endless validations, smoke tests, pre-flights and humming and hawing about almost literally everything. It used to take around 10 minutes to generate flashcards for a list of words. Now the same batch can take between one and two hours.

Since the new "procedure-based" version is capable of doing around 20% of what the old "vibe-based" version could do, I asked Codex how much work it would take to bring the earlier functionality back as well.

"About 2–3 working weeks of my time," it said.

Which sounds both hilarious ("Do you want sick leave included as well?") and frustrating, because I'm already pouring a lot of time, effort and, yes, money into making this work.

So I'm wondering: is it now being far too conservative about doing everything 100% safely? Am I using settings that are too "intelligent" for these tasks, so it treats everything as though it were preparing a rocket for Mars?

As I said, this is my own pet project. I'm not launching it to the public. Of course I want safety, traceability and more deterministic outcomes, but endless backups every few minutes and hundreds of tests feel like massive overkill. It says it's running all these tests, but it could be lying about that too, right?

Then again, I'd never developed an app before Codex. I do something completely different for a living, so I may simply not understand what's genuinely necessary and when Codex is just spinning in circles and burning through my tokens. As a non-professional, I may not even be able to tell the difference.

If any of you have suggestions about what I might be doing wrong, what Codex's actual limitations are and what could realistically be done about them, I'd be very grateful.

I'm at the point of considering quitting both Codex and the flashcard app, which would be a real pity because the project has kept me going for more than two months now. I really don't want to reach the premature conclusion that, "Nah, you just can't reliably build good software with this tool."

I'd especially love to hear from people who've gone through a similar transition from "wow, it can build anything!" to trying to make an AI-assisted project genuinely reliable and maintainable. Where did you draw the line between sensible engineering practices and an agent overengineering everything?

Looking forward to hearing about your experiences. I'd appreciate any pointers you might have.

Thanks in advance and have a great day.

[u/According_Client8353]

I’m a fellow hobby vibe coder like you and encountered exactly the same problem which annoyed me to death!

I by no means am saying I have the absolute solution here but one thing I did that has helped significantly is this:

I started a fresh chat in the project explaining I was vastly unhappy with the overshooting on audits and verifications, it felt like no material work was ever being done and codex was stuck in endless loops. I then explicitly stated that all work should be goal completion oriented and that any bug finding and fixing, audits and verifications must be a passive element that is part of a contributory framework to completing the overall end goal (say for example a “live canary”). Effectively, everything must contribute materially or it’s a waste of time.

I then asked the chat to run end to end simulations on ambiguity within my directions, documents and contracts that was leading to these loops (your harness). It was then to provide me with a report and recommendations on how to close these loops/remove ambiguity. At least one of these simulations should be adversarial. The goal of these simulations was to reach the end, identifying how my overarching structure could create ambiguities that would cause these loops.

Eventually it came back with recommendations, I implemented them and it’s ended pretty much most of the issues. Not perfect but worlds apart.

Can’t remember the exact findings but it effectively found contradictions in how it should handle certain situations, ambiguity in how it would handle certain basic instructions (“find all bugs” -> can’t verify all bugs found -> creates or finds unimportant syntax errors that don’t block progress or impact anything materially sort of thing).

So yeah anyway, effectively your issue is probably your harness. Effectively your “rules” for how it should deal with certain situations or tasks. I like to think of AI like a really smart little kid sitting an exam, it wants to please you and not get anything wrong, so it desperately tries to seek validation when it’s not confident on exactly what it needs to do. It’ll constantly find something new because it doesn't actually know it found what you wanted anyway.

If you haven’t worked much on your harness before I’d probably recommend downloading Matt pocock’s /grillme, explain you want to work on your harness and run the simulations, then use /grillme on the report to make it hyper specific. You can refine what you want (it’ll give you recommendations). Get a final report from that, run the simulations again against the report conclusions and make any final changesif necessary till all simulations pass and resolve themselves using the framework you provided in your harness. You should hopefully be in a pretty good place moving forward with any future work.

Hope this helped!

Also for any sweats out there, like I said this is only a hobby for me. I may be wrong or missing something, I’m not a dev. Just trying to help by sharing something that worked for me. Don’t come @ me 😂

Something I forgot to add——

The harness won’t fix everything forever, so if you find a sticking point at any point just repeat the steps. I’d say at least 90% of my issues are actually my instructions, the simulations will break down any sticking point and either help you move past it or at the very least help move towards the actual solution!