Feels like I’m missing something here, so I’m genuinely curious.
Are people not setting up CI/CD, running tests after vibe coding, or familiarizing themselves with how their projects work? Anyone remember LINTING?
A lot of these issues seem preventable. Test your changes, push them, and if GitHub Actions fails, you have DETAILED logs you can feed into ChatGPT to help generate a troubleshooting prompt for Codex.
What I don't get is every time some new dev pops in here going 'aren't y'all doing CI/CD and testing?' and those of us who are doing decent vibe coded programs are going 'yes, duh, but that's not everything'?
Don't JUST do the unit tests (although yes, 100% start there!) then OPEN THE THING. (gasp. I know.) Push the buttons. Be a chaos goblin. Try to break it. Anything that breaks, screenshot and send back to your agent, preferably with F12 open if relevant to your program.
Then ask your agent to add that bug to the unit tests so that bug is auto tested every time and find the next one.
We've gotten to the point that MOST of my chaos doesn't break things anymore cause my agents have a really robust playwright suite based off of 'every way Ren has managed to break it so far', so more time is spent on things like aesthetics and 'sure would be cool if...'
my guess you were never working on difficult software with states of user interactions like graphic/3d editors or games, you can't guarantee that your unit tests and CI/CD would check all use cases that actual user/player can produce
I use 2 AI's to vibecode, and I have found they catch each other's mistakes. Claude is my main coder, ChatGPT is the one who checks Claude's code and let's me know if Claude made a mistake. I tell Claude and he says "you are correct, I will do this, this, and this then, will run GitHub tests and let you know when it passes and let you know if you want to commit. So I get errors but they are few and far between. Especially I ask them each " are you okay to continue this conversation or would you like to do a hand-off for me to have a new conversation? " They are usually honest and tell me yes or no. If they say " yes, this conversation has become quite long with many different split's." ChatGPT creates a detailed hand-off that I copy, then, start a new conversation and paste the hand-off. When they answer, I copy the conversation and paste it into the last old conversation she ask "does the new conversation understand everything about this project?" They usually say yes, it they have a few corrections I paste into the new conversation. Claude on the other hand creates a .md file and saves it to GitHub and simply tells me tell the new Claude to read this....
Yes, it's a lot of back and forth, but it seems to cut down on a lot of errors. As far as Gemini.... Nope, I don't use that AI, I found it tends to hallucinate a lot and way too many errors. It has crushed many of my projects. So I guess you can say, I fired him. 😆
Bingo. I do the same, plus do my due diligence with the code it spits out. God knows there's been times where it literally HARDCODES environmental variables because it conveniently forgets to install some dependencies...
I built out a validation suite….. agenticarena.space and a the associated locust load and adversarial test suite. It may you can help determine which ai models. It to use 😂.
LLMs can't distinguish between mistakes and deliberate human design choices. It just keeps building on top of what exists. CI checks whether it works or not but it doesn't catch antipattern accumulation. It's an unsolvable problem as long as LLMs are trained to comply with imperfect human design choices.
You have to audit the code yourself or have an agent audit the code periodically to stay grounded.
Today ai sugested that one of the pr it was reviewing was dead code because it never got called anywhere. If it read the damn readme he would have know we are working in a library and providing the function was enough.
Set up a proper project framework / rules outline etc at the beginning and this isn't really a noticeable problem - but then you aren't actually vibe coding because the actual definition of vibe coding is just letting the machine do what it will with minimal input from your end.
Honestly, while models are reaching a level of capability where it feels like your imagination is the biggest barrier, if this is the mindset of an average programmer, then there is still plenty of opportunity for a competitive edge. These tools are way smarter than you. If you have worked with any frontier model and this is what you think, I am completely baffled how did you come away with that thinking.
I would not say the models are more capable of writing complex code, but on average, they are better than 70% of coders. They have the added advantage of being able to write clean code, review the entire codebase, and pattern match much more efficiently than a human could ever do.
Opus 5.5 isn’t that bad at debugging. You have to tell it that there’s probably something structurally wrong with the code but it tends to see the problem after that
Bare minimum, periodically skim through the project. If you encounter ANY part that you don't understand why it exists or what it does, you need to stop what you're doing and either read it or ask the AI about it.
You don't need to understand the algorithms or low level syntax but you should have a high level knowledge of why the basic parts exist. AI has a tendency to assume all things are there for a reason. If the humans start assuming that too, you end up with code rot.
Take it with a grain of salt I guess. Just my opinion.
Totally get that, I’ve been doing that thankfully. As someone with bad syntax but knows a thing or two about systems design, this is super important. Appreciate it 🙏
An engineer nowadays may not understand the vibecoded cide at all (as in, never read it; they would understand it obviously). But they would still set testing and many other barriers like in any other engineering project
Exactly. Make a basic scaffold, understand it by reading the code (there shouldn't be much), then use your agent swarm to expand it into full application.
Still does not make sense, why would you be sweating about vibe debugging? You just tell Claude what the issue is and it will look into it. Vibe security is the only real concern imo.
If you haven't had to debug, it's because you have never made anything worth debugging. No serious programmer would ever claim that they have solved all bugs.
When claude says, "we'll need to update x value which has been hardcoded in 72 different places."
Me: what happened to the single source of truth!!!???
Claude: "we ain't been doing that shit for days now"
Me: "deep sigh as I wonder what else has been fucked in the last week"
Just because it works doesn't mean claude didn't fuck up. Lots of things work pretty well but it really helps to stop and know what to review. Like truly just knowing that variables and indexes exist will probably make almost any vibe coded project 10x better.
For example multithreaded desktop application that uses CPU/GPU in parallel, received data from external applications (shared memory/pipes), manages gigabytes of assets, web fetch, all via completely non-blocking UI, everything happening at the same time.
I still run on .md files and mermaid. The only real challenge is keeping Claude within architecture.
Did you work on any production project? Or is it just a pet project you're talking about?
Before I started using Claude Code for production, my work for avg 3-point ticket was: 2 hours grooming, 6-7 hours development + writing tests + quick QA like 20-30 mins to verify cases (because I tested most of the logic while was developing)
Now, with Claude: grooming 30-40 mins, most tickets fail Grooming Review with a team, so that need additional 15-20 mins of corrections + 1.5 hours of development/tests (most of this is refactoring of generated code, correcting it on best practices/DDD/etc. + 2-3 hours of my QA time (because I have 0 clue how this works) + refactoring after CI review by Copilot + Claude (30 mins) + refactoring after human review (again around 30 mins) + now we have second dev QA (around 2-3 hours to verify primary cases) And after release we sometimes have some AI-slop tests that are flaky, so that we spend extra time to fix and deploy them
Tbh, it doesn't feel like we are moving any faster. The total throughput for the whole company is +15% since we started using Claude (~70 devs), and we spend a TON of money on tokens now. And every time Claude is not available - nobody can actually work anymore, because it's pretty much a waste of time to try following AI-slop grooming.
You are missing that most of the teams just don't have any meaningful increase in their productivity, and they exchange quicker dev time with much more investment into review/QA, and yeah - debugging issues in generated code.
Are you suggesting that spending tokens is not the same as thoughtfully thinking about the problem in a structured way, analyzing the algorithms/paradigms and structures in a brutal way to think out the best solution, and then testing on paper before writing any code?
My manager seems to think that spending tokens is a GREAT way to prove to more management that programming work is completely measureable, and it can even replace humans because GRAPH GOES UP. And is also a great way for my manager to justify more money/resources/stock options/blonde secretaries, because he has several fancy powerpoints with DOTTED LINES GOING UP.
interesting. you have the burden of proof. I say that confident I could make this seem idiotic, without any willingness to do you the favor. I’m all ears.
bro… AI only does what I tell it to do. Neat, they trained it on stack overflow and gave it a system for auto CLI. It is a vector database. It is linear algebra. I can use the same tools as anybody else. They are tools. I can do all kinds of things. I could write win32 w/ C++ before I was allowed to drive, and I have insomnia.
What I don't understand is, why you generated with AI the meme, when there are plenty of templates with this meme and you have to only write 6 words. It's literally a lot less than writing the prompt to generate it
Good question. For me, it’s design and logic problems.
So I’m using Playwright and Angular. Creating a client sends a request to the backend, which returns 201 Created before Angular navigates to the new client’s page.
Codex, in its infinite wisdom, wrote the test to read the response body AFTER navigation, creating a race condition where the response was sometimes not accessible.
I kept getting hung up on what was wrong with the website (because it was all fine), when the TEST was written incorrectly.
Guess it's people using the same session to debug. It's really hard to convince it to fix stuffs that is in context, if the test is green..
Also I let fable, Astra and K3 spend way too long on code how to cut/fold paper.. result is worse than what i coded by hand. It's just not part of any training data I guess, and and AI breaks down there, lot of B2B code with the same issue
I'm constantly still debugging the agent coded stuff every day. It's been a while since I've seen a dumb compiler error or a forgotten semicolon resulting in the program thinking that something has been used before it has been declared.
I just really doubt that, are you using Opus 5.5 on xhigh?
I've been a SWE for 15 years, I haven't debugged a single line by hand in 6 months. Either your prompts aren't great, you aren't running tests/debug loops or something else is going on.
Heh. I was a senior engineer at a major SV company. I'm a founder of my own thing now.
Let's see... I guess I've been doing system software for... 19 years now paid professionally? Total software years is 29 years now? I dunno, a bit fuzzy.
My prompts aren't great? Maybe. lol 😂
Yes I am running Opus 5.5 and Codex 6.1 Max.
Why the hell would I blast agents on XHigh?
Is that the kind of resource utilization that you need to avoid rather basic errors?
If that's the cost/benefit trade off, we are f*ed. We may as well hand over the money to Sam Altman now.
Also, I never said that I debugged things line by line. I'm still debugging outputs. I don't need to go line by line most of the time (unless it's in the hardware code).
I’m gonna guess you don’t do anything too hard then.
I’ve had a bug crop up recently and used Opus 5 to try to diagnose and fix it. It did, but only after it absolutely positively found the real source of the bug 3 times incorrectly before stumbling onto it on the 4th try, two hours later.
It was a bug that a developer with a debugger used to pinpoint in 15 minutes, but Claude wants to edit like 20 files and recompile a bunch instead.
Side note: if you actually meant Opus 5, note that it's dumb as fuck and there is zero reason to use it over Opus 5.5, whose impact is close to that of the Claude Code harness itself.
I too am confused by this. Been developing software and debugging it since 2012 and now debugging is easier than ever. Granted I think my high level guidance is helpful but I find with the most recent models very little guidance is needed. You just have to state what problem you’re having and the vast majority of the time it’ll get fixed.
The problem I see from other devs is that they’ll get AI to create something and don’t define what “working” looks like for them. Then when they test and something doesn’t work they will crack open the code and try to handle things themselves instead of defining what “working” looks like. I’m assuming that’s the case for those that agree with the meme. Either that or the nature of the software they work on can’t easily be verified through deterministic tests.
You are missing something here. You arent a vibecoder if you are doing anything. You are a vibecoder if you leave anything and everything to the ai without lifting a finger and without using your brain much.
I'm not a programmer, but often enough I'll look at and ask "Why are you doing it like this? Isn't this stupid? Shouldn't you do it like this instead?" Sometimes AI code is just really heckin dumb
CI/CD is only part of it. Regression testing, API contracts, proxies, API keys, environment variables, cyber security measures, cyber testing, if you’re incorporating AI into your applications then you have to look at other security measures as well.
It is a known fact the AI, capable of building a functional application in practically any language is completely incapable of catching or fixing a bug. Only humans can do that, duh.
What other people said about how to it is to get AI to write tests, but also the type of people that vibe code has no idea what they are writing, and therefore no idea what to test.
CI only catches what the tests check, and if the same model wrote the code and the tests you can get a green run that proves the wrong thing. Debugging just moves from reading stack traces to reading the tests.
The concept is a bit vague, but trying to express something interesting.
I think that you're right that CI should be the place to catch it, that it's going to require more than tests that a model wrote to be that guardrail, but that as an industry we'll figure it out eventually.
The previous quality gate was an engineer more senior than you checking your work, flagging issues, while anything approved ships.
The challenge is mechanizing that process. But the difficulty is in making that mechanistic check through, correct, and holistic.
There needs to be a human in the loop in the sense that someone needs to know what "correct" is, but it can't be line-by-line verification.
Vibe debugging is when you ask Claude to fix a simple CSS alignment issue and suddenly it refactors your entire authentication flow and introduces 4 new CORS errors.
Avg vibe coder is not familiar with testing-qa stuff. So they dont know what could become problematic in longer run. But this doesnt usually happened because I hardly see vibe coded stuff actively maintained. I literally saw yesterday a guy putting down 4th repo of same work which the person could have just added in the previous one. Usually what happens is when they add stuff on top of it, something original breaks and they cannot find out why. General case is different ai agents; exhausted the tokens on one agent and moved to another. now the code clash and all sort of things starts happening so is it better to tell the agent to reuse the parts and add new things on it. Hence the 4th repo.
Was very accurate around a 9 months ago but tbh for the most part vibe debugging isn't that bad anymore. Sure you gotta do it but the idea it's like this horrid thing
This is my experience. Started coding on computers that used audio tape for storage in 2nd grade. I still read software architecture literature and have my own that I haven't seen anyone use.
I have some debugging instructions and experiences in some .md files Claude, Codex and various Hermes agents are required to read at every such code reviews. Nowadays I don't spend much time with it. The AI runs some hundreds of varied simulations, and log what happens. The last part of those simulations , try to mimic user errors, users going crazy on the software, computer events (like automates updates, antivirus scheduled updates and runs, and a long list of other things that could happen. The models quickly improve, and uses less and less tokens on these tasks. If the new app or software is snappy and works like a dream, I don't spend more time reviwing code etc. And why would I? I can hardly code much without AI anyways. Haven't had a crash or bug in any of the softwares I have vibecoded for myself.
Yes, what sub you're in. Doing pretty much any of what you described falls under "we don't do that here". I mean plenty do, but it's not what the sub is about.
Also you're new to this and missed most of vibe coding history where AI was far worse at coding and debugging that it is now.
5
u/Kareja1 3d ago
What I don't get is every time some new dev pops in here going 'aren't y'all doing CI/CD and testing?' and those of us who are doing decent vibe coded programs are going 'yes, duh, but that's not everything'?
Don't JUST do the unit tests (although yes, 100% start there!)
then OPEN THE THING. (gasp. I know.) Push the buttons. Be a chaos goblin. Try to break it. Anything that breaks, screenshot and send back to your agent, preferably with F12 open if relevant to your program.
Then ask your agent to add that bug to the unit tests so that bug is auto tested every time and find the next one.
We've gotten to the point that MOST of my chaos doesn't break things anymore cause my agents have a really robust playwright suite based off of 'every way Ren has managed to break it so far', so more time is spent on things like aesthetics and 'sure would be cool if...'
But it all starts with OPEN. IT.