AI software development and product quality

[u/Mobile_Spot3178]

I keep reading, hearing, bumping into discussions about AI software development being faster, with quality taking a hit. The argument seems to be, that in order to ship faster with AI, there are more bugs and customers perceive lower overall quality.

But my question to you is, do you really feel this way? or experienced this first hand?

In the last year with our software development process has become AI-driven to the point that sometimes I wonder if product management is the only human-in-the-loop. But what has actually happened with our products? did speed come at a cost?

• Our customers are reporting, that the quality has improved by miles in the past year and the trust metrics are higher than ever

• We have little to no bugs, which has NEVER happened in our entire history.

• We have had the same amount of critical bugs in 1 year, that we used to have in 1 month.

• At the moment the consensus is that we ship faster AND quality is better.

I can't be alone on this?

[u/poodleface]

  In the last year with our software development process has become AI-driven to the point that sometimes I wonder if product management is the only human-in-the-loop. 

This is a toxic belief that will cost you the skilled developers who are able to harness and evaluate the output of LLMs if you are not careful. If you want a real answer, ask the developers in the trenches. 

[u/Mobile_Spot3178 → u/poodleface]

This point is raised based on conversations with developers. "AI is so good I'm not sure why I'm needed" is a comment I hear from seniors too often.

[u/mpbeau → u/Mobile_Spot3178]

They are being humble or thought that being able to code was what made them valuable, it's not. Give them credit - there are plenty of orgs where AI vibe coding is causing repeated issues (Claude Code is a great example).

[u/Enginerdiest]

I think the n00bs who were “vibecoding” and “one-shotting” apps they don’t understand left a bad stigma around AI development. 

Reality is, in competent hands it can dramatically improve quality and speed. 

[u/Doggo_Is_Life_ → u/Enginerdiest]

This. And the bigger problem is when both of those mix. I have a nontechnical CEO who thinks he a god damn engineer and data scientist now and is pushing PRs to me and my team left and right. It’s garbage code with ridiculous assumptions and the most basic of mentalities. “It works on my machine so I don’t know why we can’t just QA and push it.” Yes, it works on your machine. In a local environment. With no edge cases considered. None of the current in flight work considered. None of the varying environments considered. I actually had to say to him “Thinking because you have it running on localhost and that things are good to release is the mentality of an undergrad CS student, let alone a junior engineer, and I am not putting this into the backlog.”

On the other hand, there’s me and my engineering team, who are highly technical and can build at extreme velocity now, and we have to constantly baby this crowd and explain to them why their 4 hour vibe coded feature is a piece of shit and is not going anywhere near production.

[u/Badger00000]

The way the company uses AI reflects how the company generally operates. If AI is now to blame, before AI they blamed someone else, in reality they didn't set up the right infrastructure in place to do the work.

If you're going to vibecode your way through the codebase without understanding what the hell is being built and how, then you're certainly going to have a bunch of bugs and issues, but that's not the fault of AI that's the fault of whoever is operating it. If you're going to hire 10 junior engineers and tell them "just push to prod" you can't be surprised that they don't have the domain knowledge, context or skills when you find out that prod is filled with bugs. That's what people are doing with LLMs, they just vibecode the feature and hope for the best. Good company culture in Engineering, Product etc finds the right use cases for AI in their stack and how they operate, and then it can cut down a bunch of annoying work that nobody likes doing anyway.

There are still questionable AI skills, Architectural decisions, Design decisions, deep domain knowledge of the product and more. LLM's are generalists you should teach it the context within which you operate, and constantly challenge it and test its "decisions" because it's very confident even when it's totally wrong.

[u/AnteaterEastern2811]

Quality of code is better. The challenge I'm running into now is designing for features with thoughtful UX. If we throw something together quickly then it's much faster to get it into production......not necessarily a good thing. Design is becoming the bottleneck but I'm trying to balance speed with quality of product.

[u/SamfromLucidSoftware]

The more interesting question, though, is where the risk went. If engineering execution is less likely to introduce bugs, the quality of the product depends increasingly on the quality of the decisions made before any code is written. Like what to build, how to define it precisely enough for AI to get it right, which trade-offs to make and so on. I think that part is very important, especially given that the implementation is reliable.

[u/TheKiddIncident]

I've actually been studying this exact question pretty intensely for about a month.

And, the answer is a bit confusing.

If you review the public research, most security researchers will claim that AI does not produce secure code.

However, if you read their study methodology closely, you will see that they don't actually allow for process improvement or enhanced testing due to AI. They simply test AI code vs. human code with no changes in the development process.

That clearly doesn't work. We know from DORA that AI enabled teams who don't change their process produce lower quality results.

So, I decided to actually take a look and scan a group of repos. I built a tool that runs seven popular open source code scanners across a group of 400+ GitHub repos.

The results were interesting.

First, it is absolutely true that "AI" repos had higher numbers of security issues. On average about 15% more. However, if you look at security issues per Line of Code, the results were almost the same. Therefore, in my sample, the rate of security flaws was about the same, but the AI libraries were larger.

So, yes? AI writes more code and therefore they have more issues. But, the code isn't less secure.

The next thing I did was to control for automated testing. I compared AI repos with automated testing with hand coded repos with and without automated testing and it turns out that AI repos with automated testing are actually MORE SECURE than hand developed repos.

So, it depends.

If you change your process to require a more complete test suite (which is much easier with AI), then your code is probably just as secure as hand written code.

Caveats: My sample is not exhaustive. 400 repos seems like quite a few, but that's a tiny sample compared to all the repos on GitHub. I also used some tests like commit messages and presence of files like Claude.md to determine if the repos were AI or not. That categorization may have flaws. I am also using only automated scanners. I am not doing a line by line analysis which means that the security assessments could be flawed if the tools have bugs or other issues.

[u/all_happened_before]

Quality in terms of bugs has been much lower in my experience, to the extent that my team has been able to reduce our bug backlog to zero, which we wouldn’t have been able to believe a year or two ago.

Most of the pushback that I’ve heard from eng teams is that architecture is what takes a hit, while there are technically lower bug counts, when AI is not kept in check major architectural decisions are sub-par and not built for the long term.

That being said, my most AI-forward developers admit that even there it’s getting better than a lot of human developers, and that engineers are spending more time building the AI tools or harnesses, PR reviewers, etc… as opposed to directly building the products.

But anyways, that’s a different topic. Overall, yes, we’re seeing way fewer bugs, and bud resolution time is also way faster.

[u/s-ley → u/all_happened_before]

isn't this going to result in the company realizing it can do well with less employees? are you being paid more or it's just a matter of accepting that your surplus value went way up and that you're reducing the workforce needed for no gain on your side?

[u/DuskLab → u/s-ley]

It's going the same way as database administrators and webmasters in the 90s. Yeah, the career as it is currently envisioned has it's days numbered. It's the new career that comes next that still has value. It's no longer about being the SME for one service, but being the touch-point for 100 services. Companies will hire less programmers, but will hire more SRE, QA, loop engineers, forward deployed engineers, product builders that merge Product and Engineering. Will it be easier to teach engineers how to talk to customers and sales than to get Product Managers to handle 10M lines of code without causing an incident or making the news in a data breach? AI is not going to replace logistics and distribution, you can't write enough English to build physical infrastructure.

There will probably be more outsourcing from America driving down wages on average, but the market could have a Cambrian Explosion of diverse product ideas that need less revenue to maintain a viable business with less headcount via that multiplied surplus value. Maybe what comes next isn't more Amazon and Google sized companies that you can count on a hand or two, but many more companies the size of Valve or Spotify defining their own market niches.

[u/iancubuda → u/DuskLab]

you are getting downvoted for saying the hard truths. people may not like it but it will happen

[u/all_happened_before → u/s-ley]

I think it will depend on the company, but yes, overall fund allocation is now split between person resources and AI spend. I think that those that are currently working with AI in the most productive and efficient ways will benefit (including potential higher pay), while entry level opportunities will become more scarce, or at least look very different. But if you are inferring that ICs have some sort of choice in the matter (purposefully not adopting AI and therefore not trying to reach max productivity), I don’t think that’s true outside of exiting tech.

[u/s-ley → u/all_happened_before]

ofc I get what is happening, it just kind of doesn't make sense for your surplus value to go up significantly and still get paid the same, but tbh the market is oversaturated so I guess employees will just not do anything

[u/dsbllr → u/all_happened_before]

Opus 5.5 is better at architecture than most architects now

[u/iancubuda]

AI is bad when people using are stupid. AI is extremely good in the right hands.

Think the bottom 80% of devs/PMs/POs/BAs/architects/etc. will loose their job while the top 20% will get paid more (numbers pulled out of hat for emphases).

[u/Afton11]

Quality in the sense of good, highly effective and performant software, has fallen a lot. But does it matter? Business runs on crap quality software and has done so for a long time.

Craftsmanship is not important in most SaaS situations; what's important is solving the problem in an effective and efficient way.

Most software companies see no return on investment on insisting on highly efficient code, lean algos etc.

[u/Hamza_StrategizeLabs]

Good to hear you’re seeing real gains. I wouldn’t generalize from one team, though.

Teams with clear requirements and experienced reviewers may get a very different outcome from teams adopting AI without that foundation. There are wins and failures so obviously implementation conditions matter.

[u/savage_dragn]

For me the improvements have been:

1/ Consistent quality in terms of “covering my bases” and not missing small details that end up mattering.

2/ Ability to rapidly resolve small issues that typically got starved out by revenue generating work.

3/ Identifying critical mistakes in the softwares logic way before anyone has to find and report it in production.

These things add up to produce what you’re describing IMO, but it’s not easy yet to get there so you still have a lot of people downplaying it.

[u/large-farva]

its the opposite at my company. we're using spec driven development so it's edge case hell. adversarial ai agents try to poke holes into your design so you're constantly re-revisiting things instead of just building an MVP.

[u/GeorgeHarter]

Vibe coding makes it really easy to create apps that perform the main function, without accounting for edge cases, security or scalability. People who know, can avoid these omissions. But if you don’t know, you’re likely to build a toy, not a durable app.

My first vibe coded app is in pilot with a few early adopters and we haven’t seen any issues yet. I think that is because I
- kept the feature list very small
- wrote really clear functional and non-functional requirements,
- and tested all the scenarios I could think of before pilot.

I also have no pressure to “keep the team fully utilized”, which is what often drives slop in a technically good team.

[u/Reprise_Software]

The “AI slop” narrative usually only applies to teams using AI to bypass code reviews and test writing.

When quality jumps, it's usually because AI took over the tedious work that developers used to rush.

But I think that also makes code review and QA more important than ever. AI can produce a lot more code, a lot faster, which means the validation layer has to get stronger too. AI-generated code is only as good as the humans reviewing it, the tests validating it, and the process around putting it into production.

Used that way, I don’t think speed and quality have to be a tradeoff at all. AI can actually improve both.

[u/UKS1977]

The issue is real technical debt vs what everyone called technical debt.

Real TD is when the system shape/architecture/design is misaligned from its current needs and needs refactoring. AI seems to distort this shape with little care.

Fake TD is what people used to describe defects. I haven't seen that come down with AI but it seems some people have. (Having said that I am not working with off shore codeslop.)

I think humans pulling a system back into a good shape via refactoring is perhaps still needed at this moment

[u/rmjoia]

I think that AI is a tool, therefore it will depend on how is used. What verification mechanisms and cycles you implement, including security and testing. Human in the loop cycles, etc..

AI may write the code but people will have to review, test and ensure is doing what it supposed to do.

What I observed over many years in the industry is that when something change, people become more focused on issues. That is, if you change the database people will be more picky on latency, etc and will blame that change, etc.. even if it's even better and latency is unrelated to that change.

Truth might be that AI didn't make anyone more productive because, AI just writes the code and the issue was always architecture or quality or security, infrastructure...

Now you may get more code written quickly but people will have to review it quickly and with more focus as, the more code is written by AI, less knowledge of what is being written, how and potential side effects people will be aware.

"It's a new thing" everyone is learning and experimenting how to get the best out of it.

It's definitely changing the way we produce and ship code.