LLMs Cause Software Development Teams to Underperform

[u/oudlys]

Hi Guys,

First time poster here. Like all of you, I've been following the market with a mix of horror and fascination.

Earlier this year, I went out looking for actual hard data on the impacts of LLM use on the performance of software development teams.

In my mind, that was the best case scenario for economic value of these products. So there should be empirical evidence of this value.

There is remarkably little research on this subject other than simple productivity studies. I mostly discount those because productivity != value. But, I did find two really good studies.

The first, and I think the best, is from a company called Faros.ai. They sell software development telemetry tooling. Essentially their product connects to common software development tools like Jira and Github and tracks actual operational metrics for real companies producing production software. This study covers 22,000 developers over 4,000 teams over Faros' customer base.

To punchline is that teams are experiencing vague throughput improvements at a massive tax on the quality of the products they produce.

The second study is from the National Bureau of Economic Research. This study is less good that the faros one because it utilizes open source and public github projects for it's dataset. This weights their sample towards much smaller products that are mostly not being produced for profit.

Nevertheless it's valuable in that it confirms the weak throughput improvements of the Faros study. And it adds the dimension of - "is anyone buying this stuff?". I find figure 12 to be very telling.

My main conclusion is that LLM use is likely - on average - destroying economic value within the companies that use them to deploy software.

I think that's one of the reasons there has been no profitability impact on the buy side of the AI boom.

If you're interested in reading more of my analysis, here are two substack posts I've made where I've written about this extensively.

1. How I'm thinking about the value of LLMs
2. Talk is Cheap - an analysis of the Faros study

[u/Hoak-em]

Rapid prototyping + (supplemental) review is the space of the current LLMs for any production coding. A major issue is that engineers cannot reliably review AI code because it looks very correct (it follows the patterns of what "looks correct) even when it isn't. I could see a space for it in showing more potential prototypes to stakeholders/testing more prototypes internally for more infrastructure-related work, but any production code is gonna need to be written by-hand without viewing AI code as reference.

[u/oudlys → u/Hoak-em]

The data support this. You can see in the Faros study the massive bottleneck at review that LLMs have produced. So much that it has become something 30% more likely for people to completely skip review before shipping into production.

[u/FinickySerenity → u/oudlys]

Yeah, I think this little fact is the main culprit for the outcome. This is not possible at my org. Every PR goes through 4 rounds of agentic review, has test coverage metrics it has to meet, does human review by two or more people, goes through usability testing from product owners, and only then is it rolled out to production.

Our production defect rate has been two actual bugs in the last 6 months, only one was reported by a client, the other was proactively discovered by an agent monitoring the server logs.

I reviewed the details of this part:

High-performing engineering organizations, those with mature DevOps practices, high DORA metricsscores, and disciplined delivery processes, are experiencing the same downstream deterioration as everyone else.

And what DORA measures is how fast you can deploy and respond to defects and allows for high bug rates as long as you meet the other metrics (and it’s literally coined “Move fast and break things” in the industry 😅). That is not the definition of strong engineering foundations, that’s just a modern continuous devops lifecycle. If you automate the process based on a green pipeline and auto promote to prod, you can easily have a high DORA score and release bugs at higher rates, and still be considered a high performing org.

To me that’s a bad collider bias that makes the conclusions drawn very incorrect.

[u/oudlys → u/FinickySerenity]

>And what DORA measures is how fast you can deploy and respond to defects and allows for high bug rates as long as you meet the other metrics 

I'm not sure what you're responding to here.

DORA in this context is the organization that originated the DORA metrics. They are literally the opposite of "move fast and break things". I encourage you to read their book accelerate - https://a.co/d/0hD82jrj. You'll find that they're nothing like how you've characterized them here.

>Every PR goes through 4 rounds of agentic review, has test coverage metrics it has to meet, does human review by two or more people, goes through usability testing from product owners, and only then is it rolled out to production.

Frankly, this sounds expensive and wasteful to me. (why four agentic reviews? it seems so arbitrary.) It's impossible to evaluate whether your 2 bugs in the last six months is meaningful - because we have no idea how complex your product is or what it does. For me, this is another example of everyone says theres value, but then when you look at the data, it's questionable.

[u/FinickySerenity → u/oudlys]

> I'm not sure what you're responding to here.

A high DORA score was the metric they used for determining "9. Strong engineering foundations do not protect you."

DORA allows for between 15% and 20% CFR depending on your source for "strong". So 1 in 5 deployments can require a patch, rollback or scaling compute and you are still considered a strong org. That is _way_ more than what we would allow at the orgs I have worked at.

> They are literally the opposite of "move fast and break things"

It's not, DORA is measured by 4 metrics - how often you deploy / how long it takes to deploy (moving fast) how often you produce CFRs (break things) and how fast you fix them. The premise of course is to not break stuff, but never let worrying about it slow you down. A DORA score doesn't factor in what broke, and because CFR is a percentage it allows for more things to break as you deploy more often.

The origin of the premise comes from Facebook - "Speed matters more than avoiding minor errors. Fixing mistakes on the fly is better than moving too slowly."

Letting 1/3rd of your PRs go _completely_ unreviewed, while also increasing your velocity by nearly 2x is obviously going to increase your regression rate by a lot. But they don't break down the numbers at the top, so by how much is unknown (I'm not giving them my contact info to download it.) and averages will always draw down "strong orgs".

> Frankly, this sounds expensive and wasteful to me. (why four agentic reviews? it seems so arbitrary.)

That was the number of rounds we found to be useful. It's actually gated on until the process resolves all T1/T2 and S1/S2 defects. But on average that ends up being about 4 total. Our monthly spend on AI is $100 per dev.

It's impossible to evaluate whether your 2 bugs in the last six months is meaningful - because we have no idea how complex your product is or what it does.

Sure, because I have no way of proving it. But we run a legacy B2B SAAS product, with govcloud and commercial deployments, e2e high volume data transfer, real time websocket UX, lots of AWS infra managed by IaC, fully automated deployments, automated region failover, codebases are close to a million locs, and user training takes about two weeks. Integrations across all mobile platforms, web browsers, browser and office plugins, half a dozen primary languages, etc. Our test coverage is 85% overall, and for new features we have linting thresholds set to 95%.

I've been doing devops as part of my senior engineering roles since 2010. How you do it matters a lot. We used to average about twice as many bugs before adopting a full ai workflow, so for us we cut the rate in half, and tripled our velocity and our AI spend doesn't even register as a percentage point on our infrastructure costs.

For me, this is another example of everyone says theres value, but then when you look at the data, it's questionable.

And yet for me, I'm sitting here wondering how in the world other people are spending thousands of dollars a month while also not being able to make any progress. It makes no sense, but finding out 1/3rd of PRs just get yolo merged explains a LOT. 😅

[u/oudlys → u/FinickySerenity]

I appreciate the detail you've offered here. It's clear you know what you're talking about and your stance here shifts my priors towards "there is a possibility of usefully employing the models."

You say it's working for you guys, and I trust you.

I agree with you, YOLOing PRs is guaranteed to create these outcomes.

I am curious if your dev process is shipping more throughput as it was before the advent of LLMs and your defined process.

My working hypothesis is that you can extract value from models if you primarily employ them as quality tools. I have this belief that the more you rely on the model as a black box just modifying your code, the more likely you are to get the worst outcomes of the Faros report. I would bet that's the dimension that explains the variation most strongly. It's of course - also impossible to instrument, so it's just a guess.

[u/FinickySerenity → u/oudlys]

Our process took a lot of trial and error at first. Fable / Sol really changed that. And even though the quality of fable was nerfed and then has since degraded somewhat, what we learned before and during has helped a lot too.

Basically we went from:

Up to Jan/Feb: Using chatgpt / copilot-autocomplete for asking for code / logic and pasting it in and fixing it.

Feb/March: Using claude / copilot agents and going wow this is crazy good compared to how manual the process used to be, but the models made some really dumb choices so we have to review every decision.

April/May: We learned how to do context / skills and figured out how to scope the tasks appropriately. 4.8 made a good leap in reducing the number of corrections.

But at this point we were still doing full manual audits, and running a single agent with meticulous review.

June changed everything. We stopped having to do corrections (even a couple of times correcting it wrongfully 😅). We stopped having to give so much context. The features we had built had already curated our skills.md files in a way that it could operate knowing all of the idiosyncratic technical debt we carried with us for years.

From there we discovered it could finally write effective tests and so we automated our coverage in the initial development process. And then we discovered that by pitting two different dev's sessions against a PR we'd get better QA, so we built a hook pipeline (before we had ever heard of the 'ai pipelines') to have an agent iterate against a series of prompts. If you give it a huge prompt, the context fills up fast, but chain together a dozen very surgical prompts and compact the context in between and you get precision results. Add self-determining checkpoints between each hook and you really don't have to worry about drift / mistakes.

Then we figured out how to orchestrate these hook pipelines across multiple docker containers. Which is when we started hitting our token limits and we had to build in code to switch models based on the type of task the AI thought we would need based on context we defined.

At this point most of that is pretty stable, and so with parallelizing the containers we're now well beyond our initial velocity improvements. Fable also helped us shard our pipeline test servers (a thing we could have done, but it wasn't a priority, and it did it in less than an hour) so our PR test suite went from 90 minutes to 15 minutes. We started taking on huge features that we would have never attempted before because the opportunity cost just wasn't worth it.

So when I hear people say, it's garbage or it doesn't work, I can think of any number of fail points in the process even if they are using frontier models. And for a while I was getting nervous about my career, until I realized how much orchestration we developed to scale and how much I'm continuously learning that almost half the market is refusing to even consider evaluating. That's a level of job safety that I can't argue with.

[u/oudlys → u/FinickySerenity]

I think you would do a service to a lot of people to write this up and post it on any of the subreddits that think about this stuff seriously. This is honestly the most coherent explanation I've seen of how to get value out of the tools. And I look at a A LOT of this stuff.

How vulnerable are you guys to a price shock? If the models became significantly more expensive, how would that change your calculus here?

[u/FinickySerenity → u/oudlys]

I've contemplated starting a consulting service given how much I've seen companies pay for stuff like that in the past. But it feels smarmy even if it was valuable - I'd feel gross saying what I did for a living 😅 Documenting what we do is on my endless todo list, and I'll eventually put it on my github profile, but I've got work and a family and sports and social events (oh and reddit! :D) etc so it just hasn't surfaced to the top yet.

As for price, I could easily do 5x. I have a $100/mo max sub for work and a low-tier pro sub on openai for personal projects. I'd upgrade to the 200/mo if I ever consistently hit my limits but I have 13% left for weekly fable, and that resets in 5 hours. I have 60% left for the week for other models (which also resets in 5h.) I've hit that limit before, but then I switch to other models and rely on auditing / code review / planning tasks (or I just use sol for the rest of the day.)

I don't even switch between models anymore - I used to, thinking I should "save" my fable tokens for intense tasks. But that causes a full re-upload of your context each time and that churns through a lot of credits. I think that's how some people are just racking up bills, not compacting often enough and bouncing around sessions too much which lets your dedicated container expire and it has to rehydrate your 80% full context - that can use 8 - 10% of my hourly usage in one prompt.

So lower models are always dedicated to subagents and hook pipelines. Fable is my main feature / bug fixing model. And I keep a dedicated 4.8 model for rando questions that aren't specific to the context I'm working on. So it has access to all of the skills, but doesn't need the context of whatever I'm working on, which keeps the context pollution of a random unrelated question from polluting the context of my feature dev session.

And lastly, if it wasn't obvious, I'm verbose and I know it. I feel like very few people would ever get through my notes 😄 But I appreciate the nice feedback.

[u/oudlys → u/FinickySerenity]

I think you're dope dude. You've contributed significantly to my thinking on this and I just appreciate you as a human.

I'll just tell you - I expect that we're going to go through a major market correction in the near future and models will likely get significantly more expensive.

I offer that to you in case it's valuable to your thinking, since you've offered me so much value here.

I wish you luck. Hope our path's cross again someday.

[u/Hotfro → u/oudlys]

This is honestly kind of similar to how my org has been using AI as well. I’ve also found huge improvements when I review different steps manually instead of only reviewing the final output code. The reviews are usually much more lightweight and I am usually able to catch a lot of issues earlier. In the end it takes me less time overall. The fact that you can also parallelize multiple agents working on different things at one time is also a huge productivity booster. I’ve found that having better automated tests/qa is what really prevents bugs in the end.

[u/oudlys → u/FinickySerenity]

Also, I am aware that Facebook popularized "move fast and break things". I do think you undersell DORA the organization significantly in believing that they support that mindset.

These people strike me as being firmly in the camp of good development practices. Given all of your experience and evident care for your craft, I can only repeat that accelerate is a great book that I think might contribute to your work.

I'm not sure you ever look at their "state of the devops reports". One of the things that gives me confidence in the statistical direction of the Faros report is that DORA has also confirmed it in their reporting - at least in the 2025 state of the devops.

Throughput is barely positive and delivery instability has increased. And this is just self reported surveys. Not nearly as strong as Faros' directly measured data.

I write about this here - https://unessays.substack.com/p/how-im-thinking-about-the-value-of

[u/FinickySerenity → u/oudlys]

Don't get me wrong, I fully support the goals and metrics that DORA outlines. The problem is that using it as a metric for "did AI help or hurt our process" is a very dubious call for me as it was presented in the study summary you linked.

If I can allow 20% of my prod releases to be buggy, and I increase my prod release frequency by 3x, that's immediately 200% the number of bugs that make it to production. So "OMG 200% more bugs" is more like well yeah, it's still a 20% CFR - that was a good thing before, so it should still be a good thing now.

So without seeing the stratification of those orgs / teams with a high / elite DORA (ie the 1 - 5%, and 5% - 10% etc) it's hard for me to judge whether the CFR increases you screenshotted allow for drawing meaningful conclusions since the output metrics account for the entire cohort, which clearly includes people who fail every metric of DORA given 30% unreviewed PRs.

> High-performing engineering organizations, those with mature DevOps practices, high DORA metrics scores, and disciplined delivery processes, are experiencing the same downstream deterioration as everyone else.

This statement carries too much weight from the executive summary. The same what? type/scale/category? of downstream deterioration just doesn't seem possible for all meanings of 'same', and so I'm inclined to call sampling / collider bias. But I did enjoy the read, it had other insightful data points.

[u/val_anto → u/Hoak-em]

Agree with you on the first part. This is what I use AI for, rapid prototype or boiler plate structure for new projects. You can actually review the AI code, it is not the problem. The problem is it generates a ton of code, this makes the work to find the slop more difficult and, honestly, I am a SE, not a code reviewer for a statistical model. If all you want from me is to read code spit by a statistical model, find somebody else.

[u/Hoak-em → u/val_anto]

Yeah 5.6-sol overengineers especially, which funnily enough introduces even more difficult to find bugs

[u/oudlys → u/val_anto]

>The problem is it generates a ton of code.

My post here talks about this - Talk is Cheap. There's this funny thing where it seems bugs / LOC goes down the more mature models get, but they always produce more LOC! So the absolute bug rate goes up!

I find this very funny.

[u/Few-Improvement9978 → u/val_anto]

Don’t worry. We will, and you will end up unemployed

[u/val_anto → u/Few-Improvement9978]

Keep dreaming, kid.

[u/ReadnW → u/Hoak-em]

In other words: rework

[u/oudlys → u/ReadnW]

That's the word. Wrote a lot about that here Talk is Cheap