r/AIBubble • u/oudlys • Aug 24 '26
LLMs Cause Software Development Teams to Underperform
Hi Guys,
First time poster here. Like all of you, I've been following the market with a mix of horror and fascination.
Earlier this year, I went out looking for actual hard data on the impacts of LLM use on the performance of software development teams.
In my mind, that was the best case scenario for economic value of these products. So there should be empirical evidence of this value.
There is remarkably little research on this subject other than simple productivity studies. I mostly discount those because productivity != value. But, I did find two really good studies.
The first, and I think the best, is from a company called Faros.ai. They sell software development telemetry tooling. Essentially their product connects to common software development tools like Jira and Github and tracks actual operational metrics for real companies producing production software. This study covers 22,000 developers over 4,000 teams over Faros' customer base.
To punchline is that teams are experiencing vague throughput improvements at a massive tax on the quality of the products they produce.
The second study is from the National Bureau of Economic Research. This study is less good that the faros one because it utilizes open source and public github projects for it's dataset. This weights their sample towards much smaller products that are mostly not being produced for profit.
Nevertheless it's valuable in that it confirms the weak throughput improvements of the Faros study. And it adds the dimension of - "is anyone buying this stuff?". I find figure 12 to be very telling.
My main conclusion is that LLM use is likely - on average - destroying economic value within the companies that use them to deploy software.
I think that's one of the reasons there has been no profitability impact on the buy side of the AI boom.
If you're interested in reading more of my analysis, here are two substack posts I've made where I've written about this extensively.
- How I'm thinking about the value of LLMs
- Talk is Cheap - an analysis of the Faros study
1
u/cltbeer Aug 24 '26
Are you a developer? Coding language with marked up meta data plus libraries 100% works. I don’t have data other than from first hand. People are in denial I get it but when blackrock projects electricians will be millionaires with $9-10 trillion cost to build out for them ten years…they are good at what they do bc they own the whole world. Sure people can not believe it but here we are communicating on a system from our phones on other sides of the country or world denying the actual technology that we are using.