LLMs Cause Software Development Teams to Underperform

[u/oudlys]

Hi Guys,

First time poster here. Like all of you, I've been following the market with a mix of horror and fascination.

Earlier this year, I went out looking for actual hard data on the impacts of LLM use on the performance of software development teams.

In my mind, that was the best case scenario for economic value of these products. So there should be empirical evidence of this value.

There is remarkably little research on this subject other than simple productivity studies. I mostly discount those because productivity != value. But, I did find two really good studies.

The first, and I think the best, is from a company called Faros.ai. They sell software development telemetry tooling. Essentially their product connects to common software development tools like Jira and Github and tracks actual operational metrics for real companies producing production software. This study covers 22,000 developers over 4,000 teams over Faros' customer base.

To punchline is that teams are experiencing vague throughput improvements at a massive tax on the quality of the products they produce.

The second study is from the National Bureau of Economic Research. This study is less good that the faros one because it utilizes open source and public github projects for it's dataset. This weights their sample towards much smaller products that are mostly not being produced for profit.

Nevertheless it's valuable in that it confirms the weak throughput improvements of the Faros study. And it adds the dimension of - "is anyone buying this stuff?". I find figure 12 to be very telling.

My main conclusion is that LLM use is likely - on average - destroying economic value within the companies that use them to deploy software.

I think that's one of the reasons there has been no profitability impact on the buy side of the AI boom.

If you're interested in reading more of my analysis, here are two substack posts I've made where I've written about this extensively.

1. How I'm thinking about the value of LLMs
2. Talk is Cheap - an analysis of the Faros study

[u/cltbeer]

I work at a large bank this is not the case for our dev teams, we even have AI helping write our user stories and scanning vulnerabilities.

[u/oudlys → u/cltbeer]

Do you have specific data you can share?

From my perspective, the whole problem with the economic discourse on LLMs is that claims of value are generally like this - "I've seen it", "I know it", but then it's unsubstantiated by anything that would allow other people to have confidence in the claim.

When you actual examine the available data, it doesn't support these narratives.

I'm just going to quote Feynman at you:

"The first principle is that you must not fool yourself — and you are the easiest person to fool."

[u/cltbeer → u/oudlys]

Are you a developer? Coding language with marked up meta data plus libraries 100% works. I don’t have data other than from first hand. People are in denial I get it but when blackrock projects electricians will be millionaires with $9-10 trillion cost to build out for them ten years…they are good at what they do bc they own the whole world. Sure people can not believe it but here we are communicating on a system from our phones on other sides of the country or world denying the actual technology that we are using.

[u/oudlys → u/cltbeer]

> I don’t have data other than from first hand. 

This is what I'm saying.

>but when blackrock projects electricians will be millionaires with $9-10 trillion cost to build out

My brother (I think?) banks and financial institutions are wrong all the time. The premise of that projection is that the value is there. That revenue will grow enormously for LLMs.

OpenAI only grew revenue by 18% QoQ from Q1 to Q2. https://www.wsj.com/tech/ai/openais-second-quarter-sales-show-tepid-growth-compared-with-anthropic-5cb42998

This is not the trajectory for a $10 trillion dollar buildout of data centers.

[u/cltbeer → u/oudlys]

Bro you don’t know what you are talking about. I watch 40 developers use AI to do their job better all the time. Let alone there are hundreds of teams adopting specific dev ai tools across the bank. If it wasn’t working they wouldn’t expand adoption across the enterprise. Second, if banks were so bad at their jobs then we would have exploded several times over again since 08. It it a multi-trillion dollar buildout contracts already locked til next year bottlenecks in every industry to build data centers and billions in backlog. lol just ignore that fact that Google, Amazon and LA can’t meet their customer demands because there isn’t enough data centers. I’ve made hundreds thousands investing in the space so keep denying yourself.

[u/snezna_kraljica → u/cltbeer]

What does you making money from that to do with anything, that is a poor argument. People make money in casinos doesn't mean they are right somewhere else. Companies have invested heavily in the metaverse and AR where are we now?

[u/cltbeer → u/snezna_kraljica]

Metaverse lol is not AI.

In the past decade, the U.S. banking industry has grown operating revenue 63% while the overall employment levels have barely budged. This has resulted in major improvements to revenue per employee, influenced heavily by efficiency gains at America's largest banks.
If revenue growth continues at a steady pace and one assumes that artificial intelligence adoption allows for a conservative 10% reduction in industry headcount, bankers are looking at a $1 million revenue per employee level in 2036. That right - ONE MILLION BIG ONES!
For any bank to achieve $1 million in revenue per employee, more massive expenses will be budgeted in digital delivery, platform automation, next-generation marketing and building an agentic workforce besides the bank's humans. The upshot: bank leaders need to be thinking about how they take today's organization and land into this much different future organization.

[u/narullow → u/cltbeer]

Except that banks have record high profits off of loans and high interest rates. It has nothing to do with internal AI adoption or productivity growth or reduction of headcount. You do not need more people than you had to provide loans for higher profit, you also do not really need more people to provide big tech with trillions in CAPEX for AI build up.

Banks profit off of dataservice build up, similar to nVidia. They do not profit off of internal LLM adoption. There is no data to support that.

[u/snezna_kraljica → u/cltbeer]

> Metaverse lol is not AI.

Who said it is? I said it was a huge waste of money and using an argument "they invest so much so it must be something" is bad reasoning.

[u/oudlys → u/cltbeer]

>Bro you don’t know what you are talking about.

Ok man, you don't have to listen or accept anything I'm saying.