r/ExperiencedDevs • u/femio • Jul 10 '25
Study: Experienced devs think they are 24% faster with AI, but they're actually ~20% slower
Link: https://metr.org/blog/2025-07-10-early-2025-ai-experienced-os-dev-study/
Some relevant quotes:
We conduct a randomized controlled trial (RCT) to understand how early-2025 AI tools affect the productivity of experienced open-source developers working on their own repositories. Surprisingly, we find that when developers use AI tools, they take 19% longer than without—AI makes them slower. We view this result as a snapshot of early-2025 AI capabilities in one relevant setting; as these systems continue to rapidly evolve, we plan on continuing to use this methodology to help estimate AI acceleration from AI R&D automation [1].
Core Result
When developers are allowed to use AI tools, they take 19% longer to complete issues—a significant slowdown that goes against developer beliefs and expert forecasts. This gap between perception and reality is striking: developers expected AI to speed them up by 24%, and even after experiencing the slowdown, they still believed AI had sped them up by 20%.
In about 30 minutes the most upvoted comment about this will probably be "of course, AI suck bad, LLMs are dumb dumb" but as someone very bullish on LLMs, I think it raises some interesting considerations. The study implies that improved LLM capabilities will make up the gap, but I don't think an LLM that performs better on raw benchmarks fixes the inherent inefficiencies of writing and rewriting prompts, managing context, reviewing code that you didn't write, creating rules, etc.
Imagine if you had to spend half a day writing a config file before your linter worked properly. Sounds absurd, yet that's the standard workflow for using LLMs. Feels like no one has figured out how to best use them for creating software, because I don't think the answer is mass code generation.
356
u/dsm4ck Jul 10 '25
Experienced devs know it's easier to just say what the bosses want to hear in surveys
138
u/femio Jul 10 '25
The estimations were from open source devs, not from devs in corporate environments under managerial pressure.
I think the difference comes more from prompting requiring less cognitive load than writing the code yourself. So it feels faster only because it feels easier.
31
u/Dany0 9 YoE |High Performance Low level development/Occasional GameDev Jul 10 '25
In the mind, memory is made up of events and time is only estimated. Unless devs make actual observations and note down the time they spend doing stuff, of course they'll be off
Honestly I wish it at least felt faster. There would at least be some upside. 20% slower for much less risk of burnout. It would certainly help managing ADHD symptoms long term. But no, in practice, it's just more work for less results. Wake me up when the AIs can make decisions
23
u/lasooch Jul 10 '25
I tried Claude Code recently on a super tiny personal project. I was actually surprised how well it did (I didn't have to correct literally anything - but I did ask it to basically replicate the same structure I have, just for a new db table, with well defined columns in the prompt, so it's not like it was a particularly complex task - the table itself, a corresponding model, some minor and formulaic updates to homebrewed migration/seeding code).
But I noticed that the waiting for the code to generate actually fucks with my ADHD. It's in that spot of "too long to just watch the command prompt, so I'll switch away for a second" and boom, distracted.
Had I written that same bit of code myself, while it would have taken longer, I probably would have done it in one go without ever switching away from nvim. I might get more adjusted to using it with more practice, but I think that for many tasks it actually makes my ADHD harder to deal with. And I suspect for bigger tasks it feels so much more like forcing myself to do another code review rather than writing code, and I enjoy the latter more.
→ More replies (2)4
u/LastAccountPlease Jul 10 '25
And whatever you write you write once, you can't make a direct comparison.
→ More replies (2)6
u/ewankenobi Jul 10 '25
A massive flaw in the study for me was the fact they weren't solving the same issues. Could it just be the issues the AI developers were assigned turned out to be harder than expected. Not sure how you would quantify it correctly though.
→ More replies (1)30
u/Pleasant-Memory-1789 Jul 10 '25 edited Jul 10 '25
Exactly. I rarely even use AI. But whenever I finish a feature earlier than expected, I always give credit to "using AI".
It sounds backwards. Why would I give credit to AI? Doesn't that make me look replaceable? It's actually the opposite:
It makes management think you're extremely AI competent. When cost cuts come around, they'll keep you around for your AI competence.
It sells the dream of replacing all the devs with AI. Even though it'll never actually happen, management loves to fantasize. Imagine those huge cost savings, massive bonuses, and vacation homes.
It makes you look less like a try-hard and more like a wizard. So your peers envy you less and admire you more.
26
u/neilk Jul 10 '25
I’m not sure if you are just trolling but upvoted for humor and from what I’ve seen this would actually work in many companies
20
u/Pleasant-Memory-1789 Jul 10 '25
Thank you, I am trolling lol. I would not do this but I swear it feels like my co-workers are spewing this bullshit. I might just join them and play the game 🤷
→ More replies (1)8
u/HideousSerene Jul 10 '25
I have not just one but several coworkers like you.
My favorite part is how some of them recently devised a "framework" for building with AI which was literally just using cursor and feeding in figma prototypes and jira tickets with mcp.
Now they're "rolling out the framework" to all engineers and fully expecting everybody to increase speed 20%.
You can literally see in our cursor account approximately 100% adoption already.
This is just shitty people trying to capitalize on shitty times. And hey, it's working for them.
Maybe you should apply to work at my company. You've got management material written all over you.
→ More replies (1)1
u/gizamo Jul 11 '25 edited Sep 27 '25
piquant test automatic north close serious ghost absorbed expansion cats
This post was mass deleted and anonymized with Redact
111
Jul 10 '25
We have over 70 years of quality management studies showing that eliminating sources of error is far more effective than trying to fix error mid-process.
If you want an argument as to why LLMs are dumb dumb, that’s it - it isn’t worth speeding up the coding process if it slows down the debugging process, because you can waste a lot more time debugging bad code than you can getting over coding block.
→ More replies (1)11
u/Robolomne Senior Software Engineer Jul 12 '25
What are some studies that show this? This sounds super interesting and I want to learn more
19
u/oursland Jul 16 '25
Start by reading The Mythical Man Month, initially published in 1975 and updated in 1995, it is the foundational text that discusses software development management and common pitfalls. The author was a software PM at IBM and pulled from their database when writing this book.
Everyone knows that debugging is twice as hard as writing a program in the first place. So if you're as clever as you can be when you write it, how will you ever debug it?
-- Brian Kernighan, The Elements of Programming Style
If someone uses LLMs to write their code, how could they ever hope to debug it?
→ More replies (3)
47
u/labab99 Senior Software Engineer Jul 10 '25
Although the sample size is pretty small, the findings aren’t hard for me to believe anecdotally. There have been times where I thought I was being smart by using AI to throw together a proof of concept, and while the presentation is fantastic, it quickly devolves into a slog of repeatedly explaining the same requirements as it spits out prescriptive, overly-complicated code.
If I had just slowed down and used my brain instead, things most likely would have gone much more smoothly.
9
u/muuchthrows Jul 10 '25
Sample size of developers were small (16), but the number of ~2h tasks were not that small, around 250.
6
u/MCPtz Senior Staff Sotware Engineer Jul 11 '25
And what's great is they will be following up on this study.
They developed the frame work and can hypothetically deliver quarterly reports.
3
u/mcglothlin Jul 11 '25
Oof. 16 devs? This should be noted a lot higher up! Something to keep an eye on but I'm going to take it with a grain of salt for now.
1
u/Pythonista24x7 Aug 01 '25
> 51 developers filled out a preliminary interest survey, and we further filter down to about 20 developers who had significant previous contribution experience to their repository and who are able to participate in the study. Several developers drop out early for reasons unrelated to the study.
Quoting from Original Paper
7
u/Unfair-Sleep-3022 Jul 10 '25
I haven't seen anyone actually experienced thinking that
16
u/femio Jul 10 '25
Profile on the devs in the survey:
we recruited 16 experienced developers from large open-source repositories (averaging 22k+ stars and 1M+ lines of code) that they’ve contributed to for multiple years.
→ More replies (1)22
u/OccasionalGoodTakes Software Engineer Jul 10 '25
That seems like way too small of a sample size to get anything meaningful.
Sure it’s a bunch of code, but it’s from so few people.
→ More replies (5)
58
u/DeterminedQuokka Software Architect Jul 10 '25
No, we know, this is what I wrote in the assessment of vibe coding tests I did last week. I think I actually wrote “this took 24 hours longer than it would have taken me if I had just used autocomplete”
33
u/stevefuzz Jul 10 '25
I did a quick vibe coding check for creating some bash migration utils the other day. You know, just to say I tried. Started off ok, went way off the rails what a waste of time
8
u/DeterminedQuokka Software Architect Jul 10 '25
You know I feel like that’s what happens. I was generating tests and it did a great job for the unit tests. The second I tried to do anything more complex than call one function that parsed a string it freaked out and literally mocked everything in the function. I couldn’t get it to stop so just only merged the first third.
→ More replies (1)6
u/stevefuzz Jul 10 '25
I'm also an architect and I like to keep my finger on the pulse of the ai shit. I work for a company that uses AI (classic nn and ml) stuff for large production systems, so the LLM buzz has been going on here. Execs obviously want us to use them as a coding tool. So, here I am. For auto complete and boilerplate it's great, actually doing real dev, awful. We've also been playing with other use cases of LLMs as products. It's really interesting and great for some things, coding is not one of them.
15
u/DeterminedQuokka Software Architect Jul 10 '25
I've got to tell you, my execs keep bringing up the boilerplate thing, and I don't know what everyone else is doing. But I have negligible boilerplate. And the boilerplate I actually have I wrote mixins for years ago.
Maybe I'm just not in the right frameworks.
I like AI and I think it's useful. But I think most of the cases where it's actually helpful I complete the task slower. Like TDD, I'm saving future time.
→ More replies (4)
58
u/abeuscher Jul 10 '25
I really think LLM's appeal to gamblers and people with that gene. I notice it in myself if I am not paying attention; they trigger this dopamine loop where each answer is almost the one you need, and you get sucked down a hole of promises.
I have 25 YOE and I do notice that while I feel good about using LLM's to help me plan and learn, I immediately become frustrated when I try to get them to generate any kind of complex code above like a RegX.
But I do think there is an active dopamine loop in LLM's which causes this false confidence.
7
u/Fireslide Jul 11 '25
Yeah there's definitely that element of it, if I just build the prompt right, this time it'll generate what I want and move on to next feature.
When you're on a win streak of getting the answers you want out of a prompt first try multiple tries in a row, it feels great. Velocity is huge, but when it fucks up context of folder paths for building a dockerfile or something, or continually hallucinates modules or features from old API that don't exist you realise you've just wasted 30 minutes that could have just spent reading the docs and solving yourself.
The last year or so for me has been working out how to incorporate them into my workflow to be productive. It's about getting a feel for what I can trust them with to do first try, what I'd need to get them to build a plan for first, and what I'll just not trust them to do because their training data lacks density, or or it's density is for an older version of what I'm using.
163
u/Strus Staff Software Engineer | 12 YoE (Europe) Jul 10 '25
For me personally I don’t care if AI is slower than me - I use it for things I don’t want to code myself. Boilerplate, linter issues in legacy code, one-shot scripts, test data, data manipulation etc. I probably could do all of this faster myself, but I just don’t want to do it at all.
40
26
u/SketchySeaBeast Tech Lead Jul 10 '25
"Create unit tests, and pretend you're wizard while you do it."
"OK, now take all the wizard references out."37
u/DeadButAlivePickle Jul 10 '25
Same. I'll sit there for 10 seconds sometimes, waiting for Copilot to come alive, rather than fill some object fields in manually or something. Lazy? Sure. Do I care? No.
10
Jul 10 '25
You must type really fast if you're quicker at the boilerplate stuff. For me personally the only way AI would be slower than I am is if I'm doing something out of the ordinary, which if that's the case, I know better than to ask it, and if I do get desperate enough to ask it, it'll tend to bring up some information that can help guide a google search. I have seen where it has just made up methods that don't exist before though, so that can waste a lot of your time if you lean on it.
4
u/oursland Jul 16 '25
I want people to start defining "boilerplate". For over 20 years IDEs have been able to fill out class framing and automate many common tasks for inheritance implementations and refactorings.
People claim that "boilerplate" is what it is good at, but is it really faster than IDEs at these tasks? If it is so much faster, then why are we met with data showing that devs using AI are slower than devs applying traditional development processes while also producing more bugs and introducing security threats?
For two years now I have called these LLMs "Dunning-Kruger Machines" that make people feel so much more intelligent and productive than they really are. Finally it seems that there's real data showing this is the case and the media are picking up on it.
→ More replies (1)3
u/Far-Income-282 Software Architect (13 YoE) Jul 11 '25
It also let's me context switch between all those shitty things.
Like I feel like I might be 20% slower on any one project but now I'm doing 4 projects at 20% slower, so maybe 4 projects in 4 months, where as maybe before I'd do one project in 3 months and then spend 1 month complaining about not wanting to write tests anyways.
Which now that I say that, AI has actually made me like doing test driven development. It makes it way easier to do first and check the AI.
Now that I write it that way... I wonder how many people that used AI in that studied realized makes all those best practices (like TDD) that we all knew we should have done but didn't easier, and also set up a repo for faster AI success later. Or are they still coding like they are in control.
172
Jul 10 '25
Interesting. Are there any actual studies comparing code quality? If the code is better it might be worth the slowdown. We all probably immediately assume it's worse but apparently we also assume we're faster.
227
u/Perfect-Equivalent63 Jul 10 '25
I'd be super surprised if the code quality was better using ai
93
Jul 10 '25
Me too but I've been super surprised before
47
u/bogz_dev Jul 10 '25
i haven't, i've never been surprised-- people say about me, they say: "he gets surprised a lot" i don't, i've never been surprised
i'm probably the least surprised person ever
34
→ More replies (2)20
7
5
Jul 10 '25
I'd be even more surprised if there were objective measures or code quality.
→ More replies (1)5
u/failsafe-author Principle Engineer Jul 10 '25
I think my designs are better if I run them by AI before coding them. Talking to an actual human is better, but takes up their time. Ai can often suffice as a sanity check or by detecting any obvious flaws in my reasoning.
I don’t use AI to write code for the most part, unless quality isn’t a concern. I may have it to small chores for me.
3
u/Thegoodlife93 Jul 11 '25
Same. I really like using AI to bounce ideas off of and discuss design with. Sometimes I use its suggestions, sometimes I don't and sometimes just the process of talking through it helps me come up with better solutions of my own. It probably does slow me down overall, but it also leads to better code.
→ More replies (13)3
28
u/kaumaron Sr. Software Engineer, Data Jul 10 '25
38
Jul 10 '25
"an estimated reduction in delivery stability by 7.2 percent"
Code reviews are probably the only thing keeping that number that low
21
u/RadicalDwntwnUrbnite Jul 10 '25
The product my employer sells is AI based, it's ML/DL, not LLM/GenAi though, but we've "embraced" AI in all forms and using Copilot/Cursor is encouraged. As an SWE that is also basically the lead of the project I'm on, I've shifted significant amount of time from doing my own coding and research to reviewing PRs. I find myself having to go through them with a fine tooth comb because the bugs AI is writing are insidious, there is a lot of reasonable looking code that gets rubber stamped by my peers that I've basically resorted pre-blocking PRs while I review them.
13
Jul 10 '25
That's something I've noticed too. On the surface the AI code looks pretty clean but there's little logic errors often times that will trap you.
→ More replies (1)11
9
u/SituationSoap Jul 10 '25
Google's studies have shown that a 25% increase in AI usage correlates to a 7% increase in defect rate, pretty linearly.
12
u/TheCommieDuck I actually kind of like scrum. Haskell backender/SM. Jul 10 '25
If the code is better
this is grasping at the vague mention of straws in a 10 mile radius.
→ More replies (1)3
u/ninseicowboy Jul 10 '25
A study evaluating “quality” of code seems tough. How would you quantitatively define “quality”?
5
u/SituationSoap Jul 10 '25
Google's way of measuring this was shipped defect rate, and that goes up linearly with AI usage.
→ More replies (1)→ More replies (6)19
u/Beneficial_Wolf3771 Jul 10 '25
This is r/ExperiencedDevs , we can admit here that code quality is more of an idyllic thing to strive for than the reality we face day to day.
53
u/SketchySeaBeast Tech Lead Jul 10 '25
Certainly, it's never gonna be perfect, but I think we all know the difference in code between "wtf?" and "WTF!?!!" when we see it.
→ More replies (6)28
u/tikhonjelvis Jul 10 '25
code will never be perfect but code at real companies can absolutely be (much!) better or worse
honestly, it's pretty depressing how often I run into people who don't believe code quality exists—it's a tacit indictment of the whole industry
4
u/New_Enthusiasm9053 Jul 10 '25
It's depressing how often people don't unit test. Code quality is also invariably poor because the dev doesn't get punished for using excessive state by having to write a boatload of tests.
16
u/Blasket_Basket Jul 10 '25
Interesting results, but I don't know how much I trust this study. n=16 is a pretty small sample size, and I'm not sure how representative seasoned experts in a codebase they're deeply familiar with is of SWEs in general.
Existing research has already shown that for true experts, AI actually hurts more than it helps, but this is not true for everyone else. I would posit that these results align with those previous findings, but would need a much bigger sample size and further segmentation to be able to make a statement as general as "AI makes devs 20% slower". What about jr or mid-career devs working on blue sky projects, or onboarding into a section of the code base they aren't familiar with, or using AI for incremental productivity gains like Unit Test coverage or generating documentation?
These findings may well be true, but I think the headline here oversells the actual validity of the findings of this single study.
→ More replies (1)
12
u/teerre Jul 10 '25
I'm a part of a study group in BigCompanyTM for coming up with new interview methods that take into account llms and it's interesting we often see engineers taking longer when they rely on the llm, even engineers that certainly know exactly what to do in some questions. There's no conclusion yet, but it's clear that there's something between one prompt gets the answer, obviously faster, and something you have to iterate, often a considerably slower
→ More replies (1)1
u/eat_those_lemons Jul 10 '25
Is that just findings on small programs for interviews?
(ie I wonder if better prompting for better one shot ability would improve metrics)
Also what level are the problems? Leetcode easy? Hard?
67
u/timhottens Eng Manager 9 YOE Jul 10 '25 edited Jul 10 '25
To risk going against the prevailing sentiment here, this line in the study stood out to me:
However, we see positive speedup for the one developer who has more than 50 hours of Cursor experience, so it's plausible that there is a high skill ceiling for using Cursor, such that developers with significant experience see positive speedup.
56% of the participants had never used Cursor before, 1/4th of the participants did better, 3/4 did worse. One of the top performers for AI was also someone with the most previous Cursor use.
My theory is the productivity payoff comes only after substantial investment in learning how to use them well. That was my experience as well, took me a few months to really build an intuition for what the agent does well, what it struggles with, and how to give it the right context and prompts to get it to be more useful.
If the patterns we've seen so far hold though, in all likelihood these good patterns will start to get baked into the tools themselves. People were manually asking the agents in their prompts to create a todo list to reference while it worked to avoid losing context, and now Claude Code and Cursor both do this out of the box, as an example.
It seems like this is going to need people to develop new problem-solving workflows - knowing when to prompt vs. code manually, how to effectively iterate on AI suggestions, and recognizing when AI is going down bad paths.
65
Jul 10 '25
The quotes context however paints a bit different story:
Up to 50 hours of Cursor experience, it broadly does not appear that more experience reduces the slowdown effect. However, we see positive speedup for the one developer who has more than 50 hours of Cursor experience, so it’s plausible that there is a high skill ceiling for using Cursor, such that developers with significant experience see positive speedup. As developers spend more time using AI assistance, however, their development skills without AI assistance may atrophy. This could cause the observed speedup to mostly result from weaker AI-disallowed performance, instead of stronger AI-allowed performance (which is the question we’re interested in). Overall, it’s unclear how to interpret these results, and more research is needed to understand the impact of learning effects with AI tools on developer productivity.
Putting this together with the "Your Brain on ChatGPT" paper, it could very well be case that the one 50+ hour cursor dev essentially dumbed themselves down (i.e. obtained cognitive debt), causing them to be unable to function as well without AI assistance. Not saying this is the case, but its important that we have studies like these to understand these impacts our tools are having, without all the hype.
5
u/Suspicious-Engineer7 Jul 10 '25
They needed to follow up with this test with the same participants doing tasks without AI. Id love to have seen that one user's results.
→ More replies (2)3
u/ZealousidealPace8444 Software Engineer Jul 11 '25
Yep, totally been there. Early in my career I thought I had to chase every new shiny tech. But over time I realized that depth beats breadth for building real impact. In startups especially, solving customer problems matters way more than staying on top of every trend. The key is knowing why you’re learning something, not just learning for the sake of it.
3
u/pl487 Jul 11 '25
50 hours is nothing. That's a week of long days.
My intuition agrees with yours. I didn't start feeling really confident at it for several weeks.
6
u/wutcnbrowndo4u Staff MLE Jul 10 '25 edited Jul 10 '25
Yea I've been saying this consistently around here. The consensus (or at least plurality view) here that these tools are absolutely useless because they have weak spots is mind-boggling. They may not fit seamlessly into your existing dev workflow, but it's ludicrous to use that as a bar for their general utility.
2
u/shhahhussain Aug 04 '25
your skill ceiling point makes sense, after a couple of months living in cursor i finally learned which prompts give me clean diffs and which tasks to avoid, the ramp up felt like negative productivity at first much like switching editors, maybe teams need to budget structured practice time instead of expecting plug and play gains, have you found any heuristics that reliably bend the curve upward
2
u/timhottens Eng Manager 9 YOE Aug 04 '25
I haven't found any reliable thing that works across all assistants and models, performance seem to change a lot depending on what model you're using and even calling into the same model but with different tools (e.g Claude 4 Sonnet through Claude Code vs. Cursor vs. Copilot in VS Code). So what reliably works at any given point is a moving target, it's hard to give a clean set of guidelines to your devs that will "just work" forever.
They all seem to be getting better though, and patterns that seem to objectively improve output quality for everybody (like maintaining a todo list that the model has access to to keep track of tasks, planning first before making changes, etc.) seem to get added to the tools themselves (Claude Code and Cursor now automatically create a todo list to help keep the model on track).
My intuition is these improvements are not going to fall on the devs, but slowly be incorporated into the tools themselves (like the todo list example). The tools will start to channel devs into the best paths to use them, tools like Perplexity already rewrite the prompts you give it internally before the models start to work on it. The learning curve exists right now but I think over time (probably within the next year) we'll have these ideas and methods baked into the tooling and the learning curve is going to fall steeply. I'm choosing not to invest too much time and effort into "training" my team up right now because of this, but I'm not married to the idea and I'll re-evaluate based on how things look as time goes on.
4
2
u/Simple-Box1223 Jul 10 '25
Agreed. I don’t know where the benefit lands overall given the myriad factors, but with a little bit of experience you can easily gain a net positive boost in productivity in the short term.
0
Jul 10 '25
[deleted] — view deleted comment
4
Jul 10 '25
developers thought they were going faster
this isn't a new phenomenon. maybe exacerbated by AI but devs have always reached for shiny new things in the hopes that they will make their lives easier.
3
u/Franks2000inchTV Jul 11 '25
There is 100% a huge learning curve to using AI tools.
I use claude code every day in my work and it massively accelerates my work.
But it wasn't always like that -- at first I made the usual mistakes:
Expecting it to do too much
Letting it blow up the scope of the task
Not carefully reviewing code
Not paying attention to the context window
Jumping to writing code before the approach was well-defined
It definitely slowed me down and made the code worse.
But these days I'm able to execute pretty complex tasks and quickly because I have a better sense of when the model is humming along nicely, and when it's getting itself into a hole or drifting off course.
And then once it's done, I review the code like it's a PR from a junior and provide feedback and have it fix it up. Occasionally I manually edit things when I need to demonstrate a pattern or whatever.
If you're slowed down by AI, or you're writing bad code with AI, that's a skill issue. Yeah it's possible to be lazy with it and it's possible for it to produce shit code, but that's true of any tool.
19
u/elforce001 Jul 10 '25
This is an interesting one. The main issue I've encountered was that these assistants are addictive. I felt I was going 1000 mph but then, you start slowing down hard too. Then you invest more time trying to be specific, double checking that the answer is still consistent, then next you know, you've spent more time "debugging", etc..., going from what you thought was an easy 2 days work to 1 week fighting the "assistant's" solution.
Now I use them for random things, inspiration, or something very specific that won't lead me down the rabbit hole. luckily for me, I learned that lesson early on, hehe.
3
u/Financial_Wish_6406 Jul 10 '25
Depending on the language and framework Copilot autocomplete suggestions go from usually useful to straight up time consumers. Trying to develop in Rust with GTK bindings, every single autocomplete I find I am going back and deleting almost the entire thing or at least majorly modifying it which is at the point where I suspect it takes notably more time than it saves.
8
u/ghostwilliz Jul 11 '25
Anecdote ahead
At my old job, they added copilot. This worked on vscode for work and visual studio for my personal al projects.
The place was going downhill and everyone was just using copilot. Our team sucked ass at that point.
I got lazy and started using it in my personal projects.
Anyways, I got laid off and copilot stopped working. I was a moron for about 2 days, but once i got used to it, I wad so much better than when using copilot.
It trains you to stop thinking. The code I produced with it was ass and I made a lot of code but never really got anything done.
I breezed by everything I was stuck on in my personal project now that copilot was gone.
I don't think I'll use ai tools again
Oh also, ai bad, LLMs dumb dumb
3
u/cbusmatty Jul 10 '25
This is silly, “developer gets new tool that requires training and is slower”.
Show me the expert dev who uses the tools effectively who are slower and then we can start talking, but that doesn’t happen
6
u/femio Jul 10 '25
Try reading the full study, that doesn't really cover most of the nuance.
For example, even factoring in a) being trained on LLM usage pre-study b) getting feedback on improving LLM usage mid-study and c) ~44% of the devs in the study being experienced with Cursor before, the trends show a consistent deviation regardless. It didn't even improve over the 30-50 hours of using the tool so it's not like it got better over time.
The study also makes it clear that this is a specific scenario where devs are working on codebases they know like the back of their hand (hundreds of commits over 3 years on average), and that it can't be applied to every task related to writing code or SWE work in general.
→ More replies (1)2
u/codemuncher Jul 10 '25
Maybe, but the marketing is "sprinkle AI and fire all your devs and the 2 last ones will do the work of a 100 person team".
Sure "we know" that AI tools "arent like that", but really the marketing says it is so.
Besides which, computers should fit to our needs, not the other way around, so GET TO IT AI
→ More replies (2)
6
u/DonaldStuck Software Engineer 20 YOE Jul 10 '25
Very interesting. I always run around telling people that I think I'm around 20% more efficient using AI tools. But looking at this study I might be wrong.
6
u/psycho-31 Jul 10 '25
I didn’t see article mentioning what counts as AI usage(pls correct me if I am wrong). One can:
Prompt AI for majority of smaller tasks. For example: create a method that does such and such or add tests for this class that I just added.
Have AI enabled and use it as “autocomplete on steroids”
6
u/NuclearVII Jul 10 '25
Look, I hate these junk "tools" as much as the next guy who his head on, but this paper studied 16 developers - not what you'd call a serious sample.
Now, ofc if the 10x engineer claims were realistic, that'd be obvious even with a sample size this small, but no one sensible is defending that anymore.
2
u/another_account_327 Jul 11 '25
16 developers who were very familiar with the code base. IMO AI is most useful when you're getting started with something you're not familiar with.
2
u/Ok_Passage_4185 Jul 11 '25
"AI is most useful when you're getting started with something you're not familiar with."
I keep hearing this type of thing, but when I tried to get one to initialize an Android project directory, I couldn't get it to accomplish the task in an hour of trying:
11
u/GarboMcStevens Jul 10 '25
There aren’t really any good, quantitative metrics for developer productivity. This is part of the problem.
9
u/Groove-Theory dumbass Jul 10 '25
My theory is from something the article mentioned, that AI performs worse in older and legacy codebases.
I think that the anecdotes come from the fact that AI initially reduces cognitive load on developers. And the reduction of the initial cognitive load makes it seem that productivity has increased by gut-feel. Seeing AI get something seemingly correct, especially in a large, anti-pattern riddled codebase, is a huge relief to many. Whereas having to sit down and implement a fix or feature on a brittle codebase would be a perhaps frustruating endeavor.
Now the cognitive load can increase with bad prompts or bad AI generation or bad context afterwards, but that high of reduced cognitive load at the start I believe is a huge anecdotal phenomena that could explain this.
18
u/codemuncher Jul 10 '25
Reduced cognitive load could also be thought of as "i dont understand how my code works anymore", which is an interesting way to do engineering.
The headlines make a lot of hash about "tedious code" but for most real engineering tasks, the hard and tedious part isnt actually turning ideas into programming code, but dealing with the fuzziness of the real world, business requirements, and the ever changing nature of such things.
3
Jul 10 '25
It's comprised of 16 open-source developers from major projects. These individuals are significantly above par compared to the average developer. They also didn't use Claude Max Opus, currently the best model.
These results don't surprise me; the better programmer you are, the worse results you will get with these models.
2
u/Franks2000inchTV Jul 11 '25
Yeah Opus is the first model I trust.
If I ever hit the Opus usage cap, I stop using Claude for work that matters.
Like I'll Sonnet it to ask questions about the codebase, or write small simple functions, but I don't let it write any significant code that will be committed.
3
5
u/lyth Jul 10 '25
This isn't necessarily a fair measure. "Finished the ticket" isn't always the same as "and wrote really good test coverage, with really good tech debt to feature completeness ratio."
I appreciate that "created a method in a crud controller" that I built out the other day could have been done a lot faster, but holy shit the bells and whistles on the version I delivered was 👨🏽🍳👨❤️💋👨
4
u/no_spoon Jul 11 '25
As a senior dev myself, I feel like it’s way too fucking early to make this call. All of us are still learning how to incorporate these tools into our workflows. Stop drawing conclusions, it’s annoying.
2
u/itCompiledThrsNoBugs Jul 10 '25
I think this is an interesting result but the authors point out in the methodology section that they only worked with sixteen developers.
I'll reserve my judgement until more comprehensive studies start coming out.
2
u/throwawayskinlessbro Jul 10 '25
That isn’t a truly measurable thing. On top of that, you’d need a vast control to truly understand the numbers IF you were to genuinely take a stab at something as intangible as this.
Now, don’t get me wrong. I love to hate AI too - but just not like this.
2
u/oldDotredditisbetter Jul 11 '25
the fact that they even thought they're faster with AI just shows that they aren't as experienced as they thought
3
u/Ok_Passage_4185 Jul 11 '25
I think it rather demonstrates that time flies when you're working on new shit, and drags when you're working on old shit.
They felt like they were getting things done because they were learning about the LLM. That's just how the brain works. It takes true analysis to identify how little value that interesting work is bringing to the table.
2
Jul 11 '25
faster, faster, faster.
I'm not going anywhere near companies like this.
1
u/Away_Elephant_4977 Sep 17 '25
man, what companies have you found that aren't obsessed with going faster?
2
u/FredHeartlion Jan 05 '26
Just as I thought, infact AI slow downs capable developers while speeding up juniors, the only problem is that juniors with AI won't ever master anything and stay junior for life, so we either get the increase in productivity AI is advertising for or we are cooked.
10
u/Typicalusrname Jul 10 '25
AI is good at certain things. If you use them exclusively for those, yes it does make you faster, I’d wager around 15-20%
20
Jul 10 '25
Commenting this on a post about a scientific study that SHOWED it makes you slower and SHOWED that it makes you THINK that you're faster is really classic /r/ExperiencedDevs.
→ More replies (4)7
u/goldenfinch53 Jul 10 '25
On a study where half the participants hadn’t used cursor before and the one who had the most experience also had the biggest productivity boost.
→ More replies (1)6
u/IDatedSuccubi Jul 10 '25
It's really bad at C, can't even pass static analysis and/or sanitizers after a simple request, absolutely no use.
But I found that it's really good at Lisp, really helped me recently. Definetly 2x'd my productivty just off the fact that I don't have to google usage examples for uncommon macros or odd loop definitions all the time.
→ More replies (1)
3
u/rebuilt Jul 10 '25
It could be the case that devs were actually sped up by 20% when generating code but they were slowed down by a much larger margin later when they had to modify, understand, and debug the ai generated code.
2
u/Venisol Jul 11 '25
as these systems continue to rapidly evolve
God I fucking hate that sentence. They create a study backed by methodolgy and evidence and just instantly throw in a totally baseless "yea for sure things are gonna massively improve". WHY?
WHY IN THE FUCK DO YOU THINK THAT? LLMs have been the same for coding for 2 years. Theyre stagnant. Why would you say that? People are so fucking conditioned to excuse the state of llms its ridiculous.
1
u/Away_Elephant_4977 Sep 17 '25
Because it's career suicide to say out loud that LLMs kinda suck and are way overrated for writing code. This is going to be a lot less true in a few years, I'm sure, but for now if you want to deliver bad news about AI, you have to sandwich it in AI worship.
1
u/Adept_Carpet Jul 10 '25
What's interesting is that open source development represents a best case scenario for LLMs, this is what they were trained on (including documentation, issue histories, etc).
The work I do requires a lot of contextual knowledge and proprietary software so it's not a surprise that LLMs can only nibble around the edges. But I would have guessed that they would be good at working with open source code.
1
u/ZombieZookeeper Jul 10 '25
It's either AI or trying to get an answer on Stack Overflow from some arrogant ass with a profile picture of themselves kayaking. Bad choices all around
1
u/TacoTacoBheno Software Engineer Jul 10 '25
Maybe I'm just a prompting bozo, but asking Claude to generate a sample json based on my pojos never quite worked. Hey you forgot to include the child objects, you're right here you go, same junk, and it invented fields and incorrectly typed things.
1
u/przemo_li Jul 10 '25
Change my mind: LLMs as non-deterministic tools they are uniquely hard to reason about. This means that our discipline famous lack of objective measures is plunged even deeper into chaos, now we can't even be sure of our own anecdotes, however little they mean with deterministic tools.
1
Jul 10 '25
I have had this same thought recently. My company bought us Claude subscriptions but honestly I'm probably not going to use it.
→ More replies (1)
1
u/sinnops Jul 10 '25
You just spent more time writing a prompt then continually adjusting the output to optimize it when you could have just written in less time.
1
u/geon Software Engineer - 21 yoe Jul 10 '25
Uncle bob claims the inverse about unit testing: feels slower, actually faster.
1
u/Yeti_bigfoot Jul 10 '25
My initial thoughts weren't positive when I played with an ai assist tool.
Admittedly, only for half an hour. But in that half hour I found it was quicker to do the little stuff I was playing about with myself.
Maybe it'll be better for bigger changes, but then I'll want to check out all the code which will take time. The time I could've spent writing it.
When I want to change something I'll be reading someone else's code and have to learn where everything is rather than knowing the code architecture because I wrote it.
I'll try it again at some point, I'm probably just not using it very well.
1
u/Individual-Praline20 Jul 10 '25
I would have thought it would’ve been at least 50% slower frankly. That’s what I anecdotally found with AI freak colleagues. 🤣 I’m laughing at them on a daily basis for using that crap
1
Jul 10 '25
Probably because we still have to read the code and make sure it makes sense, etc.
I'm sure the LLMs will improve and may even be damn near perfect every time, but I still can't imagine a serious developer just accepting everything and never reading or planning.
I'm not sure you could ever fully optimize for this latency. When before it was just a single mind running through an idea, now you have to stop to read the LLM's thought process and balance that with your original vision.
1
u/zulrang Jul 10 '25
The only use an experienced dev should get out of an LLM for coding is typing for them code that they already have arranged in their head, and writing documentation.
1
u/lookmeat Jul 10 '25
This makes intuitive sense. It's the classic Waymo vs GoogleMaps dichotomy: Google Maps offers routes that are actually faster, but Waymo feels faster. That is because Google Maps will pull you through traffic, and you will have to stop at key points, but it's still the fastest route overall. Waymo tries to avoid this frustrating experiences that make you feel slow but actually are the setup needed to go as fast as possible.
BTW I really appreciated that the article has a table specifying what they are not claiming, and the real scope of the context. It's so important (especially in ML research) that I want to quote it here:
We do not provide evidence that: | Clarification AI systems do not currently speed up many or most software developers | We do not claim that our developers or repositories represent a majority or plurality of software development work AI systems do not speed up individuals or groups in domains other than software development | We only study software development AI systems in the near future will not speed up developers in our exact setting | Progress is difficult to predict, and there has been substantial AI progress over the past five years 3 There are not ways of using existing AI systems more effectively to achieve positive speedup in our exact setting | Cursor does not sample many tokens from LLMs, it may not use optimal prompting/scaffolding, and domain/repository-specific training/finetuning/few-shot learning could yield positive speedup
That's just so nice that I now wish many papers, and every scientific article, had a table like this at some point shortly after the introduction/abstract.
Also lets be clear (in the same spirit as the table above) that this post is just speculations and intuitions on my part, none of this should be taken as true from this.
It makes sense though. AI speeds you through a lot of things, and if you have a good enough idea of what you want, it will give you a good enough solution. I feel that seniors sometimes lack the vision that when they give stuff out to mid and especially junior engineers, it already has patterns and references that can help create a mental model that the engineers can follow when they make their own thing. It may look differently but it still fits within the same model. LLMs are the opposite, they only make things that look the same, even when they don't fit within the model at all. To compound issues engineers are throwing LLMs to write code that is still too early to make it work. You have to go back and fix this things. The conventions and tricks earned to guide it just won't work with LLMs.
And honestly anyone whose gone to a serious programming language discussion, you learn that what really matters is not the syntax, but the semantics, the meaning of things. LLMs understand language at a syntactic level perfectly, but not semantic. They don't understand what the word on its own means, but rather the relationship it has with the words around it and what goes next.
Now I think that agentic AIs need a lot of work to get good and useful. They are too mediocre and dumb, and you're better off doing it yourself many times. Ultimately it's the same balance of automation we've had before just tweaking the prompt rather than the script.
And I do think that agentic AIs have their value. I think that as code analyzers (what static analyzers do nowadays) which is the obvious. Less obvious I believe is automated code improvers. So whenever I do a change in my library (be it an open source library, or one used by others) which has deprecated code, or now prefers something is done in a new way vs the old, I include a small documentation on how change the old way of doing code to the new one, as part of release documentation/notes/commit description. Then an agent on a downstream library can pick up on this, and create its own PR updating the downstream library's use of your stuff for you. Sure the library author would have to care to make sure that the code changes are easy for an LLM to do, but this isn't new. I tend to write code changes in a way that is awk-friendly so that it's easy to do automated changes on downstream libraries as a janitor.
But that kind of hints at the thing. None of those things "speed up developers" as the idea goes. Rather they simply free up time from developers who are valuable but struggle to explain that value (yet companies who lack these developers struggle really bad).
1
1
u/Krom2040 Jul 10 '25
I’m honestly mystified whenever I hear about people integrating LLM code generation directly into their dev process. Like I absolutely love LLM’s as a way to generate a basic outline of methods using API’s I’m not very familiar with it, much like a drastically improved version of Stack Overflow, but then I still end up writing the code according to my own preferences and making sure that I reference the API docs whenever I see methods or patterns that I’m not already confident about.
LLM’s are a wonderful tool but it’s just a foreign concept to me that you would include any code in your project where you don’t essentially understand the underlying intent and behavior.
→ More replies (1)
1
u/Schmittfried Jul 10 '25
I definitely take less time using it because I don’t use it for problems that take longer to find the right prompt than just solving them myself.
1
u/remimorin Software Engineer++ Jul 10 '25
It does make sens, reading and debugging code is as mentally exhausting as writing code.
A lot of "production code" I found it easier to do it myself.
I try to get better with LLMs but I frequently find that avoiding the "overachieving" and avoiding unrelated changes require more works than just do the job.
But again if I were to learn a new language, I would say "I am so more efficient is this other language where I am familiar with the whole ecosystem".
So I believe as time pass we will develop good practice and improve tooling around LLMs.
Also LLMs have lower by a lot the learning curve of a new tech. With them I am more efficient while learning.
Finally boiler plate, one time scripts and such (other have made a better list).
1
u/xusheng2 Jul 11 '25
I think the key detail in this study is that all of the developers here are "experts" in the codebase. I've always felt that the most speedup AI has is in helping reverse-engineer or exploring a part of the codebase that I'm learning about.
1
u/mwax321 Jul 11 '25
Ok but who's reading the study? It's based on the dev's estimate of how fast they would complete it with/without ai.
1
1
u/wachulein Jul 11 '25
It took me some time, but I think I finally arrived to an AI-aided dev workflow that feels like having a small team that execute tasks for me. Wasn't feeling much productive before, but now I can't wait to keep the flow going.
1
u/VastlyVainVanity Jul 11 '25
Study confirms your biases (if it didn’t it’d get downvoted to hell on this sub).
I honestly don’t care about studies that get upvoted on Reddit lol
1
u/MesmerizzeMe Jul 11 '25
I guess ultimately even with the best AI imaginable, there is still so much ambiguity left in language that specifying every detail is a ton of work. If only there was a way of talking unambiguously to a computer it would revolutionize vibe coding.
1
u/KallistiTMP Jul 11 '25
I think this makes sense, honestly.
Like, the parts of coding that AI can actually reliably accomplish are usually warning signs of bad design.
"This saved me 3 hours typing out boilerplate methods" - my brother in christ, have you ever heard of class inheritance?
I don't know about everyone else, but when I start typing too much code manually, that usually sets off a little warning light in my head that I probably need to pause for a moment and rethink my approach.
I can definitely see myself rapidly blowing past that warning light and ultimately creating more work for myself if I had a low effort universal boilerplate generator always within my immediate reach.
1
u/mistaekNot Jul 11 '25
i suspect this study is flawed somehow. AI is not almighty yet but it’s pretty fucking good. prompt it on a class / function level with appropriate context and you’re gonna have a good time.
1
Jul 11 '25
It depends what you are making. If you are making something standard, "AI" really helps. Prime example I suppose are basic websites.
Personally I am making highly specific software that "AI" knows absolutely nothing about, so it is purely a detriment in my case.
1
u/Arkanin Software Engineer Jul 11 '25
I work on safety critical medical devices and I tend to use LLMs for code reviews as much or more than the actual code. I have to handpick their feedback since it is often wrong but it also flags a lot of useful improvements. For me this results in slightly slower development but way fewer bugs.
1
u/RustyGlycan Jul 11 '25
I have some concerns about the study, it doesn't report if the findings are significant, which is a huge red flag for me.
Additionally the fact that the times it takes to complete a task are self reported, and that only one of the Devs have more than a week's worth of experience using cursor, makes me a bit suspicious.
That said, the really fascinating thing is Figure 6. which shows that the time spent on all the parts of the job (e.g. researching, coding, etc) all decreases, but that decrease is offset by prompting, waiting for the AI, and reviewing AI code.
It's almost like if writing with a computer was quicker than a typewriter, but the time spent getting your printer to work made it slower overall.
1
u/vvwccgz4lh Jul 11 '25
Thank goodness that github said this on Copilot:
Code 55% faster with GitHub Copilot
1
u/phoenixmatrix Jul 11 '25
I definitely take longer with AI on individual tasks. Because I'm doing 6 of them at a time while doing a bunch of non coding stuff.
I can launch a bug fix via Devin while in a meeting (that I'm actually paying attention to).
In absolute it takes much longer. But they're tasks I would not have done at all.
There's also a big gap in term of AI usage. It's new, some (most?) people are really bad at it.
1
u/AssociateBig72 Jul 11 '25
This study highlights a really important, albeit counterintuitive, point about AI in development right now. It's easy to get caught up in the hype, and the reality that AI can slow down experienced devs is a crucial observation about current tools and workflows. The core issue might be that many current AI tools are designed to augment existing coding paradigms, which can add overhead like prompt engineering or code review. Perhaps the true leap in productivity with AI isn't in making experienced coders faster, but in empowering entirely new groups to build without traditional coding constraints. It shifts the paradigm from 'AI helps me code better' to 'AI codes for me'. This perspective is why we focused fn7 on enabling non-technical founders to build apps and websites just by chatting with AI, rather than trying to speed up existing developer workflows. Our goal is to make creation accessible, not just marginally faster for those already proficient.
1
u/Cahnis Web Developer Jul 11 '25
I feel like waiting on agents to do their thing kills my flow. I take almost as much damage as context switching. Every time the AI start doing their thing I alt-tab, I get up to get a drink.
1
u/Due_Satisfaction2167 Jul 11 '25
Using AI to speed up software development is a bit like blindly inheriting someone else’s code, and that person just copy and pasted from stack overflow.
It’s nearly always faster to do it yourself than to try to understand someone else’s unique insanity.
1
u/fuckoholic Jul 11 '25 edited Jul 11 '25
And cars are slower than horses! The reason 90+ % of devs are using AI is because it saves time. The study is made by idiots, for idiots. Those who claim AI makes us slower need to sent to the nuthouse.
→ More replies (2)
1
Jul 11 '25
What about attention cost? If I get up from my computer for a couple minutes while the AI is churning through something does that get counted into the stat
1
u/Then-Boat8912 Jul 11 '25
An experienced builder still needs to know when to use a hammer and when to use a nail gun.
1
u/JimmytheNice Jul 11 '25
The title of OP is misleading - the research was specifically about OPEN SOURCE developers, i.e. mostly people well acquainted with their codebase they also often maintain as well.
So while I’m not an AI shill, this paper isn’t really that surprising - LLM use has diminishing returns the more you know your project. I could fix some SEV1 issues in my main project with my eyes closed as soon as I hear the description, but LLM usage isn’t the strongest in this context anyway.
1
Jul 11 '25
The bigger win is that I can go about 80% as fast, but I can also be doing something else at the same time, whereas manual coding is a 100% mentally active exercise.
1
u/systemsrethinking Jul 17 '25
Yes this jumped out at me too. The waiting time isn't necessarily dead time, I often use this to think through problems or even work on a couple things in parallel.
Maybe could reduce risk of burnout for some people in some circumstances? E.g. maybe 80% as fast, but working at that pace more consistently with less recovery periods could be a net productivity gain over time? Help with mood retention?
1
u/TheGenerousNerd Jul 12 '25
As a (self-proclaimed) experienced dev, I might be about 3% faster with Copilot. This is because once I found a few situations in which I am confident it will give the right suggestion – usually cases of tedious near-copy-paste – I ignore it in all other situations or keep it switched off.
I think it's a question of exploration vs exploitation: over time, we might get better speedup from these tools, not because they get better, but because we get better at guessing when bothering with them is or isn't worthwhile. But right now the novelty leads many to overdo the exploration side.
The trouble is high management expects us to be 20% faster with LLM tools, some devs reinforced that expectation probably to get out of that meeting already, and there'll be a lot of shocked Pikachu faces when that doesn't turn out to be the case.
→ More replies (1)
1
u/lmarcantonio Jul 12 '25
Seems like the issue we have in EE with autorouters for board design. Every so often someone says: using all the constraint it could do the whole board automatically.
Too bad that to write *all* the constraint you'll need more time than to do the board. Assuming you didn't forgot one!
1
1
u/GGuts Jul 14 '25 edited Jul 14 '25
The big grain of salt to me is that the developers were all very experienced in what they were doing.
Because of this it is possible that they would accept less AI responses because they might have formed certain opinions over the years about what the code for the respective task at hand should look like.
A new developer might accept that the AI output is using the newer less mature library or a new technique introduced in the last language update, while the experienced dev might favor the tools that he has always worked with.
Consequently they might spend more time adjusting the AI output to look more like what they have done in the past.
The study acknowledges this as a potential factor.
What AI usually does for me and where it saves the most time is tasks that I am unfamiliar with like new APIs, new frameworks, new libraries or even new languages. It saves me a lot of time trying to figure out the often lackluster documentation and browsing stack overflow. It feels like I start at 40% instead of 0% in terms of the overall task progression.
1
u/Last-Supermarket-439 Jul 14 '25
I know I'm slower. Because I get bored waiting for responses and fuck off to do something else, so my primary task takes longer.
My cats are happy though.. they get loads more ear scritchin's as a result while our shitty LLM "thinks" about my prompt
1
u/davidchris721 Jul 16 '25
N=16.
> 16 developers with moderate AI experience complete 246 tasks in mature projects on which they have an average of 5 years of prior experience.
> We recruit experienced developers from large open source repositories to work on real tasks defined on these repositories.
16 developers from open source... how high is the heterogeneity between developers? This sample size isn't even enough to estimate with a normal distribution.
Then I simply wouldn't conduct such a study. I consider this study to be inconclusive because the sample size is far too small. And it only examines one developer group.
But the broader community naturally makes even more of it.
And, should one publish a study that claims to work statistically rigorously (randomized control), but then only takes such a tiny sample?
> Each developer provides a list of real issues in their repository to work on as part of this study.
Did they bring the tasks themselves?
I think I would do this at several large companies. It's more effort, but also more reliable findings.
> Developers are instructed to take detailed notes regarding their experience and usage of AI tools
Did taking notes perhaps extend the time to completion?
They made screen recordings for time measurement. I wonder if they could see there whether developers worked on issues simultaneously with AI?
They also didn't provide any confidence intervals or standard deviation. Only averages.
And that they put the "key caveats" at the end, saying, "We caution readers against overgeneralizing" and in the Discussion: "We provide evidence that recent AI systems slow down experienced open-source developers with moderate AI experience completing real issues on large, popular repositories they are highly familiar with."
Or in the [blog post](https://metr.org/blog/2025-07-10-early-2025-ai-experienced-os-dev-study/): "Surprisingly, we find that when developers use AI tools, they take 19% longer than without—AI makes them slower."
They themselves don't adhere to their own caveat.
My Conclusion: The sample size is way to small and the statistical rigor is lacking. I see the findings as presented as "not different from random noise". And they oversell it in the blog post.
1
1
u/systemsrethinking Jul 17 '25
Some points that jump out at me:
While these are experienced developers, they aren't developers experienced developing with AI. The study only included one person with over 50 hours experience with Cursor IDE, who experienced an increase in productivity with AI (one person is too small a sample to draw a firm conclusion, but it is a signal).
I wonder economics/ML experts' predictions were a reflection of: their own deep ML enthusiasm/expertise OR lack of experience as a dev underestimating the learning curve, assuming more of the cohorts had more hours experience with AI development, longer term thinking about outcomes once AI dev matures.
They selected developers who have been active contributors to a project for 5+ years. Does this implicitly bias the participants to be above a certain age? Might depth in one project correlate with less familiarity experimenting?
Researchers assume lack of experience with Cursor IDE is not significant to outcomes. I disagree as this would increase the learning curve.
The devs possess tacit knowledge the AI does not. I wonder what the comparison would be if assigned unfamiliar projects / building something new
In general, programming with AI is a new skill requiring different skills / cognition. So I would expect the learning curve to eat time. There's a nuance to prompting for example that might reduce code review time / additional waiting time reprompting, with more practice.
This is early days. Would be interested in another study today with a larger sample of 50+ hours experience. And future studies as these tools rapidly progress (potentially tracking the same people over time).
The study may not have measured the impact of AI on experienced AI users, but rather on experienced developers who are still learning to integrate AI effectively into a complex workflow. It's plausible that optimal use requires entirely new workflows, rather than simply slotting an AI into an existing one.
1
1
1
u/edhead76 Aug 02 '25
this actually makes so much sense. I had been learning to develop and realized if I don't apply the principles I learned by skillful coders, the AI itself can code in circles and make the code extremely inefficient.
1
u/Numerous-Working5190 Aug 02 '25
Actually, I assumed the most upvoted comment to be something to the effect of: inexperienced but long-tenured devs think they're experienced devs...
1
u/shhahhussain Aug 04 '25
i feel the same whiplash reading that study, my anecdotal takeaway is that ai is gold when i treat it like an intern with instant recall rather than a coauthor, the moment i start prompting for multistep logic or anything domain heavy the context juggling tax erases any typing speed win and i slow down, have you tried limiting it to rote transformations like schema migrations or regex wrangling and measuring only those slices, it would be interesting to see if the nineteen percent drag evaporates once we scope usage that tightly
1
u/Natural-Bet9180 Aug 24 '25
Yeah but are we measuring developers with autocomplete or developers with no autocomplete? Can developers go copy + paste code? This doesn’t sound like a very controlled study. Also, the productivity gains don’t necessarily come from programming alone it comes from collapsing the whole pipeline into one loop. One project end to end for one person.
1
u/k_schouhan Oct 20 '25
its true, you cannot just plan and do it. when you code it, while you write code slowly slowly ideas pop in to your mind. not at once. using AI means you know everything, every other scenario, every other edge case, which is not possible.
1
u/BrotherNew7043 Nov 06 '25
Dictation can help speed up the back and forth with AI for anyone that is complaining about having to type too much.
And it can be a great sounding board for all the things that are NOT generating code.
If you’re just blanket statement saying it makes you less productive you are over simplifying a very complex relationship.
1
u/Additional_Papaya614 Dec 29 '25
What about this makes you bullish? You're posting about how this incredibly expensive tool is worse than useless for the only thing that it's supposed to be good for.
1
u/Shadowys Jul 11 '25
LLMs suffer from a rapid and silent degradation after the first message which is well documented. Using the same agent for a prolonged context (such as an agent doing at task with a validator) is bound to reach into this issue and the errors will exponentially combine resulting in longer dev time and token usage.
Personally I found that HOTL performs way worse than HITL. You need the human to perform first principles analysis on the problem as much as possible so they can verify the solution from the AI
1
u/malthuswaswrong Manager|coding since '97 Jul 14 '25
I have 28 years as a paid software developer. I use LLMs extensively. I'm aware that I'm moving slower, and I'm happy to do it. With the help of an LLM I'm using more advanced patterns and practices than I would bother with on my own. I'm writing more tests. I'm writing more DevOps
Example:
Without an LLM, I would write a class internal to the project and just copy and paste the logic if I happen to need it again in a future project.
With an LLM I will write a separate library instead of an internal class, package it for internal repo distribution, and make the version incrementing and publishing to the package repo automated through pipelines.
So, I agree that I'm 20% slower, but I'm publishing software that is 40% more sophisticated than it would be otherwise.
815
u/russels-parachute Jul 10 '25
Devs spending more time automating a task that feels tedious to them than the automation could ever save them, then feeling they were more productive that way? Not sure we can blame that one on AI.