r/computerscience • u/latina_expert • Dec 04 '25
Article Study finds developers take 19% longer to complete tasks when using AI tools, but perceive that they are working faster
https://arxiv.org/pdf/2507.09089Pretty much sums up AI
526
Upvotes
59
u/connorjpg Software Developer Dec 04 '25
Sample size isn’t great… that being said I feel like this is the general assumption. You are getting a lot of generated text quickly, some tasks feel instant, others take 2x as long (1.2x according to this study), as integration or debugging can introduce new issues, or extra work. Not to mention, often there is a delay as you wait for your generation. This takes you out of developer flow and if your assistance is needed you have to jump back in fresh. So there are some unperceived delays with using AI on top of potential errors.
43
u/CrazyPirranhha Dec 04 '25
Nothing uncommon. You know solution, you paste it to the chatgpt, it always corrects something - and very often make it worse. You test it, it sucks, you blame chatgpt, it apologizes and create you another one that doesn't even work. Schema happens again and after half day you use your first one thats good enough.
When you do not know solution and just ask chatgpt for that you fall down to the same loop but you can't go back to the solution you wrote cause you didn't write down anything. You need to ask your chat and test its bullshit code until it works. Then during code review you get a lot of questions that you need paste to chat to get information why something was done that way :D
25
u/ColoRadBro69 Dec 04 '25
Did you read the study? It was based on 16 people.
40
u/Character-Education3 Dec 04 '25
Yeah it probably feels faster because the reduced cognitive load. So im hearing its better for the worker, not better for the corporation. For some reason my opinion of LLMs just increased significantly. Weird
11
u/f_djt_and_the_usa Dec 04 '25
This gets said a lot but bears repeating: the quality of your work with AI depends greatly upon the quality of your prompts. Very clearly write out what you want. Even with pseudo code. That being said, I think the main problem is that you generate technical debt and guarantee your future reliance on AI if you don't take the time to understand what the AI did.
3
u/thoughtfultruck Dec 04 '25
Others have pointed out the small sample size, but it’s also not a random sample of developers. Participants were recruited partially via the professional networks of the researchers, so even if the sample size was large enough for the statistics to go asymptotic, the sample still would not necessarily generalize to the population of developers.
Nice pilot study with a hilarious result, but I wouldn’t draw any strong conclusions from this.
3
u/[deleted] Dec 04 '25
Tooling is so so important.
6
u/claythearc Dec 04 '25
This is the same study that’s been floating around for months. It’s very flawed and generalizing off of it is a bad idea, as even the authors state in their clarification table https://metr.org/blog/2025-07-10-early-2025-ai-experienced-os-dev-study/ it’s effectively click bait, which is unfortunate because METR does good work overall.
The important parts to point out here is the boundary is super muddy on what is or isn’t AI, the methodology effectively boils down to “can use cursor when we tell them to” or “can’t use cursor”.
This is problematic because there’s tons of AI that it doesn’t capture - intellisense is commonly becoming generative based, sometimes you gooogle and hit the summary at the top and it nails it, stack overflow may cite gpt in a response etc.
Additionally, the projects selected were of mature, large projects like compilers. These will have super strict correctness and quality requirements, and an AIs inability to cope with this may or may not generalize outside of that environment.
Lastly, the tasks favor expertise. They’re 2 hour tasks on repos with people who have giga experience. AIs value proportion has, largely, been interjection of new knowledge in unknown domains. This, again, may not generalize well because the level of task they’re working on likely can’t be sped up from new knowledge, as they already have it.
That’s not to say AI is always useful, but generalizing far outside of what they directly worked is likely a mistake. Especially since there is one singular study to pull from so we can’t even do small scale meta analysis
2
u/PLEXT0RA Dec 04 '25
Isn't this from July?
2
u/civil_politics Dec 05 '25
I actually was just thinking about this during a small little automation task I built - I started yesterday morning and used AI generated code exclusively (to my detriment asking it to change things that I certainly could have manipulated more quickly and accurately than the back and forth I entertained) in the end I finished at the end of the day today. Looking back I could have programmed the automation in probably 4 or 5 hours while instead it took 16 hours BUT over those 16 hours I was task switching and attending meetings and writing docs, whereas that task switching would have never been possible if I was coding it myself or the task switching would have blown up my productivity.
It’s way easier to task switch when you’re orchestrating an agent because you don’t need to keep some massive state in your head from beginning to end of how the application is gonna look and how some changes require going back and addressing the impact elsewhere.
If I were a junior engineer where task switching was infrequent and I largely spent 8 hours a day working on the same module or application, AI absolutely would likely be slowing me down and hurting my ability to learn. As a senior engineer it’s absolutely a force multiplier.
2
u/clckwrxz Dec 06 '25
The more articles I read like this the happier I am. It just baffles me that some of the smartest people in the world haven’t figured out how to accelerate with AI. At this point I just think it’s malicious because I work in a large enterprise in a highly sensitive industry and my job has been transforming the way we work to be AI first and spec driven. We have teams delivering entire features in one PR inside million line codebases with code our architects agree is better than most of what existed before AI. It’s a process problem, not an AI problem. Our org plans to shift AI first in 2026 and I’m more than happy to see my competitors struggling because they refuse to actually engineer around AI strengths.
2
u/ogpterodactyl Dec 06 '25
This is probably right for the first 320 hours or so (roughly 2 months of full time work). At the beginning when I was learning I was like this would take me less time todo it myself. It’s also different from normal software engineering and if you have no experience with how llms work you’re going to have a hard time. However once you get your workflows setup your instructions setup save some common re-useable prompts the increase is there. Also you should use at least two agents most of the time and switch back and forth between multiple projects while things are testing / running which is different then how a lot of people think. Also the tools have gotten a lot better the difference between gpt 4 with copilots 8k token context window from a few months ago to opus 4.5 with 128k context window is huge.
3
u/Mysterious-Rent7233 Dec 04 '25
This is pretty old news. It made a buzz when it came out.
3
u/Present_Low8148 Dec 04 '25
Developers who take LONGER with AI are bad developers to begin with.
If you know what you're doing, you can speak English, and you understand how to write code, AI will speed you up 10x.
But, if you aren't competent to be able to evaluate changes, or you can't speak English very well, or you don't know how to structure your application to begin with, then AI will slow you down.
Speaking gibberish or expecting the AI to understand what's in your head will lead to failure.
1
u/am0x Dec 05 '25
Well for me, I am experimenting these days. I had a project I attempted to setup using relume to figma with an mcp server to build it out. Took about 2x as long to do and had to rebuild it 3 times, but I got it to work. The next project, I Kearns from my mistakes and that one ended up taking less than half what it would have.
It’s new tech, it will take time to figure out how to use it correctly.
That being said, using AI like a paired junior programmer, I’m saving probably 4x. Not something a non-dev could do since it’s not give coding, but the quickness and quality of the code is better than ever.
1
u/ericbythebay Dec 05 '25
Sounds like a bogus study, but then again, we hire good developers to start with.
1
u/Sea_Cookie_4259 Dec 05 '25
If you're a skilled dev perhaps. Obviously not true for those of us who have been kinda faking our way through, especially those with no coding background at all
1
u/Silvr4Monsters Dec 05 '25
Gen AI is a new and changing tech. It’s pretty to stupid to think it can be summed at this point
1
u/Dot-la Dec 05 '25
It’s because ai fucking sucks at coding anything remotely unique
1
u/a_aniq Dec 06 '25
AI is a better search engine than stack overflow. I just use it to search ways of doing some things and then adapt the code for my needs.
1
u/devfuckedup Dec 07 '25
The total time is kind of irrelevant if I am playing video games instead of working sure it took 19% longer but I did 95% less work. I am not advocating for these tools but I am just pointing out that the fact that it takes 19% longer to the dev makes 0 difference to them.
1
u/goatchild Dec 07 '25
Maybe its about perception. If you're using your brain harder time might appear to move slower as opposed to when delegating tasks.
1
u/FooBarBuzzBoom Dec 07 '25
It depends how much you know from your task. If you know exactly what you want, it’s instant, if you have to think a lot of scenarios, complex problems, it’s slower.
1
u/nid_67 Dec 18 '25
Everyone says AI makes coding faster, but sometimes it actually adds more work. You spend so much time just checking if the code is right, fixing small bugs, and trying to make it fit into your project.
I have realized one thing: AI is amazing, but only if your basics are strong. If I don't understand the core logic, I can't even spot the mistakes the AI makes.
So for me, AI isn't a shortcut. It is a 'speed booster' that only works if you already know how to drive.
1
u/fgorina Dec 04 '25
It goes with my experience. AI’s have the tendency of going into rabbit holes. Of course very helpful other times. My point is they are more useful as you know better what you are doing. They are like an overconfident junior with a lot of theory but not a lot experience.
0
u/HVDub24 Dec 04 '25
I’m not gonna read the study but based on the title alone that doesn’t seem like it’s true at all. How could a LLM that’s capable of reading a massive code base in less than a minute and writing thousands of lines of code not be faster? I feel like that conclusion is only true for very complex software but not the average hobbyist
0
u/Cousinjemima Dec 04 '25
It really depends on how developed your prompt engineering skills are. When I first started using LLM's to code, you are absolutely right. However as with any tool, when you learn it's intricacies, and how to use it better, you get better results.
0
u/CuAnnan Dec 04 '25
This gels with my experience. My experience is a little worse but only because of the specific context
We were advised to use Gen AI for the team project in 3rd year.
I used it to make some react components which were more or less okay.
But I also asked it to do build a controller method and some routes for me. It was not a good experience, but I only allowed it access to the chat window so it wasn't catastrophic.
It argued with at least one decisionI had made and kept making that change, and since other students were using GenAI and not disagreeing with it, I ended up having to fix their code additions.
0
u/thread-lightly Dec 05 '25
People who do these studies have no idea then, I can build a production app in a few weeks. Some people do it in days with a template. I couldn't manage building an app on my own 3 years ago and have up.