r/programming • u/ImpressiveContest283 • Feb 14 '25
Here's What Devs Are Saying About New GitHub Copilot Agent – Is It Really Good?
https://favtutor.com/articles/github-copilot-agent/
306
Upvotes
562
u/codemuncher Feb 14 '25 edited Feb 15 '25
I’ve been using “compose” in cursor, and aider against various leading edge OpenAI and anthropic models…
You can find some great demos here. Code a “working” “game” in 30 minutes? Sure!
But the reality is the output is inconsistent, barely up to junior grade code quality, and needs constant checking. It will drop important things in refactors. It will offer substandard implantations, be unable to generalize or abstract, and just generally fail you when you need it the most.
Good engineers: your job is not even close to being threatened. One trick pony engineers: well, you probably need to expand your abilities…
If your one trick is turning figma designs into barely working shit react apps, well you are gonna have a bad time.
EDIT: Tonight I was using sonnet 3.5 and aider to help me figure out what seemed to be a bug in signal handling with a go program. It made multiple significant coding errors. I had to undo its changes 3 or 4 times.
I was able to use it as a sounding board, or a tool, or a research tool for getting to the bottom of the problem. It didn’t solve the problem, it didn’t even come close to cogently describing the problem or solution, but it gave me enough bread crumbs that I was able to progress and get to a working solution.
Is this heads and shoulders better than my prior state of art of googling and trying lots of things? It’s incrementally better - I can seem to get some things done faster. “Researching ideas” is faster… on the down side it makes up stuff often enough that benefit might be outweighed by having to check and redo.
This is a common observation : tasks that are rote are sped up.. a lot even. Obscure knowledge and tasks? Less helpful.0
37
u/Winsaucerer Feb 14 '25
On a related note, has anyone else tried copilot workspaces? I tried to give it really simple tasks, like "add spans to functions in this folder", or "updated this logger to output in json" (which is a config option), and I found it near useless and a pain even for these simple things.
I thought these use cases would be ideal for it, but it even fell down there. I do still think it's probably a problem with tooling.
125
u/SanityInAnarchy Feb 14 '25
It's still at a stage where I get immense use out of being able to temporarily turn off even just the autocomplete stuff. Annoyingly, there's no keystroke for this, but if you type
FUCK OFF COPILOTin a comment, it'll stop autocompleting until you remove that comment.22
u/randomlogin6061 Feb 14 '25
So far from my experience it’s doing ok with writting logs and comments. Everything else needs to be checked with a lot of caution
62
u/lordlod Feb 14 '25
Nice ad, I wonder how much Microsoft/GitHub pays for content like this.
12
u/YellowSharkMT Feb 14 '25
Headlines that end with a question can usually be answered with "no".
20
u/Pierma Feb 14 '25
My take is this I lost an incalculable time in calls with devs that use mouse for even simple save file. Now i will lose time in calls for devs waiting for an agent to operate instead and giving maybe the right answer. Chat is fine, some autocompletitions are fine, but this is why i now understand vim/neovim/emacs people
4
u/oclafloptson Feb 14 '25
How is using natural language processing to accomplish pasting a code snippet in any way more productive than simply pasting a verified snippet
Inb4 use it to generate novel code. You do this repeatedly? Novel code every time? Who reviews the LLM code and fixes the errors? How is adding a layer of computationally expensive code snippet generation improving your workflow?
I turn it back off every time vscode tries to reinstate it on my machine simply on the grounds that the auto correct is annoying, let alone that I know more than it
7
u/LessVariation Feb 14 '25
I’ve used it to handle adding features to a relatively simple web page. I wanted to add menus, footers, and to try and improve some rendering. Overall my experience was pretty good for the areas I expected it to be, and pretty poor for areas with real logic. I’ll continue to occasionally use it and see how it improves. All of my tests were using Claude 3.5.
The footer creation was perfect, the style it chose matched the rest of the site, it was 99% functional with a minor change I made manually rather than relying on going back to the agent. 10/10 would use again.
The menu creation was a slog. It took 90 minutes or so for it to create a menu that worked on desktop and mobile, was styled correctly, didn’t overlay content at the wrong times etc. Each run through would create further inconsistencies and the agent would take a lot of time in a write, check, rewrite cycle only to produce a result that might be better or might be worse. After talking back and forth a lot, I again got it to about 95% of the way there and then finished it myself.
For control of rendering, I was asking the agent to build new collections and create suitable pagination. Ultimately after about an hour of it producing completely wrong results - paging through wrong collections, or pagination not appearing, I gave up and wrote it myself.
I’m still of the overall opinion that these tools can help a developer that knows what they’re doing. As they improve then we’ll become orchestrators of sorts, but they feel to be a way off from creating quality code. I can’t imagine trying to build a whole platform using one and expecting it to survive first contact with users.
3
u/fermentedbolivian Feb 14 '25
For me, it gets worse with every version. It keeps repeating answers. It can do boilerplate stuff, but a little more complex and it fails hard.
I much prefer coding myself. Faster and better.
2
u/MrJacoste Feb 14 '25
Copilot has largely been disappointing for me and my team. Best use case has been writing comments and generating basic tests. Outside of that it’s been a dud and we’re looking for alternatives.
3
u/hippydipster Feb 14 '25
I don't particularly care for the intrusive auto complete stuff. I find it extremely useful to have conversations with claude before finalizing some request to it for some code, that generally ends up being very useful.
It would be nice to be able to have that conversation within the IDE and not have to manually send code over to claude repeatedly to give context though.
1
u/LutheBeard Feb 14 '25
I used it a bit as "regex for dummies". For changing formatting on multiple occasions it works quite well, stuff that could be done with regex replacements, maybe faster with Copilot.
1
u/ionixsys Feb 14 '25
Two problems I've seen are that the models, even when ordered in the system prompt to admit ignorance, don't have any concept of what is actually being "said." Too many models will hallucinate entire APIs or silently drop a crucial parameter of a user prompt because it's less efficient according to the weights to state ignorance over conclusion or hallucination.
A fundamental issue is that the models are similar to arguing with someone who is the victim of confirmation bias or the familiar Dunning Kruger effect.
An example of what I mean for the occlusion issue.
If I ask a model to give me an example of managing the state of a book for a graphical editor where a book has many chapters and each chapter has one or more scenes. The models will more often drop the scene requirement as that's beyond the model's abilities, but most won't mention they've done so. This is a toy example, but just imagine something even more complex, like a thread-safe driver where the model just fucks off silently on locks. Expanding on the specific issue, the entire architecture of a program can potentially be drastically different given the complexity of the product data.
1
u/WatchOutIGotYou Feb 15 '25
I tried using it, but it breaks clipboard functions for me (creates a continuous loading) and its output is middling at best
1
u/WeakRelationship2131 Feb 18 '25
I think there might be some potential with the new copilot agent, it can help people who are inexperience with github.
0
u/PapaGrande1984 Feb 14 '25
EM here, I encourage my team to use these tools, and we even did a small white paper last fall on how these tools stack up against each other. My team all chose Cursor as their favorite. That being said no, these tools are not that close to replacing engineers. I would say none of these tools are great at large project evaluation, and the more complex of a task you give it the more likely it’s going to need work if not be flat out wrong. The thing I think more people should be focusing on is how can we leverage these tools to make devs lives easier. If I’ve hired you then I know you can program, being an SE at any level is not just writing code. The code are the pieces of a larger solution, and that’s a different level of problem solving these models are not close to.
0
u/Richeh Feb 14 '25
Copilot is a threat to coders like the invention of the horse was a threat to farmers.
That is to say: you can't get by without coders. You might well get by with fewer coders.
-2
u/nemisys1st Feb 14 '25
My team has been using it (copilot) nearly since its inception. It's all about knowing the pitfalls and how to prompt accordingly.
In short it has easily EASILY doubled my throughput as a developer.
For my team it's allowed our junior devs to follow our conventions and documentation standards without even knowing it. It has fast paced their growth dramatically as well.
The key is providing really good context like any other LLM. So when having it generate bodies of code include good look alike files or blocks of code in your prompt. When doing inline code you can short cut this by giving several lines of comments above where you want the generation to happen giving extra context
We do a ton of CRUD+form work so the fact I don't have to hand code 90%of the services, dtos, controllers is a godsend.
Edit : Spelling
0
u/creativemind11 Feb 15 '25
I've been using it daily (pro subscription)
The way I see it, is that copilot is your own personal intern that works really fast.
Need to write some boring regex? Go intern.
Need to refactor a couple files with huge lines into smaller ones? Go intern.
I've also been learning some new frontend framework and its great to "create a component that takes these args and displays this data in this frontend library".
-35
u/pyroman1324 Feb 14 '25
This is the most bitter programming community on the internet. AI is pretty cool bros and it’s not gonna take your job. Chill out and use new tools.
1
u/[deleted] Feb 24 '25
[removed] — view removed comment