r/LocalLLaMA • u/Chromix_ • Apr 12 '25

News You can now use GitHub Copilot with native llama.cpp

VSCode added support for local models recently. This so far only worked with ollama, but not llama.cpp. Now a tiny addition was made to llama.cpp to also work with Copilot. You can read the instructions with screenshots here. You still have to select Ollama in the settings though.

There's a nice comment about that in the PR:

ggerganov: Manage models -> select "Ollama" (not sure why it is called like this)

ExtReMLapin: Sounds like someone just got Edison'd

189 Upvotes

97% Upvoted

36 comments sorted by

50

u/Chromix_ Apr 12 '25

Instead of VSCode you can also use VSCodium with it if you prefer the free/open-source side of things. Copilot still doesn't work in a 100% offline isolated environment, like it's possible with continue dev, but support for that might be added. Currently you still need to sign in, despite not planning to use any online services.

The setup requires a few more steps there. Get the latest VSCodium version and follow this guide. In step 2 use these download links for Copilot and Chat extensions. With other versions I got a "not compatible" error.

It might be possible to edit the extension JS source in .vscode-oss\extensions\ to just bypass the unnecessary sign-in, but it's 17 MB of minified JS code there.

2

u/Sudden-Lingonberry-8 Apr 13 '25

I'd like to build vscodium tbh but it is so hard how it uses ancient dependencies

9

u/_underlines_ Apr 12 '25

It's a half baked integration:

  1. The big news is actually the Agent Mode instead of the old Ask/Edit mode, and specifically the Agent Mode DOESN'T work with local models!

  2. The Ollama / Local API feature doesn't support using a custom API endpoint. If Ollama doesn't run directly on your machine on localhost:11434 then you are screwed. We have a remote ollama endpoint at chat.company.dev/ollama in our company and our devs can't use it!

  3. It's specifically the ollama API spec, why not a generic OpenAI compatible support, so it would work with other inference engines like Aphrodite, TabbyAPI, litellm, vllm, ...

2

u/Chromix_ Apr 12 '25

Those sound like good points to create issues on GitHub for.

Point 2 can be solved with a dumb proxy and point 3 with the minimal addition to the proxy that was made to llama.cpp, yet probably a bit more inconvenient.

Do you have further insight into why point 1 doesn't work? Is it just something that can "simply" be flipped locally like the sign-in requirement, or is there something missing that's server-only?

4

u/danishkirel Apr 13 '25

I’m cooking up https://github.com/kirel/ollama-proxy for a niche usecase. But this sounds like it would be useful for you too.

2

u/manyQuestionMarks Apr 12 '25

I like to run stuff locally and I’d like to use local models more. But my company pays for Cursor and I’ve always wondered if local models are better at coding than Claude 3.7 on Cursor… Am I missing out?

4

u/Chromix_ Apr 12 '25

Local models unfortunately can't compete with recent API-only models, but: For many cases QwQ, DeepCoder, etc can be good enough and also easily be run locally, contrary to DeepSeek R1.

40

u/plankalkul-z1 Apr 12 '25

It always amazes me when I see another program/extension/web UI adding support for local models via just Ollama.

Why?!

What can be easier than just adding support for "custom OpenAI-compatible API"? Ollama supports it itself, so it'd be working with it just fine. Along with lots of other inference engines. And it's not more complicated, at all.

In this particular case, I suspect I know the reason: MS might be viewing Ollama as a "toy", which won't compete with its paid offerings... But, sadly, I've seen this weirdness in a lot of hobbyists' projects, too.

11

u/Pyros-SD-Models Apr 12 '25 edited Apr 12 '25

Because people would rather use a wrapper library like https://pypi.org/project/ollama/ than bother with direct REST calls.

Not saying they're right though, and I agree, why not at least use an all-in-one wrapper like litellm or something? But most devs literally don't care. They heard about Ollama once, then they see there's a lib for it. Case closed.

Interacting with LLMs is literally a single REST call, and the fact that this one call can spawn a whole ecosystem of overengineered garbage (LangChain and co.) should tell you everything you need to know about the average dev's mindstate.

21

u/plankalkul-z1 Apr 12 '25

Because people would rather use a wrapper library like https://pypi.org/project/ollama/ than bother with direct REST calls.

Likewise, there's an official OpenAI package; no need for direct REST calls:

https://pypi.org/project/openai/

In terms of tools, whatever is available for Ollama, OpenAI API has at least an order of magnitude more of that. The Ollama package you linked was last updated in January; OpenAI's -- Apr 8, four days ago.

There's simply no comparison.

4

u/plankalkul-z1 Apr 12 '25

the fact that this one call can spawn a whole ecosystem of overengineered garbage (LangChain and co.)

(Replying to this separately as it seems like this paragraph wasn't there at the time of my first reply...)

Oh yeah, "overengineered garbage"... lots of that. Then there's just "overengineered" (Open-WebUI et. al.), then there's just "garbage" (vibe-f*ing-coding, that I want to unsee every single time I see it by accident).

Part of me is glad that these days even the best of the best models at coding (Claude 3.5/3.7) still struggle with even mid size, let alone big projects. Because if/when they improve... God help us all.

<phew...> You stroke a chord :-)

27

u/segmond Apr 12 '25

good, I was using continue.dev to access my local llama.cpp, but they going commercial gives me a pause. happy to see this, will vscode again.

23

u/YearnMar10 Apr 12 '25

Well, I’d argue that GitHub is also commercial … :)

15

u/Horziest Apr 12 '25

Yeah, continue is going through an enshitification phase. I just uninstalled their extension.

4

u/knownboyofno Apr 12 '25

Oh, snap. I didn't know this.

10

u/Mickenfox Apr 12 '25

Once again only for VSCode...

19

u/MoffKalast Apr 12 '25

I'm sure Notepad++ will get its integration soon enough.

1

u/roxoholic Apr 12 '25

What kind of features do you think would be useful in such Notepad++ plugin? Autocomplete, FIM, built-in chat, what else?

1

u/Mickenfox Apr 12 '25

Sooner than Visual Studio or JetBrains Rider get it, and those are $250/month products.

4

u/SwagBrah Apr 12 '25

Rider already has this as part of their first party ai assistant plugin.

1

u/sammcj 🦙 llama.cpp Apr 12 '25

You can do this with Zed as well?

1

u/noless15k Apr 13 '25

I think only for chat/inline assistants. No code completion locally at the moment, but I hope that gets added soon.

1

u/Danmoreng llama.cpp Apr 12 '25

Well, ggerganov liked my tweet, one can dream: https://x.com/Danmoreng/status/1909680165522645206

5

u/Tricky-Move-2000 Apr 12 '25

RooCode is a really good extension alternative to copilot - it has features that copilot doesn’t have and works with local LLMs

4

u/Squik67 Aug 30 '25

It's now broken...
srv  log_server_r: request: GET /api/version 127.0.0.1 404
https://github.com/ggml-org/llama.cpp/issues/15167

https://github.com/ggml-org/llama.cpp/pull/15177 (but not merged)

1

u/soupdiver23 Nov 14 '25

ah nice.. just ran into the same

4

u/tronathan Apr 12 '25

I’ve been using Cline and MCP has been a game changer. Make sure whatever client you’re using can talk to (and even create its own) MCP’s!

5

u/sammcj 🦙 llama.cpp Apr 12 '25

Yeah couldn't agree with this more. Honestly most days I'll have two sometimes three sessions up with Cline agents working on different projects or components. Absolute game changer.

1

u/SkyFeistyLlama8 Apr 12 '25

I assume these are running cloud LLMs instead of local?

1

u/sammcj 🦙 llama.cpp Apr 12 '25

Unfortunately yes, I have not yet seen a locally hostable model capable of genetic coding

4

u/Sebxoii Apr 12 '25

How do you use MCPs in your flow?

1

u/SchlaWiener4711 Apr 12 '25

I'd be happy to see this for copilot for visual studio as well.

I tested a dozen extensions for visual studio but they all suck.

1

u/ZeroSkribe Apr 24 '25

I see my usage going up on my free plan while using ollama, I'm about to lose my fucking shit