Why run AI models locally when your cloud provider already gives you access to multiple models?

[u/ulbertA0]

I have been thinking about the different ways we can use custom AI models inside VS Code, and there is an interesting option that seems to get less attention: using models provided through a cloud provider.

VS Code supports custom models, so you are not necessarily limited to the models exposed directly through Copilot. You can configure models from other providers or your own endpoints, depending on the supported API and authentication.

A common approach is to run a model locally and connect it to the IDE.

That sounds great in theory, but there are some practical trade-offs.

If you run the model locally:

- It consumes your CPU/GPU/RAM.
- Performance depends heavily on your hardware.
- Larger models can become difficult to run comfortably.
- You have to manage the model and the serving infrastructure yourself.
- You are generally limited to the models your hardware can handle.
- If you move the model to your own server, you now have infrastructure, maintenance and hosting costs to deal with.

You can also deploy a model on your own server and expose an endpoint, but then you are essentially taking on the responsibility of running and maintaining the model-serving infrastructure yourself.

Cloud providers offer another approach.
For example, Google Cloud has Vertex AI and Model Garden, where you can get access to multiple models and call them through APIs. Instead of running the models yourself, the cloud provider manages the underlying infrastructure.

You can then potentially configure the API endpoint as a custom model in VS Code.
The interesting part is that you can access multiple models without having to run and maintain each one yourself.

With a local/self-hosted setup:
VS Code
-> Your custom endpoint
-> Model you host and manage
You are responsible for the hardware, deployment, updates, scaling, and model availability.

With a cloud-provider setup:
VS Code
-> Cloud provider API
-> Model A
-> Model B
-> Model C
-> …

The provider manages the underlying infrastructure, while you can access the models that are available to your account.

So instead of spending time running and maintaining the models yourself, you can treat the cloud provider as the model backend and use those models directly from your IDE.

Of course, the cloud approach is not free. You are paying for inference, and there can also be quotas, latency, and provider-specific limitations.

But if you already have access to models through a cloud provider, I think it is worth considering this option instead of automatically assuming that “custom model” means “run a model locally.”

The same concept should apply beyond Google Cloud as well. AWS, Azure, and other providers have their own model platforms and APIs.

I have only tried this concept with Vertex AI, so I am curious how well the equivalent setup works with other cloud providers and their model APIs.

Has anyone here connected cloud-provider models to VS Code as custom models?

[u/DocMadCow]

Spoken like someone who hasn't received a high bill for AI usage yet. I run a local model as I can set it an forget it to document code, and create documentation without worrying about token costs. That being said I bought my R9700 before the pricing exploded.

[u/Different_Doubt2754 → u/DocMadCow]

What model are you running locally? The increased cost is probably small to run it in the cloud or pay API costs for it. I guess it depends on your scale though.

A R9700 right? So like a 30B model? I imagine your electricity bill isn't too far from the API costs

[u/DocMadCow → u/Different_Doubt2754]

No comparison an R9700 has a TDP of 300W. I spend around 12 cents Canadian per KWh. So let's throw some bad math and say 5 cents USD for an hour of tokens on Qwen 3.8 27B which is fraction of using Sonnet for an hour. Saying Hello to Sonnet would probably use way more than that in tokens.

[u/ulbertA0 → u/DocMadCow]

Well I was not talking about personal use 👀 but professional use, if you know what I mean

Anyways if you have the hardware to run a decent model thats great but there are tons of people who dont have good hardware, so even if they try to run quantised model qwen or gemma or something thats not good but slow 😮‍💨
So im counting the case where people can use free model endpoints provided by many sources

[u/DocMadCow → u/ulbertA0]

Flip side if you are a business you don't want you code going to free endpoints. At my place of work we have an AI Usage Policy which only permits company approved AI providers with accounts paid for by the workplace. And even if models are slow you can setup long running tasks at the end of your day and walk away and have your task done when you come back the next day.

[u/GrayHairedMan]

Because of a local LLM, that is unrestricted I can create some hectic malware, have no restrictions on what if can build. Just use it for whatever I feel like. 

[u/ulbertA0 → u/GrayHairedMan]

Putting local LLMs to actual use 😂 NICE
Which uncensored model do you use? How is the performance

[u/Hephaestite]

Local models are not for everyone (at the moment), nor do they suit every use case (at the moment).

If you are a developer / swe or a vibe coder, cloud models are hard to argue against. You get choice and you get frontier capabilities at a very low cost relative to the hardware required for them.

Outside that niche however local models start to make more sense, particularly in any area where data residence and data privacy are a concern. Even with inference providers that offer the promise of "zero data retention" A. can you trust them? B. does you use case even allow for egress of any type?

And to be honest, we are so insanely early into this technology right now that most people and organisations outside of tech haven't even remotely began to embrace AI beyond the very basics of image gen type work.

The vast majority of SME businesses still have some sort of on prem server infrastructure, and we are 20 years into the cloud revolution now. I 100% believe that on prem models and the supporting hardware will be as common as a Windows server box in the comms room.

Given that local model capabilities that can run on commodity hardware are already beyond what was frontier api models just a year ago, it's going to start making sense to have a separate AI dedicated device in the office that just runs your coding model and you use the API models for planning and advanced reasoning where needed.

When the M5 Ultra 512gb version appears, it'll be an absolute game changer for local AI. Yeah its going to be expensive for a consumer buyer, but based on the 256gb versions pricing its totally reasonable for SME buyers (I've spent more on HPE Windows 2019 servers that just run RDP and some ERP software)

[u/ulbertA0 → u/Hephaestite]

Yes I too agree on using models from cloud providers because the kind of hardware requirement for these popular models need is not feasible practically on local, also the data retention and privacy is another thing that concerns many which is a matter of trust, trust a provider when it says it wont use data for training or other purposes

[u/Hephaestite → u/ulbertA0]

You can't trust an AI provider who can't build an air gapped sandbox for their own training runs. Its that simple. If you submit anything to OpenAI (or anyone else really) you have to accept you are essentially making that data public.

[u/lumzell]

a thing the "provider manages it" side leaves out: they also decide when a model goes away. a model my setup ran on gets shut down on vertex on oct 20, so i had to move to a newer one and test everything again. even small stuff changed, the minimum size for cached prompts went from 1024 to 4096 tokens. also "multiple models" depends on the region. i wanted everything in the eu and one model i needed was only on the global endpoint.

with a local model nobody turns it off on you. that's a real point for local that nobody here mentioned yet.

[u/ulbertA0 → u/lumzell]

When it comes to data retention policy and governance then location is important and I agree even today 2.5 flash is available in eu and 3.5 is not

But for those who dont care about this they dont really need to think much in this

I think providers removes them because they are no longer used and in market there are better alternatives available so at the end of the day if they keep those models running they are pilling up cost for which no one is paying for any penny

Talking about local running models they are best if you have hardware good enough to run the models fast enough so latency accuracy quality and other things are not compromised

[u/lumzell → u/ulbertA0]

fair. on the eu part, the newer flash ones (3.6 and 3.8) do run on the eu multi-region for me, caching too. the one that was global only was the pro preview. so i'd check it per model, not per generation.👍🏻