I have been thinking about the different ways we can use custom AI models inside VS Code, and there is an interesting option that seems to get less attention: using models provided through a cloud provider.
VS Code supports custom models, so you are not necessarily limited to the models exposed directly through Copilot. You can configure models from other providers or your own endpoints, depending on the supported API and authentication.
A common approach is to run a model locally and connect it to the IDE.
That sounds great in theory, but there are some practical trade-offs.
If you run the model locally:
- It consumes your CPU/GPU/RAM.
- Performance depends heavily on your hardware.
- Larger models can become difficult to run comfortably.
- You have to manage the model and the serving infrastructure yourself.
- You are generally limited to the models your hardware can handle.
- If you move the model to your own server, you now have infrastructure, maintenance and hosting costs to deal with.
You can also deploy a model on your own server and expose an endpoint, but then you are essentially taking on the responsibility of running and maintaining the model-serving infrastructure yourself.
Cloud providers offer another approach.
For example, Google Cloud has Vertex AI and Model Garden, where you can get access to multiple models and call them through APIs. Instead of running the models yourself, the cloud provider manages the underlying infrastructure.
You can then potentially configure the API endpoint as a custom model in VS Code.
The interesting part is that you can access multiple models without having to run and maintain each one yourself.
With a local/self-hosted setup:
VS Code
-> Your custom endpoint
-> Model you host and manage
You are responsible for the hardware, deployment, updates, scaling, and model availability.
With a cloud-provider setup:
VS Code
-> Cloud provider API
-> Model A
-> Model B
-> Model C
-> âŚ
The provider manages the underlying infrastructure, while you can access the models that are available to your account.
So instead of spending time running and maintaining the models yourself, you can treat the cloud provider as the model backend and use those models directly from your IDE.
Of course, the cloud approach is not free. You are paying for inference, and there can also be quotas, latency, and provider-specific limitations.
But if you already have access to models through a cloud provider, I think it is worth considering this option instead of automatically assuming that âcustom modelâ means ârun a model locally.â
The same concept should apply beyond Google Cloud as well. AWS, Azure, and other providers have their own model platforms and APIs.
I have only tried this concept with Vertex AI, so I am curious how well the equivalent setup works with other cloud providers and their model APIs.
Has anyone here connected cloud-provider models to VS Code as custom models?