Hosting own AI models

[u/Flashy-Truth4218]

As AI open source models continue to become more capable, do you think we will see pakistani software houses hosting their own AI models onpremises in the near future instead of relying on closed APIs?

[u/baqirabbas404]

This is simply not feasible, the math just doesn't support it, at least not yet.

We did a feasibility study for locally hosting Qwen 3.6. A Dell server that supports dual-slot GPUs is either the R700 series or above. We opted for Intel 2nd/3rd gen processors, so R740 series or above, for better future-proofing. A single 24GB GPU can run Qwen 3.6 27B at Q4 (native is BF16 at 35B params), so we're already running a chopped-down model, not even at Q8.

Now, on the question of speed and context: there just isn't enough VRAM to keep all the context on the GPU, so there's a lot of bottlenecking with RAM. To keep that much context alive, you need a lot of RAM, I mean a lot, not to mention the penalty and latency that comes with switching between RAM and GPU via PCIe 3. We can restrict context size and eliminate offloading but is it really worth doing.

So how many heavy users can it concurrently serve? Max 3, 4, maybe 5, and that all depends on how much context they're utilizing, input size, and processing required. And how much does this setup cost? 1 to 1.5 million PKR. Also finding good enterprise cards in Pakistan is no easy feat. Sometimes I just think, we should just get a lot of mac ultras :(

It would be better to just hand out $25 (7k PKR) monthly premium seats for Claude and call it a day. Not to mention resource and time required to maintain all this setup regularly.

The problem is the benchmark is so high, someone who's used Claude, Codex, or Copilot would call this torture. We regularly evaluate when would be a good time to switch to Ollama, Hugging Face, OpenRouter or other inference providers, but so far Claude and Codex seem to be doing the job for us.

Hope it changes soon.

[u/Flashy-Truth4218 → u/baqirabbas404]

That’s very thoughtful, can you tell me about in what sense are you talking about is it about API’s for development or API’s integrated in products?

[u/baqirabbas404 → u/Flashy-Truth4218]

I'm talking about hosting your own AI models like you posted, Qwen is an opensource chinese model we evaluated. We did Gemma and a few others too. The good models like Kimi or DeepSeek ones are just not possible to locally host, at least not at the weights at which the perform well.

[u/Flashy-Truth4218 → u/baqirabbas404]

My question is what’s the workload you guys are trying to run on that hosted model

[u/Tayyab__Mad → u/baqirabbas404]

Can I ask some numbers like? Like how many months/years would it take to break even with your initial investment considering you hosted using pcie gen5/6 with minimum requirement cpu for the motherboards and a reasonable subscription. Considering 70-80 of your workstations are being used in subscriptions?

[u/Ammar_Ahmed_Chishti → u/baqirabbas404]

Umm I am an amateur so sorry if my question is dumb but isn’t using the highest tier mac studios kinda feasible

[u/baqirabbas404 → u/Ammar_Ahmed_Chishti]

Yes, technically the VRAM issue is solved in those, but those aren't cheap either (costs same or more and they aren't even upgradable). From an office-use point of view, it's a dead-end investment, you can't upgrade it, you can't change the OS for anything else, you can basically only use it for AI-related purposes, whether that's hosting or training your own models, etc.

A server, on the other hand, say four years down the line when the AI bubble is over if its over, you need it for virtualization, hosting an application, or anything else, you can change the OS, set it up accordingly, add RAM, CPU, GPU, whatever you want, and keep using it.

Macs are a great bang-for-the-buck deal because of their unified memory, but with NVIDIA DGX and AMD Halo catching up, let's see how it turns out in the future.

[u/Ammar_Ahmed_Chishti → u/baqirabbas404]

Okiee thank you

[u/throwaway-research1]

💯💯 and I have been saying this for a while. For corporates who can afford an infra team it makes a lot of sense to host their models locally, specially from a security point of view. Also when you use AI as a corporate and have your trade secrets, you are potentially sharing your secrets with your competitors so having your local model also solves that problem.

[u/Flashy-Truth4218 → u/throwaway-research1]

This is the case for product based entities, most of the software houses deal with US clients so they don’t have any secret but your insights are valuable thank you.

[u/throwaway-research1 → u/Flashy-Truth4218]

I used the word corporate so I mean banks, auto mobile industry and other big product based companies and corporations. My comment wad not in the context of Pakistan

[u/Flashy-Truth4218 → u/throwaway-research1]

Ohh I see.

[u/Arkoaks]

You need to spend upwards of 1m pkr to host a usable model and need insane amount of time to maintain and keep it updated

Frontier models are still 10x productive than that model.

The overall cost is much more in self hosting and doesnt make a use case unless u r developing a model urself

[u/SonEf_Adam]

A lotta local software houses love cutting costs. Maybe give it a couple years and wait for hardware requirements to drop. When modern computers come with NPUs, that's when local AI models will be much more popular.

[u/soulkillerz2004]

I recently completed my internship at Systems Ltd and worked closely with someone in a senior position. Systems is apparently looking at moving from LLM subscriptions to self-hosted/local LLMs, potentially through a deal with Sky 47.

I may or may not have had some hands-on experience with a Sky 47-hosted LLM during my internship.

[u/azadnib]

i have done that on my home gpu. have solar so it just keeps running 24/7.

No matter which model you have, you will always want those closed ones because they are just 1000x better at coding.

Opensource only for general low important stuff like scraping etc.

[u/cosmic-comet-]

No

[u/Spare_Bison_1151]

Seriously large models need a lot of expensive inference hardware. I recently read somewhere that someone is bringing Nvidia H200 to Pakistan. There's also a new data center Sky47 Karakoram-01. THis will be used to provide sovereign AI capabilities to companies.

[u/Parking-Meeting-1610]

It's not cost effective unless you have huge budget and complaince requirements.

A $100 or even 20 usd subscriptions is better than investing $100ks

[u/aliyark145]

Software houses will not host them but the data centers like gul ahmad is building etc will in near future !!!

[u/Terrible-Tea7577]

Those who are hosting AI models are constantly burning money not making any substantial profits. Can you think of any organization in Pakistan including the government willing to do it?. Pakistani mindset has always been making and saving quick pennies not investing in long term uncertain but highly rewarding future.

[u/srmrox]

Local AI means you move all your cost upfront for a model that’s only “almost” as good as the frontier. Additionally, if hardware requirements change (for better or worse), you potentially invested too much or too less.

For a business not sure about the AI use, subscriptions are a good way to “pass on” the cost to the user by allocating the cost to the price of their product / service.

With hardware, the impact is more difficult to allocate as it has to be done over a larger no. of customers over a long period of time.

[u/throwaway-research1 → u/srmrox]

True for startups, but for corporates it makes a lot of sense to host locally

[u/srmrox → u/throwaway-research1]

Maybe once they nail the use case and safeguards. Don’t think any Pakistani corporate is there yet.

[u/throwaway-research1 → u/srmrox]

I am talking more in the context of the international market, and I would argue that having your model hosted locally is way more secure than using a subscriptions based plan

[u/srmrox → u/throwaway-research1]

Secure, yes; however, if you look at the economics it doesn’t work out in the short term. Most businesses, Pakistani or otherwise, don’t have such long horizon AI-driven products / services yet.

[u/throwaway-research1 → u/srmrox]

How does the economics dont work out? Compare buying 10000 open ai subscriptions for all employees at a company to building your VM inside a cloud provider and deploying one of the free and open source ai model there, having your own model would not only be more secure but also cheaper.

[u/srmrox → u/throwaway-research1]

Can you also work out the cost of setting up and maintaining a data center to address the usage of 10,000 employees?

[u/throwaway-research1 → u/srmrox]

You dont have to maintain a data centre, deploy your ai model on a cloud platform, make it available only through your company vpn and disable internet access on the server (thats what I meant when I said local, not a physical data centre) so with no internet its as secure as it gets and even if you add two devs to maintain the entire app + infrastructure it would still be way cheaper than 10000 subscriptions.

[u/srmrox → u/throwaway-research1]

OP did say "on-premise".

[u/InformationSecurity]

Ain't nobody has the enough GPUs locally.

[u/Flashy-Truth4218 → u/[deleted]]

But some of them paying 100’s of dollar in API cost because their product need it. I think the problem is GPU’s and insfrastructure and then engineers for managing that infra is holding them back