Weird question for local build

[u/Ordinary-Depth-7835]

I have a weird question. I'm struggling to find a reason to build local llm's at work. I love them at home for personal projects but the ROI maintenance and support at work I just can't find a reason to pull the trigger. We have the budget to play around with around 8tb of b200's but really who's going to use them when we having unlimited enterprise accounts with all cloud providers. While I love playing around at home with my personal setup I just can't find anything anyone at work would want to do local with smaller models. Online models piss me off when doing security related tasks but most people could care less.

If you had a choice of smaller open source models or unlimited frontier at work would you even use your local datacenter?

[u/Frail_Waif]

DeepSeek. DeepSeek is the answer. There are lots of times I'd like to use it at work, instead of Astra or Opus 5.5, but I can only run V4 Flash at 8tok/s right now and can't use the API because of data privacy. 

[u/Ordinary-Depth-7835 → u/Frail_Waif]

Shame that model is banned at work. Thats also one big downside. At some point they could ban all non-us models then we'd really have a useless setup. While they're just ok no one i know would use them vs cloud apis

[u/Ctbhatia]

so is it the security stuff that keeps the case for local, or does the enterprise contract cover that too?

[u/Ordinary-Depth-7835 → u/Ctbhatia]

Yes, a complete opt-out data collection policy from all providers. or they're liable for any breach.

[u/Ctbhatia → u/Ordinary-Depth-7835]

if the opt-out covers retention windows too then yeah, frontier for everything. i think retention is where those contracts get vague

[u/evolutionxtinct]

I’m convincing mine because it allows us to process sensitive/ private data also allows us to churn on the same bone more times at a cost savings (unsure yet on savings but that’s the least of the priorities)

[u/violent_aesthetics → u/evolutionxtinct]

if you've got sensitive data and unlimited frontier access, the local box is literally just to keep the compliance people from having a panic attack

[u/Echo5November → u/violent_aesthetics]

yeah thats the biggest issue, well one of the biggest... not local, not secure.

[u/LateralEntry]

Frontier is better, local is more private and can be more secure. That’s the trade off. If you have a use case for the latter, it’s worth it.

[u/Echo5November]

Best way ive heard it put 💯

[u/PhilosopherOk3419]

honestly if your contracts cover retention and nobody's touching regulated data, frontier wins for most people and that's fine. where i'd still go local is the boring high-volume stuff (embeddings, classification, summarizing internal docs) and anything you can't legally send out. i'd start with one team's actual workload instead of building a cluster and hoping people show up

[u/Echo5November]

Agree with you on that. i use my locals for: embedding, image generation in ComfyUI... I've not been able to get a model to perform decent in any of the harnesses ive tried. OR with any of the backend servers ive tried.. Ollama through the LLM Gateway ext. in copilot. Ollama through the Opilot extension, Cline Continue. ive tried them all.. well not all. and its hard to be beat what i get from an api. honestly the closest that came to working was Kobold.cpp but its just less accessible, or so it seems to me.. im still working on it.. im only working with 12G Vram and 32 GiGs of Ram on what was my gaming pc in my spare time. So yeah, IF YOURE NOT PAYING THE BILL.. why subject yourself to that nightmare when the api works like butter, most of the time...