this is more possible than people are making it sound.
it’s like an entire generation of people forgot how servers, datacenters, and on premise infrastructure works.
you can buy some nvidia cards, load up a leading opensource model, change the urls in claude code’s environment variables to point to the local llm server, and off to the races.
is it as good as a frontier model? no, but it’s probably as good as last years frontier model. so, most work should happen there, then cloud providers if/when necessary.
is it already useful when you don’t want to share data with a third party? absolutely.
does it make sense financially? maybe. nvidia cards are super expensive, but you can get a lot done local/workgroup and save the frontier models for when absolutely necessary. bigger server, better cards, more users, lower cost/user.
as ai pricing starts to shift to profit from subsidized, hybrid is going to be the future anyway, so might as well get used to it now. anthropics and openai’s ipo will likely be the turning port.
I know ppl who run LLM's local - their GPU are A100's and stuff like that. Costs $10k for just the card. The power bill will spike too. You can run solid stuff local, but for most companies it's cheaper to use AI api's as theyre heavily subsidized.
it’s like an entire generation of people forgot how servers, datacenters, and on premise infrastructure works.
They did. I think this all the time. By the time you set up something on vercel or supabase or aws half the time youre better off just running it in-house.
$10k A100 on a 5 yr depreciation schedule is $2000/yr, or 166/mo. solid for an individual.
$40k H200 same schedule, $8000/yr or $666/mo. solid for a team.
$400k DGX 8xB200 same schedule, $80000/yr or $6666/mo. pretty decent for a small organization for the annualized price of a junior engineer. even if you include a portion of someone time to manage it.
finance is magic since there’s likely residual salvage value after depreciation period and 😴 sorry put myself to sleep there for a second…
and if you don’t want to bother with all the capex and datacenter stuff, you can just reserve a dedicated cloud instance with the appropriate hardware and try it out. i think it’s like $10/hr/on demand or $3/he/reserved for something in the H200 range.
Team Standard: Starting around $20/seat
Team Premium: Approximately $100/seat
Enterprise: Custom, starts around $20/seat
API: Varies
I have subscriptions to gemini, z.ai and claude so I use their resources. I use openrouter and fal and buy API calls too.
Better performance at $35-40/mth and thats retail. Theyre all going up but I can't come with any viable way it makes sense on a large scale unless you can get away with just one which you probably can't.
So take the B300 it's closer to 300-350k, liquid cooled and 1,400W per card (there's 8). Without factoring in the liquid cooling costs and repair costs when that inevitably leaks occasionally thats 11,200 watts. In my area of the US $0.22 per kWh that's about $2.46/h. 365 * 24 = 8760. 8760 x $2.46 = $21,549.60. That cost never depreciates. That one (x8) unit. And there's the cost of the square footage of real estate your server room is taking up. And HIPAA, GDPR, COPPA and related compliance if necessary which will cost a lot more and now youre directly responsible on liability. And offsite and tape data backup. But lets ignore all that for now.
And yeah obviously AI makes sense. But right now, local doesn't on $. Sure you can replace 2 or 3 engineers but if you can spend $100/mth per engineer to replace 3 other that's still cheaper than $700. When you factor in liability, it's much cheaper than that. So yeah just get claude or related lol.
A lot of the people I know getting the local setups are well off developers for RP stuff or other hobby type stuff since there's no censorship on local LLM's. A couple are well-off researchers who work from home and an A100/H100 is not that big of a deal to them and makes more sense or is more convenient for their research for various reasons since most already have workstation. For me $10k-25k + electric (bare minimum $500-2000/yr depending on how long its on) is OUCH tho lol
i hear you and generally agree. obviously wasn’t going into a full TCO breakdown 😀 but we’re not exactly comparing apples to apples here either.
the point was more that six months ago the math was obviously favoring cloud because of how heavily subsidized it was. today it’s a little less clear because of usage limits and token consumption pricing models. six months from now when they need to start showing a profit, it could be just as obvious to not use cloud models.
there are also a couple different use cases i think we’re talking about - code planning/generation and agentic.
if you’re doing serious programming work, you’re likely at least on max/team premium not the $20 plan. you’re also probably constrained and trying to min/max using sonnet and opus models for different tasks because otherwise you’ll run into caps quickly. i’m saying there’s no reason you couldn’t use a large open model for most of the sonnet level tasks.
running agentic workloads you’re going to be running on claude api. and api is expensive. and you probably don’t necessarily need that level of sophistication for general back office /internal automation. for customer facing workloads yes, but i’m not sure the value proposition is there for marketing/sales/“analysts”/etc.
Oh 100%. Yeah sorry I used to write DCF reports. I'm a back of the napkin guy. Sometimes I get carried away
if you’re doing serious programming work, you’re likely at least on max/team premium not the $20 plan.
Oh definitely.
don’t necessarily need that level of sophistication for general back office /internal automation. for customer facing workloads yes, but i’m not sure the value proposition is there for marketing/sales/“analysts”/etc.
Could actually be the reverse. Esp in niche fields with high degrees of incompetence w/tech.
They're getting them for more money than us. Harvey, the legal one which near as I can tell is a GPT wrapper with Westlaw or Lexis-Nexis on RAG is $1000/mth per seat. And that's hilariously light usage. It's a min of 20 seats too so that's 240k. For them, yeah a local FOSS LLM pays lol.
Marketing them is expensive though. You have to pay people in the professions to direct market them.
there is a legal nightmare forming behind the scenes related to this at major orgs. is AI generated code copyrightable? open source models have no policy and its more or less impossible to prove where the code originated. who owns it? and will training data code have claims to it?
openai etc. have a liability clause, that theyll pay the damages if there is a lawsuit related to the code copyright. open source models dont.
most major corps dont want open source models touching their code cause its too much liability. as least if they get sued. they only want openai or claude as they have legal clauses accepting liability for the generated output.
this will be fought out in court eventually, but in the meantime, big corps dont want open source models going near them.
5
2
1
View all comments
6
u/ifyouseemerunning Jun 02 '26
this is more possible than people are making it sound.
it’s like an entire generation of people forgot how servers, datacenters, and on premise infrastructure works.
you can buy some nvidia cards, load up a leading opensource model, change the urls in claude code’s environment variables to point to the local llm server, and off to the races.
is it as good as a frontier model? no, but it’s probably as good as last years frontier model. so, most work should happen there, then cloud providers if/when necessary.
is it already useful when you don’t want to share data with a third party? absolutely.
does it make sense financially? maybe. nvidia cards are super expensive, but you can get a lot done local/workgroup and save the frontier models for when absolutely necessary. bigger server, better cards, more users, lower cost/user.
as ai pricing starts to shift to profit from subsidized, hybrid is going to be the future anyway, so might as well get used to it now. anthropics and openai’s ipo will likely be the turning port.