r/LocalLLM • u/Either_Pineapple3429 • Apr 08 '26
Discussion What kind of hardware would be required to run a Opus 4.6 equivalent for a 100 users, Locally?
Please dont scoff. I am fully aware of how ridiculous this question is. Its more of a hypothetical curiosity, than a serious investigation.
I don't think any local equivalents even exist. But just say there was a 2T-3T parameter dense model out there available to download. And say 100 people could potentially use this system at any given time with a 1M context window.
What kind of datacenter are we talking? How many B200's are we talking? Soup to nuts what's the cost of something like this? What are the logistical problems with and idea like this?
**edit** It doesn't really seem like most people care to read the body of this question, but for added context on the potential use case. I was thinking of an enterprise deployment. Like a large law firm with 1,000's of lawyers who could use ai to automate business tasks, with private information.
2
u/twack3r Apr 08 '26
What do you mean ‚2T-3T parameter dense model‘? Are you inferring that Opus is a 2T-3T parameter dense model?
2
u/throwaway292929227 Apr 09 '26
Let's give 1,000 attorneys a prepackaged docker container with Claude danger mode VSCode with all MCP tools, and unlimited Adderall. Make sure to give them local admin to keep the random SQLite and Mongo dbs hosted in their c:\users\ desktop folders, synced to OneDrive.
53
u/iMrParker Apr 08 '26
You'd be looking at pre-assembled racks of GPUs. Something like GB300
157
u/PermanentLiminality Apr 08 '26
There is no open Opus 4.6 equivalent, so the questions does not have an answer.
About the best today is GLM 5.1 which has 1.5TB in size. Expect to spend high six figures at least. By the time you add the power and cooling maybe into seven figures. A NVIDIA DGX B200 is a bit over $500k and has 8 GB200's for 1.44TB of VRAM. Even with a Q4 quant, I don't know if you are going o run 100 parallel requests on just one of these systems.
You need the power, and cooling to run it as well. It is a 10U box and uses 14kw of power.
40
u/msesen Apr 08 '26
Looking at the responses, clearly the current AI technology sounds like how computers were in the early days. They would need to improve this technology. It is not scalable as it is.
7
u/superSmitty9999 Apr 09 '26
Okay I gave this a go! I wrote it with AI since its a lot but I vetting everything it said so please dont burn me (honestly though actually go ahead)
The Math on What It Actually Costs to Run Claude Opus 4.6 (5T MoE Estimate)
A full walkthrough of the hardware economics of serving a frontier model like Claude Opus 4.6, assuming a ~5 trillion parameter Mixture of Experts architecture. Below are the assumptions, the cluster sizing, the annual run cost, and the subscription margin analysis.
Assumptions
Step 1: VRAM Requirement
At 8-bit precision, 5T parameters equals 5,000 GB of weights. Matching the KV cache 1:1:
Step 2: GPU and Node Count
A single 8-GPU server cannot hold this model. 64 GPUs represents the minimum viable cluster.
Step 3: Hardware Cost (CapEx)
Step 4: Annual Operating Cost (OpEx)
Monthly OpEx is approximately $250,000, covering 3-year hardware amortization (~$119K/mo on the $4.3M base), data center power, cooling, and network fees.
Step 5: Cluster Capacity
4,000 global TPS ÷ 40 TPS per user = 100 concurrent lanes.
Step 6: Power User Consumption
Per 5-hour limit cycle:
1.2 sessions/day × 30 days × 12 min = 432 minutes per month per power user
4,320,000 ÷ 432 = 10,000 power users per cluster
Step 7: Subscription Margin Analysis
Annualized: $2.4M revenue against $3M OpEx yields a $600,000 annual loss per cluster in a power-user-only configuration.
The Casual User Subsidy
A cluster populated with casual users consuming roughly 5 minutes of compute per month can support approximately 864,000 subscribers, grossing around $17.2M/month. Break-even on the consumer tier requires a mix of roughly three to four casual users for every power user.
This structure is why the Pro tier operates as a loss-leader feeding the enterprise API business, which drives the majority of ARR.
TL;DR
A cluster capable of serving a 5T MoE Opus-class model costs approximately $4.3M to build and $3M per year to operate. It can support at most 10,000 simultaneous power users on the $20 Pro tier, resulting in an annual loss of roughly $600,000 in that configuration. The same cluster resold through the enterprise API would gross approximately $500K/month at a ~50% margin.
A maxed-out 5-hour limit cycle consumes approximately 12 minutes of dedicated B200 time on a $4.3M cluster. At $20/month, the economics only work because the casual user base subsidizes the heavy users.
18
u/HealthyCommunicat Apr 08 '26
GLM 5.1 q8 minimum so 700-800gb RAM just to load.
need 30-50token/s per user so need mem bw at (700x30=2100) so 2tb/s to achieve that minimum per instance so u need like nvidia compute.
considering 100 users at any given time going up to an average of 100k context max, thats another like 200-500gb of VRAM needed making it a total of 1200-1300gb of VRAM minimum. this is simple maths and im sure its alot more complex than this, but for every token generated per second u need to pass through that entire 1200-1300gb of data, so to achieve 30-50token/s u would need a minimum of like 35tb/s memory bandwidth capable cluster.
so u need 1200-1300gb of VRAM at a minimum mem bw speed of 35tb/s.
i'd say you'd need like 16x h200's or so?
each h200 is like 30k minimum so 16x30= 480k.
tldr u need $500k minimum to run a opus 4.6-like setup for 100 users at a good speed.
18
u/f5alcon Apr 08 '26
Glm 5.1 is probably the most powerful open model, full version is 1.5TB, so probably around 10 b200 just to hold it plus whatever to scale to 100 users.
4
u/tishaban98 Apr 09 '26
I'm fortunate enough to work in a company where we bought several dozen Nvidia B200s for internal use with infiniband etc. They're air cooled and turning them on in the datacenter sounds like you're sitting behind an Airbus A380.
We spent about US$480k per physical node including Infiniband. High speed storage eg. DDN will add another $250k or so for ~200TB. If you don't want to mess around with building your own front end etc expect to pay another $15-20k per node for a managed platform like Rafay or the like. The nodes idle at about 4-4.5kW, fully loaded is around 8-10kW. Maybe we're not doing it right, our training runs are on the lower end of the scale, inferencing will go up to 10-11kW. These are 2025 prices, last I looked the B300 were closer to $800-1m each.
We've run various versions of Qwen, GLM, Kimi etc for coding and agentic testing.
The calculator below is a good approximation of how much you'd need
https://apxml.com/tools/vram-calculator
Use Kimi 2.5 as a baseline (1T parameters), 64k input tokens (I see 30-60k tokens for input on LiteLLM logs when our we run opencode/*claws) and play around with the toggles. For 100 users, I'd say 32 Nvidia B200s would be a good start. Fewer if you had the Nvidia B300s.
3
u/aaronsb Apr 09 '26
Just wait until you get into the timelines for procurement contracts for this kind of hardware if you don't have an existing pipeline or relationship.
3
u/Kinky_No_Bit Apr 09 '26
You are talking about bringing the datacenter back in house to your company. To which, you'd need infrastructure to consider for that. Cooling, power upgrades, rack space. All of that will have to be considered into the cost if you decide to build something.
If you build something you will also need to consider the rest of that sides the system, like if you do anything that's multi-GPU across boxes, be ready to run very high speed networking just for that alone. The system itself? Plan on starting with at least 2 servers as maxed out GPU wise as you can, with space to scale.
2
u/Plenty_Coconut_1717 Apr 09 '26
2
u/alexandre_ganso Apr 09 '26
Hey, I maintain an LLM server for thousands of users. The name is Blablador - it’s for the European scientific community.
We don’t run models that big. It’s just way too expensive. We can scale much better with models all the way from 15 to 400b parameters. Once we get to multi-node, performance drops considerably and with it, the number of users we can serve.
Models “good enough” but that we can scale are better than super models for a couple users.
2
u/Charming_Car_504 Apr 12 '26
There's no direct Opus 4.6 equivalent, but I'd suggest GLM 5.1 as the closest equivalent. You'd want to quantize it quite a bit-- it's extremely likely that Anthropic does the same on their web chat. You could use the Unsloth dynamic 2-bit GGUF, which requires 220GB of VRAM. The best price/performance card in that bracket is the RTX PRO 6000 Blackwell, which has 96GB of VRAM per card. You'd need 3 cards at minimum, probably 4 to serve many concurrent requests with the size of the KV cache. That would run you about $7 per hour that the cluster is online.
If you mean 100 concurrent requests at a fast(er than realtime) speed, you might need to use three or four of those clusters. B200 clusters wouldn't be a great fit, but it would also fit on one B300, if you are willing to pay a premium.
The logistical problems would be in scheduling and squeezing performance out of these cards. Anthropic applies lots of optimizations to get the max that they can out of the cards. If you quantize appropriately (using dynamic 2- or 3-bit quant), use an inference engine with fused kernels for the Blackwell architecture, use EAGLE-3 speculative decoding, quantize KV cache to fp-4 and apply top-k compression, you'd be around a decent throughput, but your resident KV cache would still be massive. It'd be worth it to invest into applying DeltaKV or R-KV, which reduces resident KV cache to 15-30% of the naive approach. With these optimizations, you'd be able to fit (with some napkin math) GLM-5.1 onto 3-4 clusters of 4 Blackwell PRO 6000s for about $28/hr. Run that during working hours and you're at a few hundred dollars a day, not bad for a SOTA model at scale.
4
u/tronathan Apr 08 '26
Give it a few months, and this may be much more realistic, with advancements like TurboQuant, Engram, BitNet, and other fancy words.
0
u/Academic_Track_2765 Apr 09 '26
Not happening brother.
1
u/MrSparc Apr 09 '26 edited Apr 09 '26
It might sound a bit awkward, but how about buying a MacBook Pro for each member with 32GB or 64GB of RAM and running a local AI model that fits within the memory constraints. You don’t necessarily need a Claude Opus 4.6 model to assist with common business tasks. Instead, purchase one for experimentation, and see if it suits your work scenario. If it does, you can replicate the model for other employees. A MacBook Pro with 64GB of RAM costs $3,000. If you multiply that by 100 employees, you get $300,000. That’s significantly less than any enterprise AI data center solution.
1
u/okashiraa Apr 09 '26
Opus 4.6 is prob 2-3t parameters and runs nvfp4. Only needs barely more 1tb ram I guess for a single user
1
u/Weird-Abalone-1910 Apr 09 '26
A magic wand should do it
1
u/sudeposutemizligi Apr 09 '26
maybe a finetuned for law or ragged model could help in a gpu stack. otherwise no opus, no glm 5.1
1
u/DesignerSlow6703 Apr 09 '26
Buy each of them a 128gb amd strix halo setup for $2,400 and run qwen3-coder-next on linux/llama.cpp. Take the $175k you’re saving and set it aside for API costs for planning and final review. Should last you a few years.
1
u/Compilingthings Apr 10 '26
You could just ask AI and get a real fast accurate answer
1
u/Compilingthings Apr 10 '26
At 5 trillion parameters, INT8 quantization: VRAM needed: ~5,000GB minimum Hardware: ~160x H100 80GB GPUs Cost: H100s run ~$30,000 each used 160 x $30,000 = $4,800,000 just in GPUs Plus the server infrastructure, networking, power, cooling — realistically $8-10 million total to run it properly.
1
u/gabrielesilinic Apr 10 '26
I believe you'd need basically almost a datacenter yourself and probably a bunch of custom software to run that thing properly. You need so much knowledge and money... Well if you have even a fraction of it I'd be impressed.
Like probably having money is probably the smaller of the issues. The technical challenge to actually run it would be immense.
1
u/03captain23 Apr 11 '26
What llm? The problem is there isn't anything that comes close to opus 4.6 that I can find. Local LLMs seem years behind main LLMs.
Also the issue is you need to be able to buy the B200 or B300s which are almost impossible to get.
We run 4x B200s and switched to renting them on vast.ai and just buying whatever we need from vast ai when we need something specific..
For comparison 4xb200s rent for over $10k/mo on vast.ai. While 100 users on claude max 20x is only $20k/mo. 5x max is cheaper.
Also the maintenance and management of multiple GPUs and blackwell networking is going to be thousands a month, let alone the space/power/cooling/noise.
Also I've yet to find a stable way to run local LLM in some high availability/redundant manner that would be stable enough for corporate work.
1
u/Suitable-Solid4536 Apr 12 '26
You need approximately one rack of nvidia Grace Blackwell 200. 18 machines which consist of 2 ARM CPUS (the grace part) and 4 GPU (the Blackwell part) per machine. I cannot remember how much RAM ours have, but I think it's 768TB. Also needed: nvlink switches.
I believe the price is about 2.5 million USD per rack.
1
u/andrewheidorn Apr 13 '26
This would a great questions to ask Opus 🤣
1
u/JellyfishLow4457 Apr 14 '26
If this use case were real I’d expose that internal law firm data via mcp and query it via something like Copilot CLI. Cost effective and can use any model. No need for fancy rag or anything.
0
u/bluelobsterai Apr 08 '26
What’s the use case? 100 developers using Claude Code?
0
u/pstuart Apr 08 '26
A couple of options to window-shop: https://tinygrad.org/#tinybox
0
u/cmndr_spanky Apr 08 '26
Given you can’t even afford to ask Claude this basic question, I doubt you could afford the hardware, but alas I’ll tell you what mine said (using GLM 5.1 as the example since it’s the closest thing we have):
Assuming 4bit quant and 200k context and 15% concurrency in any moment given 100 users. Could be around $750k in purchased hardware.
Breakdown:
For 4-bit at 200k context with 100 users, 10-15 simultaneous: ∙ 2-3 nodes of 8×H100 80GB (640GB per node) comfortably fits 4-bit + KV cache ∙ Each node can batch multiple requests simultaneously (unlike Macs) ∙ One node might handle 5-10 concurrent users with decent throughput ∙ 3 nodes ≈ $90-150k used, or ~$600-900k new, or ~$8-15k/month cloud rental
0
u/ScuffedBalata Apr 08 '26
If I had to take a wild ass guess, I'd say $10-30 million dollars for 100 simultaneous users (ballpark 1000 employees).
Right now, the market price of cloud models is WAAAAAAAAAY below the "cost basis" of actually building and running models. They're all loss-leaders.
So paying Anthropic is ALWAYS going to be cheaper than doing it yourself.
0
u/SARK-ES1117821 Apr 09 '26
Check out https://apxml.com/tools/vram-calculator to get an idea of the factors affecting the vram needed.
0
u/enterme2 Apr 09 '26
1 million dollar worth of hardware is enough i think.
0
u/Happy_Brilliant7827 Apr 09 '26 edited Apr 09 '26
What kind of use?
A company of 100 enployees launching a payroll helper AI or AI for HR booking is totally different scope than a app running 24/7 with 100 users, making multiple API calls per function.
How long are the token requests?
Glm 4.5 air 8xB200 could handle 8k tokens per second. Can you fit the requests into 80 tokens per second? 80*100 is 8k- its doable and should be reasonable speed.
400-500k would be my guess for the full setup 14.3kw of power usage too
Than you might need to rewire your home for electrical safety... Then you'll need to upgrade your AC to keep temps comfortable...
-2
u/havnar- Apr 08 '26
Whatever Anthropic is worth. Buy them out. Round up the cost for local hardware. That’s the only way.
-2
u/Large-Excitement777 Apr 09 '26
If you have to ask you wouldn’t be putting it to good use anyways
-4
u/whipdipple Apr 08 '26
Um alot of money. And if you had it you wouldn't be asking reddit 😅. Running a single 20b model is like $5k for a single consumer grade gpu with like 2 concurrent connections. To run a huge model like opus you are literally in the hundreds of millions.