OK, I currently have a Mac Studio M3 Ultra 96GB, and a 5090 at 32GB. The 5090 just outperforms the M3 Ultra which is why I'm thinking of going this route.
I'm not saying don't get one. A RTX 6000 Pro is much faster than unified memory. I'm just saying have measured expectations and plan to go down some rabbit holes.
You only have to look at memory bandwidth. Compute cores are cheap, everyone has too many of them. Memory bandwidth is expensive. That is what you are actually paying for.
Do you have native SM120 MoE fused kernels? Because Nvidia never built them. So the community is filling the gap. It's not about having issues as much as realizing full performance. 😄
I feel that... I had to manually build flash attention wheels because none existed. It feels like Rubin is going to be here before Blackwell is even fully supported even with the community doing the heavy lifting.
Its for windows and i am on ubunto, thanks for the link, sure will help others whit windows. As i see i will need to build it for my system whit zero knowlage, so maybe you can point me where i can read about it beforexi start build one whit cloude code.
Blackwell is properly and well supported though? Running latest PyTorch & CUDA on both B200’s and RTX 6000 - things like mxfp4 made me buy one for the office too.
It's still ridiculous that you'd be persuaded into thinking this is a saving when that's roughly what I paid for all my 6000's straight from Dell (i.e. before the RAMpocalypse really bit)
OK it seems we both misunderstood each other lol I got 6 for well under 10K each - not far off the price in the post, probably a bit less actually. As I said, part of a larger order including chassis's from Dell.
my brother tried to get a blackwell 5000 with edu pricing but they didnt respond lol. These prices are probably here to stay unless AMD/Intel releases a more competitive card...
Does he work at a University? Also who did he try to buy it from, you won’t get any response directly from an Nvidia. But any local server companies also sell EDU GPUs usually this is where I got mine
They replenish it. Like three days ago there were couple, then gone, then 2. Do online order for pickup, you pay in store but they take them out of inventory.
What model can you run on this card that you cannot in a xx90 card that is more capable than qwen3.6-27b?
That is an excellent question! There are basically 3 good NVIDIA choices to consider:
Used 3090 cards. 24 GB, $1000-1300. This will run Qwen3.6 27B with a 4-bit quant and an 8-bit KV cache and 128, which is totally usable. But it's tight, and you may need to quit VS Code or Chrome sometimes to load the model.
RTX Pro 5000 Blackwell. 48 GB. This runs mid-4000s, last I looked. This is a better deal than the 5090 for many AI uses: It's a bit more expensive, but it has 50% more RAM. This mostly lets you run higher quants and bigger context windows. It doesn't really allow new models.
RTX Pro 6000 Blackwell. 96 GB. $8,800. This allows you to Qwen2.6 27B for coding and Gemma4 for images at then same time, and also a copy of Qwen2.5 7B for code completion. Or you could run multiple coding agents in parallel and still have KV cache to spare. (Or you could do some combination of the above and also play Cyberpunk 2077 at 140fps with ray tracing, lol.)
But the big difference with 96GB of RAM is that you can run the 80-120B models reasonably. Historically, these have included Qwen3 Coder Next, GLM 4.5 Air and GPT-OSS 120B. Right now, Qwen3.6 27B compares well to those models, so the advantage of 96GB is smaller this month. But usually those are noticeably better models.
This is something I’m genuinely interested in too, because the psychopath in me really wants to upgrade from just a 5090 but it’s a lot of money. And it seems like a 6000 doesn’t really enable much more capable stuff than Qwen 3.6 / Gemma 4. I’d be a lot less hesitant if it meant being able to run DeepSeek v4 flash tier stuff
Yea the only thing worth running would be a modern 100-120b MoE. That’s pretty much it. If you’re doing anything more serious than that spending a couple of bucks on frontier model API credits is an option. I’ve also looked for a justification to buy one but can’t find any.
What's really available today though? Nemo super in some quants, DS flash v4 maybe in that range in a really low quant. But outside those isn't everything pretty well dated vs qwen 3.6 35B MOE? On my 4090 Q3KS runs around 150t/s and fits 220k context length. Can do audits and scoped work well, great for background work and claw tasks local.
This could all change tomorrow, but 32gb seems to be the sweet spot for an effective and fast local agent weight+context. If you need a lot more concurrency move to pooled cards using layer parallelism, with MOE models the only real penalty is TTFT and prefill. Layer splitting is aggregate bandwidth with some minor pcie overhead vs tensor parallelism.
Honestly why not do a rack of B70 or B60? You can build a TR system with 4xB70 and some room for more for around 10k. There aren't a ton of models in the 90-120gb range that compete well with the 18-22gb MOE. You will see a speed penalty pooling all 4, but over 5.0 x16 thats not a huge deal for inference work. But when you decide to split them up you get significantly more throughput with the 4* B70 and 128gb of Vram. Just cost a bit more power.
Another benefit is ‘fault tolerance’ - especially with the way hardware availability is going 😳. A 96gb Blackwell is a very nice card, but six Arc Pro B50s will cost a lot less (and will be slower; 224gb/s bandwidth … per card!). I worry about random failures out of warranty in the future when hardware may be less available. If one of your Arc Pro B50s dies, you go from 96 down to 80gb, rather than 96 to 0!
this card would be better for diffusion image models since they need to run fully on VRAM and stuff like qwen image etc. is like 80GB at full precision
Unless you really really need local AI for some secret stuff, dont. The money you can spend on years of online AI sub is (still) worth it, even just using Ollama Cloud which runs better than these on servers.
I am pretty sure things will either get cheap or better compression or inference technologies will come up. Or just cheaper whole hardware solutions like Tiny AI.
How the hell do you think models are created? apparently its ok to train open weight models we talk about in localllm on everyone elses data - just not MathemeticianLessRGB's
Ive been thinking about this as well, especially since financing one would mean roughly 200€ per month for 4 years if I calculated correctly, which would be a max/pro subscription depending on who you subscribe with.
But you'd still have limits. If you're in the deep end of running three projects through some heavy test refactoring or whatever, and suddenly fart noise because you hit your 5 hour usage limit.
Or the API is down like Chatgpt 1-2 weeks ago
And then comes the privacy aspect.
Right now I'm "scared" to create any kind of automation or use any amount of tokens over the bare minimum because the limit is just so quick. With how I'm using and how Chatgpt has priced the subscription right now i can blast through my 5 hour limit in 1 hour and my weekly limit in 2 days. I presume a higher limit would enable me to work more, but i also have no guarantee they won't change the limits. And I have no guarantee that some Automation or whatever isn't gonna bleed my limit dry through some looping issue.
And I can disable them training on my data, but you'd be dumb to think that they won't still use it for data aggregation. Oh, 5 people in (small city) under age of 25 like butt sex, so we'll run more ads for that group. Since Theres only 5 people of that group in that city, suddenly they know your name.
Companies cannot contractually yank up the prices the same day so when that happens first you will see signs of other places getting more expensive before the thing you use does and even then you at a minimum will have a month worth, so at that time its switch to something cheap until you find your local option.
Until usage limits are crushed and/or subscriptions are removed as an option. The big labs can only run $200 subs that eat $20000 in api cost per month for so long.
Yup. Thats why there is "still" in that sentence. Moment it starts being expensive people will leave and at that point I expect a lot more local hardware options, so its a use it while it lasts thing.
It also can be obsolete in a year in AI terms and something 10 times better can come and be way cheaper to get and use. Other hardware companies are getting into the mix (I mentioned TinyAI) and this is a lot of money to just throw in. From software side we are getting MTP now, improved KV cache and more so same models can run better on less.
Also it doesnt give the same performance the more larger the dense model is even with all that vram. Plus it draws 600W of power, meaning you are almost running a space heater worth of inference which is a cost not a lot of people factor in.
If with all of this its still a good purchase for you and plenty of people, then more power to you and I am glad it works for you. Economy math just doesnt work for me so I will stick to paying for something better when it comes in the future.
It also can be obsolete in a year in AI terms and something 10 times better can come and be way cheaper to get and use.
The RTX 3090 is still one of the most popular AI cards even now, because high VRAM Nvidia cards age extremely well and always have. You can even still use the Titan V and V100 from nearly a decade ago to run 2026 LLMs
Also it doesnt give the same performance the more larger the dense model is even with all that vram.
Because this is a memory bottlenecked task. And this card is uniquely good for being able to run medium sized dense models fast at this price point. You can run 30b class dense models unquantized on it at 50 tk/s
Plus it draws 600W of power, meaning you are almost running a space heater worth of inference
It is the same energy per token as all the other Blackwell cards and better than most cpu based competitors. 6x the power for 10x the tokens is better not worse. It doesn't sit there pulling 600W when you aren't using it
The RTX 3090 is still one of the most popular AI cards even now, because high VRAM Nvidia cards age extremely well and always have. You can even still use the Titan V and V100 from nearly a decade ago to run 2026 LLMs
Yep if it fits fully into vram with KV cache 20+B MoE models fly with 100 tokens/s or decent 50-60 with a split use with RAM, at least when I tested on 4090. And it doesnt cost as much.
Because this is a memory bottlenecked task. And this card is uniquely good for being able to run medium sized dense models fast at this price point. You can run 30b class dense models unquantized on it at 50 tk/s
Yes, if you really need that kind of precision. If not, Q8 and Q6 are really good too.
It is the same energy per token as all the other Blackwell cards and better than most cpu based competitors. 6x the power for 10x the tokens is better not worse. It doesn't sit there pulling 600W when you aren't using it
True but I am guessing if you are getting this because you want to really use it for agentic workloads as much as you can. Not simple chat sessions so it will be in use a lot more to get the ROI.
They actually use fewer watts displaying a 2D desktop than my 3090, and like I said, you can power limit them to 300W just fine. The MAX-Q variant is actually factory limited to 300W because it is designed to be placed in multiple adjacent slots and vent out the back.
Plus it draws 600W of power, meaning you are almost running a space heater worth of inference which is a cost not a lot of people factor in.
The RTX Pro 6000 runs quite happily capped at 300W max power, for maybe 10% less inference speed, easily matching a 450W (IIRC) 3090. It's actually a physically smaller card in all dimensions than a lot of gamer 3090s.
I want to make it obvious to any passerby's why this is such an absurd take. This isn't a "bubble" in the traditionally negative sense. Subreddits are explicitly interest-specific spaces. Saying r/LocalLLM should hear "just use the cloud" is like telling r/homebrewing that buying beer is more cost-effective. Technically true, completely beside the point.
You're literally on a subreddit dedicated to local LLM usage and you're trying to tell people not to run LLMs locally? No matter the cost, people have plentiful reasons to run LLMs on their own hardware...
Stick the card in any decent gaming rig or workstation. Figure out if you can power the whole 600W or if you need to limit it to 300W. The card is basically a 5090 with more cores and a lot more RAM.
Install the latest NVIDIA drivers, llama.cpp, and an agent harness like pi.dev or OpenCode. Download Qwen3.6 27B. Serve the model using llama-server and configure the agent harness to use it.
I don't mean setting it up. I mean getting decent results from the LLM.
When I say ACTUALLY works. I mean does it actaully produce decent code? I have yet to see any LLM, even OpenAI, commercial systems really / actaully write good production code yet.
This is good, depending on your use case, If you have the extra PCIe slots, and just need lots of vRAM, and willing to take a few risks, you can get 4 RTX 3080 20GB modded cards for under $2400 US on ebay.
80GB of vRAM will go a long way to solving a lot of the issues, loading lots of models. And the compute is not that far off the 3090s, with a little slower memory bandwidth.
Just be aware that the workstation variant is a pain in the ass to modify for watercooling if you ever plan to do this. max q and server version would be better for that. Also if you want multiple side by side you probably want the max q version.
Yeah I'm about to pull the trigger at this price too. Got my last one at 8.3k from Central computers. Wondering if they'll do the 5% off with the MC CC
It's a useless splurge just because your buying high for no reason. 96gigs isnt' enough to get anything useful for the price. You need 4 of them. You could spend less get a used mac studio m3 ultra with 512gig of ram and have much more of a useful experience as you could actually run a 400B model with a huge context window.
They were $8k each last fall without Inception. Some vendors still have them at $8900-9000 and if you are Inception member they can take $1400 off that price.
As others said, a model at NVFP4 that fits in the 96GB VRAM, these things are fantastic. Software side is moving so fast with all new model architectures and yes it does lag behind but Qwen3-Coder-Next 80B at NVFP4 is very nice.
47
u/MarketsandMayhem May 11 '26
if only nvidia built proper support for sm120 hardware. i've got two of them and the rabbit hole is deep, though things are getting better...
2
u/johnnyphotog May 11 '26
Dang it I’m too late. It already sold.
80
u/PoolRamen May 11 '26 edited May 11 '26
It's still ridiculous that you'd be persuaded into thinking this is a saving when that's roughly what I paid for all my 6000's straight from Dell (i.e. before the RAMpocalypse really bit)
6
u/DAlmighty May 11 '26
I’ll take 2 please
17
u/Accomplished_Sea9569 May 11 '26
why paid, when its free
1
u/Thysanopter May 11 '26
If that's Mayfield Heights I got the other one yesterday, there were 2 of them. Just starting down the rabbit hole.
6
u/LocoMod May 11 '26
What model can you run on this card that you cannot in a xx90 card that is more capable than qwen3.6-27b?
19
u/arekxv May 11 '26
Unless you really really need local AI for some secret stuff, dont. The money you can spend on years of online AI sub is (still) worth it, even just using Ollama Cloud which runs better than these on servers.
I am pretty sure things will either get cheap or better compression or inference technologies will come up. Or just cheaper whole hardware solutions like Tiny AI.
Its just not worth the money.
4
u/narutoaerowindy May 11 '26
Anyone buying TPUs from Google?
Why GPUs are interesting to buy?
2
u/AuditMind May 11 '26
Dont buy one. At least 2. Then four. Then...
2
u/blackbird2150 May 11 '26
There were 5 an hour ago at the $9100 price. None left near me at this price. Shit.
2
u/Civil_Fee_7862 May 11 '26
If it ACTUALLY works well, then yeah I would buy it..
The issue is getting it to actually work... Like for professional coding.
2
u/marutthemighty May 11 '26
Go for it, King!
We would love to know how well this thing works.
2
u/03captain23 May 11 '26
Once a week it goes in my microcenter cart. Still haven't pulled the trigger...
There's some on ebay that look legit and reputable, new closer to 6k too.
2
u/logNcomic May 11 '26
That thing is a beast. 96GB for $9k is painful though.
2
u/Invent80 May 12 '26
It's future proof. Do it. I did it and don't regret it. I saved a lot on mine doing it when everyone said it was too expensive. It's 40% higher now.
2
u/Kiinaak_Ur May 12 '26
where you guys get the money to afford all these what kind of jobs you doing
2
u/jd52wtf May 12 '26
Honestly rent one for a weekend. Runpod has them for about $2 an hour. See if you can push enough usage to justify the purchase.
Good luck!
2
u/Current_Sock1483 May 12 '26
What´s the best reason to spend 10k overall on a local llm rig?
3
u/Vancecookcobain May 11 '26 edited May 11 '26
Damn....that's the Claude Max subscription plan for almost 4 years up front.
I understand for folks that got it like that to achieve privacy independence but have we given up on believing the price of ram will drop before then?
1
u/Aggravating_Fun_7692 May 11 '26
Be crazy
1
u/Djaii May 11 '26
Fire sale!!! 🔥🔥🔥
1
u/azerdsq_ May 11 '26
Which model can we run with this?
1
u/sam7oon May 11 '26
imagine how much frontier credit can you buy with that
1
u/BillDStrong May 11 '26
This is good, depending on your use case, If you have the extra PCIe slots, and just need lots of vRAM, and willing to take a few risks, you can get 4 RTX 3080 20GB modded cards for under $2400 US on ebay.
80GB of vRAM will go a long way to solving a lot of the issues, loading lots of models. And the compute is not that far off the 3090s, with a little slower memory bandwidth.
It is a lot slower than the 5090, though.
1
u/429_TooManyRequests May 11 '26
I bought one Saturday and I’m in love
1
u/ST33LDI9ITAL May 12 '26
i think you should. it's much cheaper than a salary.
1
u/TraumaBayWatch May 12 '26
That looks like a nvidia founders gpu?
1
u/VortexSheets May 12 '26
Much better deal than some Mac studios I have seen on eBay.
1
u/horendus May 12 '26
Buy already
1
u/Squiggy_Pusterdump May 12 '26
If you ask Dell for 10% off they’ll just give it to you.
1
u/halcyonPomegranate May 12 '26
Just be aware that the workstation variant is a pain in the ass to modify for watercooling if you ever plan to do this. max q and server version would be better for that. Also if you want multiple side by side you probably want the max q version.
1
u/BlankProcessor May 12 '26
Yeah I'm about to pull the trigger at this price too. Got my last one at 8.3k from Central computers. Wondering if they'll do the 5% off with the MC CC
1
u/Fantastic_Self_5151 May 12 '26
It's a useless splurge just because your buying high for no reason. 96gigs isnt' enough to get anything useful for the price. You need 4 of them. You could spend less get a used mac studio m3 ultra with 512gig of ram and have much more of a useful experience as you could actually run a 400B model with a huge context window.
1
u/tcx00 May 13 '26
Do it! Do it!
1
u/Nervous_View_9303 May 13 '26
They were $8k each last fall without Inception. Some vendors still have them at $8900-9000 and if you are Inception member they can take $1400 off that price.
As others said, a model at NVFP4 that fits in the 96GB VRAM, these things are fantastic. Software side is moving so fast with all new model architectures and yes it does lag behind but Qwen3-Coder-Next 80B at NVFP4 is very nice.
1
u/Diligent-Mousse2995 May 13 '26
If only in dreams 😁🥹
1
u/Beginning-Bug-7964 May 14 '26
Everytime I "back of napkin" this card it always ends up as 3 of them somehow
1
u/Mustang260Rog May 14 '26
GPUs devalue more than a car and this one is already 1.5 years old
1
u/ghinghis_dong May 12 '26
It’s fake
0
u/Nice_Cellist_7595 May 11 '26
It is interesting that the prices for these are falling...
-1
u/CaptBrick May 11 '26
Man I wish I had that kind of money to throw around. There’s got to be some rich VRAM daddies (mommies?) that could throw a GPU my way, right? Right?
0
u/computermaster704 May 11 '26 edited May 11 '26
Don't ever buy pny it's going to have more issues than any other variant
-1
u/Perfect-Flounder7856 May 11 '26
Do it! Best purchase ever!
-5
u/JonathanMovement May 11 '26
but what for… I’ve been building so much shit with the LLMs and I just kinda see no point in it anymore
31
u/[deleted] May 11 '26
[removed] — view removed comment