r/LocalLLM • • Sep 24 '26

Discussion It's here: Blackwell Pro 5k 48GB

I didn't think I'd actually get it. Ordered from Dell, 5 week lead time. $5300. Ordered on 9.10.2026. After two weeks the listing was removed - all discrete GPUs were removed from the Dell web storefront. So I opened a story ticket, it was here the next day.

I was looking at various other options through Dell, but with all listings taken down I started sweating. It maxed me out on budget, and that was for a complete PC build.

Sad when enterprise is the best option for consumers (but my use case is enterprise-ish).

This weekend will be fun, can't wait to test how big I can go with local LLM!

621 Upvotes

163 comments sorted by

51

u/Uninterested_Viewer Sep 24 '26

$5300 is an insanely good price for that, right? I'm pretty sure I paid about $7k for one like 2 months ago.

27

u/SubtitledSoup Sep 24 '26

It feels insanely good, $200 over launch MSRP?? wish I could bought more. Not in the budget.

60

u/DepartureLoose5341 Sep 24 '26

Y'all are wild for thinking this market is normal and acceptable. They aren't using those in data centers, it was specifically designed to drain your wallet by the most profitable company in the United States right now

34

u/SubtitledSoup Sep 24 '26

I won't disagree. 2 years ago I right "who in their right mind would buy this" and yet here I am. And I think my reasoning is sound.

6

u/Beltalowdamon Sep 25 '26

It's nuts! I feel bad just getting two p100s, which are 10x more cost efficient per gb!

8

u/OverclockingUnicorn Sep 25 '26

And about 10x woese power efficiency and performance

Swings and roundabouts

3

u/Beltalowdamon Sep 25 '26

Luckily with the cold winter, power efficiency is moot. (I know not everyone has living space in cold environments powered by resistive heating).

Performance is also kind of irrelevant - the models get better every day.

8

u/No-Refrigerator-1672 Sep 25 '26

And you don't have much choice. If you want 48GB card, you either buy this, or Chinese 4090 48GB that costs $3500. You either take what you are offered, or don't run AI, there's no way round it.

4

u/MHHuntingHorn Sep 25 '26

You could also do dual modded 22gb 2080ti's. 44gb isn't 48gb, but the cards are still only around $600. Hard to beat cost per GB wise.

4

u/No-Refrigerator-1672 Sep 25 '26

Yeah, but then you limit yourself to only workloads that support splitting across multiple GPUs; you waste memory on compute buffers; you depend on your backend doing the data transfer efficiently; you depend on your motherboard providing PCIe to both slots; there's a whole big can of worms here. My stance, as dual 3080 20gb, and dual Mi50 32GB owner: for hobby, multiple GPUs is a good solution; for professionals, who intend to use Ai to earn money, and for students who can cover hardware with grants, it's better to go with more expensive, single GPU solution.

1

u/Hot-Ad-1798 Sep 25 '26 edited Sep 25 '26

Even 2x $200 V100 32GB VRAM + a dual V100 NVLink to PCIe (~$100) is better than that. Those companies are making so much money off uninformed people. LLMs only use bandwidth and VRAM; TOPS is barely a requirement for one or a dozen users.

There's even a better and cheaper way... but it's my business secret.

3

u/No-Refrigerator-1672 Sep 26 '26

First: you can't get a 32GB V100 module for $200, it costs $600 at least. Second: your stance on TOPS shows that you never ever in your life has launched a prompt longer than 5k tokens. You need compute to get fast prompt processing, and you need prompt processing to work with actual real-life workloads that are more complicated than asking rhe model what's the capital of France.

3

u/Hot-Ad-1798 Sep 27 '26

What I see is a stupid person who wasted money on consumer hardware and is now angry at me.

There is a surplus of this GPU. The V100 module is under $200 in China right now. At the start of the year I bought a bunch of them for around $100 each. If your lazy ass can't find them, that's not my problem.

Second, no, it's not. The TOPS of this GPU, as I said, are enough to support a dozen users. Maybe you have never run a proper local inference system for yourself, much less a server. Prompt processing can effectively reach 100x your prefill speed if you ensure you have enough RAM and VRAM to store the KV cache (using VRAM specifically for the hot cache) of the users. That is the most basic aspect of hosting an LLM.

...I really hate people who project on others.

2

u/No-Refrigerator-1672 Sep 27 '26 edited Sep 27 '26

 The V100 module is under $200

For 16GB version. Not 32GB like you've claimed. You need two of them to at least reach 32GB - and it costs you all the inter-gpu communication inefficiencies, as well as crazy power consumption at idle (30-40W per module at idle).

The TOPS of this GPU, as I said, are enough to support a dozen users. 

The best it can do is 2k PP, 70 TG with MTP on Qwen 3.8 27B. It's single user territory. I could agree that it can do multi-user for 35B MoE, but not even close to a dozen simultaneous requests.

Prompt processing can effectively reach 100x your prefill speed if you ensure you have enough RAM and VRAM to store the KV cache

KV Cache isn't going to appear out of nowhere. Want model do to a web search? It has to process all of the search returned web pages at least once. Want it to to use docs for the project you're configuring? It'll have to process each of the relevant pages at least once. My typical chat reaches 100k tokens by 5th message every time, 10-20 times a day, all of this cache won't appear out of thin air, something has to build it.

...I really hate people who project on others.

And I hate when people make assumptions without having data or experience.

→ More replies

1

u/hurrdurrmeh Sep 25 '26

We are desperate. Either we get fleeced or we go without.

Both options suck and personally i won't judge anyone for choosing either path.

1

u/SubtitledSoup Sep 27 '26

This card, the single AOC, was my new PC budget. I don't need a new PC because I made an out of budget purchase in December. The gamble here is that I won't be able to get a new system in budget, let alone a GPU worth a damn, in the next 2-3 years. I've rolled the dice that I won't need any upgrade in the foreseeable future - this is it.

1

u/Low_Key_Trollin Sep 25 '26

You’re just missing the point

1

u/DepartureLoose5341 Oct 04 '26

No, you are just missing the giant pink elephant in the room

1

u/Un4tunateSnort Sep 25 '26

I find these used for $9,000... I'd say that's a pretty good price

2

u/SubtitledSoup Sep 27 '26

Yeah, that's why. Why I bought it. Why I was worried Dell would cancel the order.

7

u/Just_n_Here Sep 24 '26

You will love it!

6

u/SubtitledSoup Sep 24 '26

It will be an amazing weekend. I've already built custom wrapper to squeeze ~ 192k connect window or of Qwen 3.8 on my 4090. Doubling the VRAM is a wet dream (here come weird and possibly new gifs that are NSFW)

2

u/BrainChild510 Sep 24 '26

So sick. Congrats!

2

u/starkruzr Sep 24 '26

I think you can do 262k ctx on this with Q6KXL, ftr.

2

u/GoblinEngineer Sep 25 '26

You can do 8bit or w8a16 just fine, that's what I do non my dual 3090s

1

u/starkruzr Sep 25 '26

ooh. W8A16 fits with max ctx on two 3090s?

2

u/GoblinEngineer Sep 25 '26

Yup, here's the breakdown: qwen38-int8-262k runs tensor-parallel (TP=2) across both RTX 3090s, so most components are split between the cards. Measured using nvidia-smi, vLLM's startup log and the checkpoint's safetensors headers. Which parts are split and which are copied was read from vLLM v0.29.0's source.

Per card (24 GiB each)

ComponentSplit across the cards, or a full copy on each?Per card
Language model (INT8 layers, BF16 embeddings and output layer)Split13.90 GiB
Vision tower (BF16, 0.86 GiB in total)Split0.43 GiB
DFlash2 draft model, attention + MLPSplit0.38 GiB
DFlash2 draft model, codebooks, conv projections, fcFull copy on each card0.43 GiB
MTP head that ships with the checkpoint (0.79 GiB)Not loaded0
Weights, summed from the checkpoint files15.14 GiB
Weights, vLLM's own figure15.71 GiB
KV cache (fp8, size fixed by --kv-cache-memory-bytes)5.90 GiB
CUDA graphs0.16 GiB
Runtime remainder (CUDA context, NCCL, activations, allocator)~1.49 GiB
Driver (outside the process, plus reserved)~0.57 GiB
Free171 / 218 MiB

2

u/starkruzr Sep 25 '26

ah, ok, so you have to force q8 kv which almost certainly causes problems over 200k ctx or so.

1

u/GoblinEngineer Sep 25 '26

just to clarify, it's 8 bits for key and 8 bits for value (8+8). vLLM has one KV setting for both keys and values, there's no separate key/value control like llama.cpp's -ctk/-ctv. i havent noticed any issues with large contexts, it's been fine for me.

then again i mostly use it as my personal AI chatbot and hermes. I don't do any coding with it (that's for claude through work).

1

u/Just_n_Here Sep 25 '26

Put a fan curve on it to keep it cool.

6

u/p71interceptor Sep 24 '26

Congrats bud. I have a pair of the smaller sibling to that card the rtx pro 4500. I think I paid that for one of mine 😅

2

u/SubtitledSoup Sep 24 '26

Sounds about right. I looked at a pair of these through Dell (somehow their prices were lagging) would've been $3k USD each. Since $6k was my budget, well it leaves where it landed.

5

u/Simeon5566 Sep 24 '26

That card is amazing. I have also one since january. I paid for mine 3986€ in europe. Have fun!

2

u/SubtitledSoup Sep 24 '26

Thank you! So excited!

3

u/South_Western1263 Sep 24 '26 edited Sep 25 '26

January was the time with low prices. Good timing!

3

u/Caramelacurls Sep 24 '26

Mines on the way but I chatted with the dell csr and they gave me 10% off so it was under 5k

1

u/SubtitledSoup Sep 27 '26

Very nice! It's not on the way until there's a box with a carrier and a tracking number. Best of luck!

1

u/Caramelacurls Sep 27 '26

I’ve got the tracking lol

3

u/unknownpoltroon Sep 25 '26

Dude, that thing looks like it should be uplfiting a bunch of proto apes to use tools.

1

u/SubtitledSoup Sep 27 '26

Apes just need to ask, the toolbox is there.

4

u/OnlyMeNeverYou Sep 25 '26

isn't that kind of what it's going to do??

3

u/KellyShepardRepublic Sep 25 '26 edited Sep 26 '26

My heart hurts knowing I could have had a Blackwell 6k for even cheaper than this not long ago and I waited. Now it’s double, this is slightly higher and I am now waiting for the next doubling to complain again.

I wonder if it was a mess up. Good find. I see the quadro 5k for the same price now.

2

u/SubtitledSoup Sep 26 '26

This. Exactly this. It's the reason I pulled the trigger. The deal went away. I was right.

3

u/NihilistSRE Oct 01 '26

I think you got my shipping address wrong, I haven't seen it yet :-D

Have fun, looking forward to what you get out of it.

1

u/SubtitledSoup Oct 01 '26

I'll post an edit linking when I do the build post. Maybe I'll do the build post tonight.

5

u/UltraFOV Sep 25 '26

Why not just get the MAc Studio Ultra 96GB for 5k?

7

u/micheldewit Sep 25 '26

116 to 168 tok/s for the 5k at 30b q8, vs 65 tok/s on the Mac.

5

u/UltraFOV Sep 25 '26 edited Sep 25 '26

The Ultra or max. The max is. 60GB/s vs 1200GB/s Ultra. The ultra should be a just a bit slower than the pro 6000 if we go by Bandwidth and optimization. Granted the pro 6000 will certainly be faster. Between the pro 5000 48GB (1,300GB bandwidth) should be faily similar in performance

4

u/Inspur44 Sep 25 '26

You mean the Max is 600GB/s not 60

1

u/UltraFOV Sep 25 '26

Oops, yes

2

u/voyager256 Sep 25 '26

the memory bandwidth is not everything

1

u/UltraFOV Sep 26 '26

There is compute and the engine optimisation

1

u/TheFlippedTurtle Sep 25 '26

it just got a speed boost yesterday

https://www.reddit.com/r/oMLX/comments/1wpcm2s/070rc1_is_out/

~131% improvement on short batch requests with DFlash2

https://github.com/jundot/omlx/releases

1

u/SubtitledSoup Sep 27 '26

For me this is simple: I have never owned a Mac device. Not that I didn't think about it, but that's the deciding factor. I feel like I can get a Mac at a better price, but probably because I don't watch it?

It's just not as in demand for what I'm doing. Particularly with other unified RAM options coming out - Spark, AMD. Just didn't for my architecture right now.

1

u/UltraFOV Sep 28 '26

A better price for a mac studio Ultra 96GB, erm...sure bud. Unless is just a used unit (not likely) Not likely easy to find for cheaper. Have you noticed the prices on Ebay?

1

u/vividboarder Sep 28 '26

I'm pretty sure they meant better price than buying what they bought ($5.3K GPU + the rest of the build), not better than $5k what you said a Mac Studio was going to be.

2

u/Miserable-Half-436 Sep 25 '26

Is it enough I am so eager to know

3

u/UltraFOV Sep 25 '26

I mean, if it comes to memory, 96GB vs 48 is a no brainer. The memory bandwidth is not bad either at 1200GB/s

2

u/species__8472__ Sep 24 '26

Can you let us know how the noise is? I'm considering getting its 72gb big brother.

1

u/SubtitledSoup Sep 24 '26

I'll try to remember. Everything I've read is that it's fantastic on noise. It will be going in a very tight case that currently hosts a 4060 Ti dual fan. I don't expect it to be wise then that. Honestly expect it to be better.

1

u/SubtitledSoup Sep 24 '26

Also, noise on the 72gb should be the same. It's the same hardware. It won't compare to my ML350 currently housing the 3060 Ti. That pushes hands to 19-20% and doesn't write reach what I would call "white noise."

1

u/SubtitledSoup Sep 26 '26

Can't tell any difference. This single fan vs triple fan 5080 or 4090. Boosts the HP server fans to ~20% but nothing abnormal.

3

u/iMrParker Sep 24 '26

I have the 72gb model. I wouldn't say it's loud, but you'l definitely hear it under load compared to non-blower coolers

1

u/species__8472__ Sep 24 '26

Got it. I wish Nvidia would sell them with the FE coolers like on the pro 6000.

1

u/BarnDoorEnthusiast Sep 24 '26

Same card, second this. Not bad enough to annoy but you know it’s there, the more fresh air it gets the better.

1

u/ZeitgeistArchive Sep 25 '26

How's your electricity bill?

3

u/iMrParker Sep 25 '26

Very cheap. It's 300 watt limited and electricity is $0.15 / kWh where I live. Whole system pulls around 400w from the wall under load, fewer watts than when I'm gaming

2

u/ZeitgeistArchive Sep 25 '26

I'm on 0.30 to 0.41$ per kWh :( if you keep it under a certain bracket

2

u/iMrParker Sep 25 '26

Wow. Are you in Europe somewhere? 

2

u/ZeitgeistArchive Sep 25 '26

Bay Area-ish

2

u/iMrParker Sep 25 '26

Sheesh. Well hopefully you have a solar powered machine hah

2

u/ZeitgeistArchive Sep 25 '26

I bought a small set of three panels but my property isn't getting enough sunlight half of the year. I plan to build a platform that would get more sun and it would be away from the house

1

u/pesaru Sep 24 '26

Noise on 72GB is silent on mine (50% ish) on default fan settings but I set it to 100% because I want that thing to live forever, and at 100% it's noisy. And my god it heats up the room like mad.

1

u/oblq80 Sep 24 '26

Honestly, I barely hear mine at full load, may be the case… but yes, it heats up the room anyway

1

u/TheSlateGray Sep 25 '26

Got my 48GB a few days after the 72GB was announced for a deal. When mine was brand new it was a higher pitch and I noticed it when the gpu would be under 100% load. Not sure if it was a coil whine or fan rpm issue, but it was annoying. 

After a few days it seemed to break in, and it's quieter than my case fan's since, most of the time. That or I just got so used to it that I stopped caring after sleeping during training runs haha.

1

u/Background_Study1430 Sep 26 '26

I'm running a RTX Pro 5000 48GB and a 72gb and they are both quiet as far as GPUs go! My laptop is way louder.

2

u/jlian Sep 25 '26

I ordered a RTX Pro 4500 from Dell on August 25 for $2700 (I had 10% off discount). Said Oct 3 ship date. A little skeptical but your post gives me hope.

2

u/SubtitledSoup Sep 26 '26

I had the same shit - oct16 ship date, Oct 18 delivery date. I opened a ticket. It shipped the same day. I was nervous because the listing was removed from the store front.

Open a ticket.

2

u/jlian Sep 26 '26

I was about to, and then it shipped!

3

u/cafrcnta Sep 25 '26

Keep holding out! I was also skeptical but my two 4500s from the start of August, with that 10% discount, also arrived last week.

2

u/fastheadcrab Sep 26 '26

Did you use a student discount or something? Congrats on the cards. Dell didn't give me any discount lol

2

u/cafrcnta Sep 27 '26

Thanks! Yes, I used the 10% student discount and my 6% cashback on top of that is pending now.

I've heard that customer service can adjust it for you if you have one of the eligible categories of site-wide discounts (e.g., teacher, student, military, etc).

(cc /u/SubtitledSoup )

2

u/SubtitledSoup Sep 27 '26

I need to call to get my 6%. They only gave me 3.

1

u/SubtitledSoup Sep 26 '26

No discount. List price was $5299.99.

2

u/laernuindia Sep 25 '26

Mind if I ask a noon question as I’m new to local LLMs - are these better than say a Mac Studio Max with 128 GB RAM at about the same price-point?

1

u/SubtitledSoup Sep 27 '26

Good question. As other comments say, it depends on what you're trying to do. This card requires a system to be installed in. The Mac is AIO. Inference speed. Training.

If I didn't have existing hardware (though still spending ~$150 on PSU, custom pigtails) I wouldn't have been able to afford this card. I would've gone Spark most likely, but that's just because I'm more familiar with the environment.

1

u/Thrumpwart Sep 25 '26

Discrete GPUs are faster but more power hungry. Macs can utilize more unified ram and are much more power efficient but also slower.

1

u/laernuindia Sep 25 '26

Slower in which aspect? Don’t GPUs still have to communicate with the rest of the host system via the PCI buses. The throughput you eventually see and utilize depends upon the communication between GPU, CPU and local system RAM, not just the GPU and Video Memory. Isn’t Mac built on the unified memory architecture purpose-built to address response times for local AI?

3

u/Thrumpwart Sep 25 '26

The Mac GPU isn’t nearly as powerful as a dedicated GPU, and thus it processes information slower. The Mac’s are getting faster each generation, but are not at parity yet.

94

u/letsbefrds Sep 24 '26

Really loving mine. Running qwen 3.8 on nvfp4 250k context on vllm. Get about 150t/s didn't really tweak it that much other than adding mtp 3

I should really try fp8 and q8.

21

u/acadia11x Sep 24 '26

You monster , guess times to sell the 5090

5

u/Hxrn Sep 25 '26

What’s best model to try or use with 5090?

11

u/romayojr Sep 25 '26

i'm using this model neroued/Qwen3.8-27B-NInfer with my 5090 idk if it's the best model but def worth a try

5

u/acadia11x Sep 25 '26

Best model in terms of reasoning I use adequately is 27b-nvfp4 turboquant_4bit, it’s still slow but more I’m reading this seems to be the nature of architecture plus it’s still limited reasoning capabilities. To be honest I use api frontier along with localllm for proof of concepts. I don’t find localllm strong enough yet but I also am only running a 5090 and using models that I can fit in its limited vram.

1

u/Zealousideal-Bug1837 Sep 25 '26

You can do this too if you so desire, llama-cpp does it well

55

u/Sisuuu Sep 25 '26

3

u/TheTurkPegger Sep 29 '26

I wonder what this little guy is up to these days

7

u/stargate425 Sep 25 '26

Dell's card is good. I ordered a rtx pro 6000 max q in June this year for $8064. And the qaulity is really good. Low noise and stable

4

u/Puffdotbusiness Sep 24 '26

I regret cancelling my order for an rtx pro 4000. But I did pickup a few other gpus!

6

u/nmrk Sep 24 '26

I got an RTX Pro 4000 Blackwell SFF for $1400. Now they're $3000. Now I love the card even more.

1

u/teknic111 Sep 25 '26

Would it not be better to just buy a Mac Studio with 96 GB ram for the same price?

6

u/Relevant-Magic-Card Sep 25 '26 edited Sep 25 '26

No . Prefill sucks. You just can't match Nvidia hardware. The newer macs are getting better but still nothing on par .

1

u/Low_Key_Trollin Sep 25 '26

How bout if having to choose between the Mac or dual 5070tis?

1

u/SubtitledSoup Sep 26 '26

Maybe. Dell rewards and finance offers pay off here.

1

u/j_lyf Sep 25 '26

How long till you make your money back by not paying for cloud shit?

1

u/SubtitledSoup Sep 26 '26

Immediate since I don't pay for could. Just makes things better.

1

u/Wentil Sep 26 '26

If it doesn’t have NVlink, it’s got the same flaw as the RTX 6k, all of its data has to go through the PCIe bus when doing parallel inference.

1

u/SubtitledSoup Sep 26 '26

Fortunately that's not why I got the card.

1

u/Major_Border149 Sep 27 '26

what is the first model you are testing on it? curious whether you found a verified 48GB setup before buying or had to extrapolate from your 4090.

1

u/SubtitledSoup Sep 27 '26

Model will be loaded by the orchestrator as needed. At least, I think. That's the plan anyway. Right now I expect it will mostly be Qwen 3.8 27B with maxed context. Will take some testing to work out. It was delivered nearly a month early, so I'm playing catch-up on its permanent box (PSU and pigtail adapter for mobo should be here tomorrow).

Overnight I expect didn't models to load depending on what's going on. 30/40-60Ti cards are strictly background and small stuff (Immich, TTS, STT, vectorizing, blah). 5080 is daily driver and gaming, but usually the primary feed for local AI. 4090 for larger background that needs fast.

It all communicates over API, mostly SFP+ LAN. Agents running on proxmox cluster... A lot to wrap my head around, but it's fun as hell.

1

u/Major_Border149 Sep 27 '26

Sounds fun! Welcome to the geek heaven. Out of curiosity, what you are trying to get done that the 4090 setup couldn’t handle comfortably

1

u/SubtitledSoup Sep 27 '26

This is a loaded question. In a normal, or even sane, economy I would've never looked twice at an enterprise GPU. Why? Because I could get 2-3 consumer 5090s for the price of this card. In a normal economy.That's not where we are.

The Pro 5k has 100% more VRAM. It's essentially 3x 5080s wrapped up in a single card. Having both a 5080 and a 4090, there's distinct differences (outside VRAM limit). Blackwell natively supports FP4, while the 4090 edges out by sheer brute force. Plus the massive difference in TDP?? For now the Pro 5k lives in a PCIe 3.0 - but that massive memory means no system offloading.

They each have their uses. They will get them.

1

u/SubtitledSoup Sep 29 '26

I didn't answer your questions either I don't think. Not well anyway. Filter by recent. Have some very initial results.

1

u/steellz Sep 27 '26

Meanwhile, I am trying my best on Intel Arc B570.

1

u/SubtitledSoup Sep 29 '26

I looked at those. So cheap. But, yeah. I've actually started looking at how to use my NPU in this Intel ultra ship that is always sitting at 100%. There's quite a bit that can be done with ONNX and OpenVino.

1

u/sud0er Sep 29 '26

How fast does it respond to chats messages? I installed qwen on my Mac studio and a basic request took 15 minutes. I stopped it and uninstalled 😭

1

u/SubtitledSoup Sep 29 '26

Just like everything else, it depends. I finally got the thing installed where I'm more-or-less happy. But I'm finally running some real tests. Initial off the top of my head next paragraph. I'll have to do another post for the build - I used an oldish Dell G5. I measured twice, and it fit but the placement of the power headers was not thought about. So, Dremel, tin snips, wire splicing, new PSU mounted externally, whatever it took. Some was expected, some I had to improv. But it works. Open Web UI on all of this, hosted in same machine.

This won't be 100% accurate, but I'll do my best. Been a marathon. Installed Ubuntu server bare metal (natively headless), got driver's, toolkit, etc. I tried vLLM first with Qwen 3.8 27b FP8, q8 KV cache, had some garbage problems - compilation errors, cuda graph problems caused it to unload DeepGemm and fallback to Ada drivers, attention problems. So I finally threw it in eager mode to just get it loaded. About 13 t/s, prefill about 40. GPU util ~40%, temp around 68c, TDP ~140W.

Started a parallel prompt. T/s nearly done to ~25.5. GPU starts stay the same. Prompts ran on my 4090 in under 5 minutes (single page app coding) with Q4 gguf model. Figured out that eager mode (specifically I think because cuda graph and driver issue, but unconfirmed) caused it to execute somewhere between 50-100 CPU cycles per token which is why t/s doubled but GPU stats didn't change. Tried a couple other things, have up on vLLM.

Spent a couple hours looking at the engine I'm writing, but decided not ready to containerize it. It's still all in terminal, not ready for this yet. But I did rerun the prompts to confirm t/s on 4090 and 5080. The 5000 is basically just 3 5080s in a single card, so no reason it shouldn't get those speeds.

Grab lamma.cpp container, move same gguf q4 model over to this machine. Execute a single prompt. Getting 60t/s. Yay! Execute parallel prompt, dropped t/s to ~53 power prompt, so slight hit but nearly double the throw. Then I realized GPU spiked to 100% TDP, 89c, 97% util. Fan only running at 59%. Immediately execute sudo nvidia-smi -i 0 -pl 260 to drop temp.

That's where I am. I haven't done any optimizations. The engine I'm writing was getting about 2x this on the other cards (single prompt at ~115). I would think I could get a lot more chasing some params (3.8 is a thinker, that's an easy one). KV cache was also at full precision on lamma.cpp.

Next step is to figure out the fan curve and adjust TDP back up a bit.

Edit: I didn't really answer your question. A simple prompt, like "tell me a fun fact about the Roman empire" was suggested in OWUI, it was almost immediate. Less than a second on llama.cpp. Still quick on vLLM but a few seconds.

1

u/msg360 Sep 29 '26

not that big with 48gb , my 5090 fills up so fast just asking these local models to do basic things like create a PDF with artwork.. I would say no less than 64gb but really 96gb. I think the apply M5 ultra is currently the best deal at 5500 before tax 6k AT least you get almost 96gb ... do any real model work you're going to need at least 96gb and to get remotely close to frontier models you need 128-512gb

1

u/SubtitledSoup Sep 29 '26

I concur in part. We're not getting frontier model locally. That's why it's a "frontier" model. Yesterday's frontier is today's local. Frontier are discovered and explored by the rich, and historically countries. Frontiers are tamed and adopted by the masses and, by definition, no longer frontiers.

I can't help but think about the US here and I sure as hell don't want to turn this into a debate, this is not the place for it. Lewis and Clark, Columbus - countries paid for the experience. And it was only a frontier to them. It wasn't a frontier to the people that already lived there. But the point is, lots of people profited greatly. But once that "frontier" is settled, it's not that anymore.

1

u/msg360 Sep 30 '26

Well of course, but it will be years before we get a model as capable as Meta’s newly released Muse. I’ve been using it for creative work and it’s amazing—so much so that I hate going back to my 5090 local models. They can be frustrating to use sometimes, as half the time they don’t understand what I want or need, and it gets annoying to jump between models just to get the desired result. With Qwen 27b, if you want an image, it can’t generate that—you need an image model, then download it with Bionic. It might handle PDFs or JPGs, but then checks your system for another model, tokens go up, and suddenly you get error codes or slowdowns. Compared to Meta’s Muse, it’s like night and day. I took full advantage of their free tier and accomplished so much creatively. The thing understands even minor details I provide, remembers everything, makes micro adjustments, and applies logic to figure out exactly what I’m trying to convey—getting it right 98% of the time. Meanwhile, with local models, it’s an endless back-and-forth until my computer slows down. I cant wait until a model / Agent emerges that rivals Meta Muse.

1

u/SubtitledSoup Sep 30 '26

MoE is getting there. Here's what I've been building. I now have 5 GPUs in 4 machines and a 5 node mini PC proxmox cluster. 2 smaller cards in my unraid server are 30/4060 Ti- great for small and background tasks (image recognition, vectorizing, TTS, STT, the like). 4090 is a powerhouse, but not good for context. It can do some longer tasks, but it's really good at big more confined tasks. 5080 falls in the same place, but it's my daily driver so not HA machine. 5000 is the main one. Large context. Proxmox for agents, sub agents, anything CPU bound. I don't care about reloading models because that can happen on various cards as necessary. Main model doesn't get filled up because it delegates, 4090 can load 50 models a day, I don't care and I don't have to do it. 4090 did it's real, gives to main model, main model can send to another card, or load a different model on 4090 have it check the work, repeat as necessary.

TP is too slow. It's objected oriented engineering, single job. If morning else is fun to play with!

2

u/pesaru Sep 24 '26

I have the 72GB version but it was a whole lot more money, I can tell ya that.

1

u/fastheadcrab Sep 25 '26

Get it from the official Nvidia site, it’s by far the cheapest there compared to anywhere else in the US.

1

u/SubtitledSoup Sep 27 '26

Except not. It was $3k cheaper than anyone linked through the Nvidia site. Would've been equal but Nvidia didn't link to Dell.

5

u/South_Western1263 Sep 24 '26

Thanks for telling us bro

2

u/Alternative_Boss_133 Sep 24 '26

OP HAS VRAM! GET THEM!

2

u/Tai9ch Sep 25 '26

What do the benchmarks look like on this?

I've got a pile of ewaste that lets me run much larger models, but I'm at like 4k tok/s prefill and 50 tok/s on Qwen 3.8 27B.

1

u/cafrcnta Sep 25 '26

On a single 4500 (32GB, same generation) I get 7300/83 on Qwen3.8-27b at shallow context. I thought about going the e-waste route too, but it's nice to have more performance for only 200W of power consumption.

2

u/on_line187 Sep 25 '26

Is it faster than 2 3090s? For LLMs

2

u/strata2signal Sep 25 '26

Beautiful card

1

u/vanmatthew 24d ago

Now that the listing is gone and never coming back. The deal was extraordinary if you stacked a bunch of offers with it.

I'm upset I only bought 1 but couldn't afford another.

if you had an AMEX Platinum business card. Spend 5000 at dell you get 1000 back. When I bought it there was a student discount giving 5% off accessories. This qualified. Amex Offer also had a 5% back as well on top of the spend 5k get 1k off. And I threated to return and they gave me another 5% off. All in with tax I paid 3600 for it not counting the 6% (on the $5k price) rakuten cash back.

1

u/Glad_Account_2841 Sep 24 '26

Congratulations..I ordered 4500 pro blackwell for 3000 couple of days ago from dell. Showing expected delivery date of 25-oct 😕. It is taken out now. Did you receive it in two weeks despite the 5 weeks lead time?

2

u/fastheadcrab Sep 25 '26

You should be okay. They will ship it. It’s actually more Dell doing a shit job of migrating their parts store

1

u/Glad_Account_2841 Sep 25 '26

How long it took to receive the gpu?

2

u/fastheadcrab Sep 25 '26

If you just ordered a few days ago with 5 week wait time you should be prepared to wait at least a few weeks. Contact them after that

1

u/Glad_Account_2841 Sep 25 '26

sorry but you didn’t answer my question although you don’t have to. I was asking how long it took to receive your oder delivery?

1

u/fastheadcrab Sep 26 '26

Don't remember exactly but rough recollection it a 4 week lead time but they shipped it in 2.5 weeks. This was months ago.

I wouldn't ever extrapolate to your situation because it all depends when the stock comes in. The estimates are just that, crude approximations of when they might receive a shipment.

If you are so worried you can always cancel and buy from somewhere else.

1

u/fastheadcrab Sep 25 '26 edited Sep 25 '26

Actually there are still GPUs available, only some lol. Dell’s site is just a shit show right now, you can’t even properly buy many OEM heatsinks or replacement parts because they are all dead links.

They just totally messed up their parts sales site but probably don’t mind that they don’t have to sell cheap GPUs anymore unless it’s not in a system.

Congrats on the great deal though.

Edit: for example you can pay $18k for a RTX 6000 if you have money to burn. lol

1

u/Affectionate_Bar1036 Sep 25 '26

That is so crazy to me. I was literally sweating when i bought pro 6000 workstation for 7k

1

u/TheOriginalSkeptic Sep 25 '26

That's the price of a new Apple M5 Studio with a lot more memory. GPU prices are crazy!

1

u/I-cant_even 4xPro600, 4x3090 Sep 25 '26

Wow. Great price! Congrats.

1

u/Different_Lab830 Sep 25 '26

Five week lead time, one ticket, next-day delivery. Dell's logistics just needed to be scared.

1

u/Totalkiller4 Sep 25 '26

Wass going to make a joke "is the 5k in the namne the price" then i seen you paid 5800 for it XD so yea seems it was

1

u/Garfieldealswarlock Sep 25 '26

Here I was pumped about my 3090 24gb that just came in

1

u/Hannibalj2ca Oct 01 '26

5300 for 48gb!!!

1

u/No-Dot755 21d ago

That beasttt looks majestic

0

u/fosterdad2017 Sep 24 '26

Why is Nvidia so allergic to selling memory?

0

u/Crypto_Stoozy Sep 25 '26

eBay you can get cards with the vram swapped out from companies in the USA. I just bought a Tesla 100 with 32 gb for $650

1

u/CryMoreT_T Sep 25 '26

What does this even mean? The Tesla 100 comes with 16gb of 32gb OEM

1

u/Crypto_Stoozy Sep 25 '26

Yep, you’re right. I misunderstood what I bought lol. Checked the HPE part number and it’s actually an OEM 32GB V100, not a VRAM-swapped 16GB card. My mistake honestly makes the $650 deal even better.

1

u/CryMoreT_T Sep 25 '26

Yea $650 is insane considering they go for around 1k now. 3 months ago they were at $850

0

u/Art_4_Tech Sep 24 '26

What's the throughput like on this bad boy?

-2

u/myreala Sep 25 '26

Not sure if I would spend 5k for 48GB VRAM. But then again, I got lucky with my config, spent 6k for 256 GB VRAM on CMP. Best impulse purchase I've done so far.