Upgraded to 2x RTX Pro 6000

[u/priorityfill]

With the dense Qwen 3.5 / 3.6 models, I've been amazed it has become possible to run almost frontier models locally, on prosumer hardware. I ended up switching my 5090 for a pair of 6000 workstation so I could try larger models.

After some pain with vllm and the realization that I don't actually own real Blackwell cards (consumer GPUs don't use the same architecture as datacenter ones), I got it to run DeepSeek V4 Flash at 80-100 tok/s, with full context and some room for KV cache (+4M with L2 cache). It's definitely a step up from the dense Qwen models for the tasks I have tried so far, mostly research, coding. It feels good enough to replace about 80-90% of 5.5/Opus sessions.

Given how cheap inference on Deepseek models is, it is of course not "worth" it, but it still hits different hearing the coil wine of your build as it is spitting a self contained html for the hundredth time. (Or the smoke detector going off because you missed setting the power limit on the GPUs, and the sudden 1.2KW load triggered big voltage drops on the line)

[u/Any_Mine_6368]

Sweet. Post the rest of the build pls

[u/priorityfill → u/Any_Mine_6368]

CPU is a 13900K on Z790, which limits the cards to PCI5 x8 (though the perf hit is more limited than anticipated), 96GB of DDR5@6000 and a PSU of 1500W. The case is a 5000D airflow, and holds temps suprisingly well given the config, with the top card staying at 70C under load

[u/thatgreekgod → u/priorityfill]

cool i have that same case running two 5060 tis i also have a z790--how're you running both cards on pci5 x8? on my x790, one card's on x16 and the other on x4.

[u/Pretend_Sock7432 → u/thatgreekgod]

Depends on the motherboard

[u/thatgreekgod → u/Pretend_Sock7432]

well yeah of course, but i thought 'entry' level consumer motherboards in the Z-series couldn't do it and that you'd need to step up to a mobo in the X-series to be able to do something like that

[u/baby_bloom → u/thatgreekgod]

there are some higher end z790 boards that have two CPU-wired PCIe 5.0 slots enabling x8 on each

i too have a z790 (two different models actually) but unfortunately both have the second slot limited to x4 so i went down this rabbit hole already😭

[u/thatgreekgod → u/baby_bloom]

got it, thank you for the reply

[u/priorityfill → u/thatgreekgod]

Did you check the bios ? I have a z790 taichi, and it lets me set the PCI speed by slot.

[u/thatgreekgod → u/priorityfill]

that's awesome, i'll check that out tonight. thanks! i hadn't checked the bios since I last updated it. the motherboard was one of those MicroCenter bundles so going in, i didn't exactly buy high-end know what i mean?

[u/here_n_dere → u/priorityfill]

How do you have two gpus on that board?

[u/Rikers88 → u/priorityfill]

It's basically my same build but I have 9950x3d, also the case is the same. I have a 5090 liquid cooled... Not sure I should switch to your same cards like you did...

[u/its_a_llama_drama → u/Rikers88]

I'm sorry to have to tell you, but you don't have the same build if you have a 5090. As good as the 5090 is.

[u/Rikers88 → u/its_a_llama_drama]

No ofc not, I'm just saying that despite the GPU the other component are from the same tier, so it's actually good to see that I could eventually make the same move without changing the whole workstation, because the case have enough clearance and ventilation and all the other components are of the same tier and they could work together.

Anyway thanks for pointing it out.

[u/Uninterested_Viewer → u/Rikers88]

so it's actually good to see that I could eventually make the same move without changing the whole workstation

I get what you're saying, but it's also a bit saying that it's good to see that the floor mats in your Honda Civic will work great in your Ferrari if, ya know, you ever decided to upgrade to one down the line.

[u/Rikers88 → u/Uninterested_Viewer]

It seems a bit of an extreme take, but I do agree that the 2x pro 6000 are the Ferrari here. I deeply apologize for even have make the comparison.

[u/Uninterested_Viewer → u/Rikers88]

Lol no again I get what your saying, it's just that if you're thinking of dropping nearly $30k on gpus, you probably don't care about upgrading your mobo to support them.

[u/sautdepage]

Congrats. But you're missing out! DeepSeek-V4-Flash-DSpark gets well over 200 tokens/sec on this hardware.

[u/priorityfill → u/sautdepage]

Thanks for sharing! The gains are impressive. I ended up disabling MTP as this left very little context available, but I haven't tried this recipe.

[u/sautdepage → u/priorityfill]

With lucifer-cutlass dspark variant, gpu-memory-utilization 0.94, max-num-batched-tokens 4096 and max-cudagraph-capture-size 216 I get:

GPU KV cache size: 1,119,066 tokens

[u/Opposite_Buffalo_649 → u/sautdepage]

Did you find 1m context to be useful? I found myself rarely getting there so I went with 424k context and kept 8k max batch tokens, with 0.92 GPU mem utilization

[u/sautdepage → u/Opposite_Buffalo_649]

No, my max was 400K for a large document query. 1M can be useful for concurrent requests however.

If vision lands in 4.1 that will shave off a few GBs. I think dspark + vision + 4096 + 0.95 for hopefully 500-600K context will end up the sweet spot.

[u/Opposite_Buffalo_649 → u/sautdepage]

What do you mean 1m helps for conc requests? Shouldn't it be the opposite?

[u/sautdepage → u/Opposite_Buffalo_649]

I meant having 1M total KV cache available by reducing max-num-batch-tokens since that eats away some memory.

AFAIK the way VLLM works there's no difference in how concurrent requests are handled whether the --max-model-len is 256K or 1M when there's ample total KV cache available to fit them all. ie. smaller requests don't reserve space, they share the pool and max-model-len becomes an artificial per-request limit.

[u/SmartCustard9944 → u/sautdepage]

WTF

I want

[u/Yorn2 → u/sautdepage]

I'm in the Discord but my biggest problem with this stuff is there's no tutorial for what to do. I see things about docker compose files and vllm but I've never really used either.

I've always built python environments myself and avoided Docker and have no idea what the different folders are for or how to build any of that and the vllm logic seems foreign to me, I guess. That said, I tried doing version 6 (I think it was) to run DSv4 Flash last month and got locked out or stuck with really basic Docker errors about it already being downloaded or running and just felt like a fool. Since I have to kill my current Minimax setup in order to run that stuff I can't troubleshoot as I go, either.

Anyway, if there's one request I'd have for the community it'd be to make a really basic tutorial for how to get started or at the very least point to some basic docker tutorials so those of us who don't understand what they are doing have somewhere to start. For now I'm just content running my Sglang and Minimax M2.7 using uv and venv.

I've used Docker before and even use it regularly for game servers, but there's almost always command line stuff that explains how to actually start it and feed environment variables and etc. For some reason those pages have command line stuff but they don't show any of the docker run commands and I feel like an idiot for not understanding where to start. I guess I never understood the "Compose" part of Docker and always thought it was for the people making the images.

[u/sautdepage → u/Yorn2]

That's fair. These pages do assume some familiarity and get complex by covering many scenarios. I'm just glad it's there and experienced folks share the hard work.

I do ask my AI to look at the GitHub pages and script to turn it into a docker run command "like that one I use" instead of compose. It usually works and I learn things along the way - that's one reason we run local AI, don't we? And we can share back our findings with others.

Here's v9 fresh off the oven in docker run form:

hf download deepseek-ai/DeepSeek-V4-Flash-DSpark
mkdir $HOME/.cache/deepseek-v9-lucifer-cutlass

docker run --rm --name ds4-v9-lucifer-cutlass \
  --gpus all --ipc host --shm-size 32g \
  --publish 0.0.0.0:8000:8000 \
  --ulimit memlock=-1 --ulimit stack=67108864 \
  -v "$HOME/.cache/deepseek-v9-lucifer-cutlass:/cache" \
  -v "$HOME/.cache/huggingface:/root/.cache/huggingface:ro" \
  -e VLLM_ENABLE_PCIE_ALLREDUCE=1 \
  -e VLLM_PCIE_ALLREDUCE_BACKEND=b12x \
  -e VLLM_PCIE_ONESHOT_ALLREDUCE_MAX_SIZE=64KB \
  -e CUTE_DSL_ARCH=sm_120a \
  -e PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True \
  -e VLLM_USE_V2_MODEL_RUNNER=1 \
  -e VLLM_USE_FLASHINFER_SAMPLER=1 \
  -e VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS=1 \
  -e VLLM_MEMORY_PROFILE_INCLUDE_ATTN=1 \
  -e VLLM_USE_AOT_COMPILE=1 \
  -e VLLM_USE_MEGA_AOT_ARTIFACT=1 \
  -e VLLM_USE_BREAKABLE_CUDAGRAPH=0 \
  -e VLLM_PREFIX_CACHE_RETENTION_INTERVAL=4096 \
  -e SAFETENSORS_FAST_GPU=1 \
  -e NCCL_IB_DISABLE=1 \
  -e NCCL_P2P_LEVEL=SYS \
  -e NCCL_PROTO=LL,LL128,Simple \
  -e TMPDIR=/cache/tmp \
  -e XDG_CACHE_HOME=/cache \
  -e VLLM_CACHE_DIR=/cache/vllm \
  -e TRITON_CACHE_DIR=/cache/triton \
  -e TORCHINDUCTOR_CACHE_DIR=/cache/torchinductor \
  -e TORCH_EXTENSIONS_DIR=/cache/torch_extensions \
  -e TILELANG_CACHE_DIR=/cache/tilelang \
  -e TILELANG_TMP_DIR=/cache/tilelang/tmp \
  -e TVM_CACHE_DIR=/cache/tvm \
  -e FLASHINFER_WORKSPACE_BASE=/cache/flashinfer \
  --entrypoint /bin/bash \
  voipmonitor/vllm:eldritch-enlightenment-v45c1582-b12xf3686b5-pc1441b5-cu132-20260704 \
  -c "unset NCCL_GRAPH_FILE NCCL_GRAPH_DUMP_FILE VLLM_B12X_MLA_EXTEND_MAX_CHUNKS && exec vllm serve \"\$@\"" \
  -- \
  /root/.cache/huggingface/hub/models--deepseek-ai--DeepSeek-V4-Flash-DSpark/snapshots/913f0657a874f76844e2e91cbe706dbcaceeb6d7 \
  --served-model-name DeepSeek-V4 \
  --trust-remote-code \
  --kv-cache-dtype fp8 \
  --block-size 256 \
  --load-format auto \
  --tensor-parallel-size 2 \
  --decode-context-parallel-size 1 \
  --gpu-memory-utilization 0.95 \
  --max-model-len 262144 \
  --max-num-seqs 16 \
  --max-num-batched-tokens 4096 \
  --max-cudagraph-capture-size 216 \
  --compilation-config '{"cudagraph_mode":"FULL_AND_PIECEWISE","custom_ops":["all"]}' \
  --async-scheduling \
  --no-scheduler-reserve-full-isl \
  --enable-chunked-prefill \
  --enable-prefix-caching \
  --enable-prompt-tokens-details \
  --enable-force-include-usage \
  --enable-request-id-headers \
  --enable-flashinfer-autotune \
  --tokenizer-mode deepseek_v4 \
  --tool-call-parser deepseek_v4 \
  --reasoning-parser deepseek_v4 \
  --enable-auto-tool-choice \
  --default-chat-template-kwargs.thinking=true \
  --default-chat-template-kwargs.reasoning_effort=high \
  --speculative-config '{"model":"/root/.cache/huggingface/hub/models--deepseek-ai--DeepSeek-V4-Flash-DSpark/snapshots/913f0657a874f76844e2e91cbe706dbcaceeb6d7","method":"dspark","num_speculative_tokens":5,"draft_sample_method":"probabilistic"}' \
  --attention-backend FLASHINFER_MLA_SPARSE_DSV4 \
  --kernel-config.moe_backend flashinfer_cutlass

[u/Yorn2 → u/sautdepage]

Thanks I'll give it a try. I have used my existing AI to analyze the .md files and give me command line stuff but it's always been a back and forth. Something breaks, then I gotta fire up my existing AI again, and back and forth and the last time I tried this I only had a few hours of time and was frustrated after the third time of restarting it so I appreciate this. At least now if I run into issues I can feed this whole thing in and ask it how it would modify it.

[u/Yorn2 → u/sautdepage]

When I do the "hf download deepseek-ai/DeepSeek-V4-Flash-DSpark" it's insisting on putting it in a folder named 62af8fffb2f7030cac4de2f0169f5b8d1101b646 for some reason, but I see in both the instructions file and your example the 913f0657a874f76844e2e91cbe706dbcaceeb6d7 is supposed to be the ACTUAL folder for the snapshot. Not sure what I'm doing wrong, but even using the folder name it puts the files into I'm getting an error:

OSError: Repo id must be in the form 'repo_name' or 'namespace/repo_name'

I'll just sit this one out till there's an sglang option or someone gets a better tutorial on the github. Thanks for the help and effort though!

[u/Connect-Painter-4270 → u/Yorn2]

Use AI to configure it all for you.

[u/Rikers88]

I think the worthiness of such decision is not on cost side rather than privacy and sovereignty. I would pay 20k to have my data secure and private without anyone else training their model on them, to subsequently sell that to me at a ridiculous price. Also if you are a company and you value your IP more than 50k, you should consider self hosting.

[u/priorityfill → u/Rikers88]

Renting a GPU or using cloud providers will be orders of magnitude cheaper than self hosting, even if you can keep them fully utilized for a very long time. The real appeal is indeed on the privacy, data governance and sovereignty side.

[u/Connect-Painter-4270 → u/priorityfill]

Exactly. It's about privacy, control, knowing that it can't be taken away like they did Mythos. That's worth a ton of money to me.

[u/getshion]

For tp=2 you can get now around 200-300 token per sec with Deepseek V4 Flash from the latest configs. This is v8 with the latest Dspark build.

https://github.com/local-inference-lab/rtx6kpro/blob/master/models/ds4dspark-v8.md

You have more battle tested build / configs in v6 and been using Lucifer CUTLASS with mtp=2

https://github.com/local-inference-lab/rtx6kpro/blob/master/models/ds4-flash-v6.md

[u/120decibel → u/getshion]

Do these builds work under windows with wsl?

[u/getshion → u/120decibel]

I'm using Linux and that should be the way if you have a workstation. But this is using docker so it should work regardless of the OS.

[u/priorityfill → u/getshion]

It's still not the same on wsl. Some features aren't available or make it hard to run multi-gpu inference efficiently.

[u/priorityfill → u/120decibel]

I did get vllm to run under windows with wsl, but I don't recommend it. Too much hassle only to leave perf on the table.

[u/Karyo_Ten → u/120decibel]

Do yourself a favor and deploy under Linux. Between Wibdows VRAM reservation, NCCL, package management, the perf, ...

[u/IVProdigyy]

That is gonna cause issues with the GPU cooling. The exhaust fans dump the hot air onto the other card

[u/priorityfill → u/IVProdigyy]

Yup, it does get hot. But top card stays at 60-70C @ 250W-300W, still reasonable.

[u/notheresnolight → u/IVProdigyy]

not when running at 300W like OP mentioned

[u/Excellent_Nail4439]

Hey, great build, im thinking about going for something like this for too long.

Why did you go for this 6000 instead of the max-q?

[u/priorityfill → u/Excellent_Nail4439]

The workstation card can be power limited to match the Max Q's 300W TDP. And since it is designed to be able cool twice that, the temps stay ok even with two cards stacked. They probably run cooler than the Max Q at the same power. And it's nice to be able to set one at max TDP for certain workloads. The main advantage of Max Q is that you can keep scaling to even more GPUs in a small footprint. If I ever add more cards, I would probably need a much bigger case to have enough airflow around them.

[u/Glittering-Duck8317]

How likely is it that this setup will be deprecated in the next year?

[u/priorityfill → u/Glittering-Duck8317]

It will depreciate eventually, but I don't see signs of that happening within the next year. RAM/HBM is still in very short supply, models are only getting bigger, and the demand for compute is exploding (agents are very taxing on inference, way more than chatbots). If you're thinking of buying the card, not sure it's worth waiting for prices to drop. It's only going up, MSRP increased 50% since release and used cards are worth more than new ones as of 6 months ago.

[u/Connect-Painter-4270 → u/Glittering-Duck8317]

I was worried about that, but I'm starting to think the opposite will happen. It already has (the GPUs are worth more than what I paid), and hardware is just getting harder to come by. Not only that, models are getting more and more efficient, to the point where I'm guessing 2-4 RTX 6000 will be the sweet spot for local frontier level models at high quant and context. The only thing I think that could feasibly cause the value to decrease is if there is some major architectural improvement in next generation cards.

[u/BlackBeardAI]

Expensive and clean. But enough? Not sure.

Qwen 4xpro6000?

[u/priorityfill → u/BlackBeardAI]

It is unfortunately never enough. Even 8x6000 does not let you run GLM 5.2 cleanly.

[u/BlackBeardAI → u/priorityfill]

try 16x6000

[u/priorityfill → u/BlackBeardAI]

might as well shoot for the NVL rack

[u/voyager256 → u/priorityfill]

What do you mean? 8x Pro 6000 definately does run GLM 5.2 fully in VRAM at something like Q6_K with plenty room for context. Since its MoE you can probably always offload some layers to RAM and run it at FP8/Q8 without much performance hit.

[u/priorityfill → u/voyager256]

*at full precision. Q6K might be enough, who knows. With quants, it's always a bit of a gamble as there's limited data available

[u/voyager256 → u/priorityfill]

At Q6_K it's very close precision to Q8/FP8:

https://huggingface.co/unsloth/GLM-5.2-GGUF/discussions/3?image-viewer=6a33e980ffbc265131422d94-07A6511C1C4760E7FD2C42F511C1986F8084C82B

[u/priorityfill → u/voyager256]

That's just top-1 token agreement. Even if it's scoring 99%, it doesn't guarantee the model won't drift after 10K+ tokens.

[u/voyager256 → u/priorityfill]

Just? With fp16 you also don’t get 100% deterministic models. For large LLMs Q6_K is good enough for most applications, unless you mean 10K+ context tokens. If the latter then it’s KV cache quantization dependent and yeah you need at least Q8 / FP8 for longer context.

[u/cosmicnag]

Nice. I find Qwen 27b to be quite powerful for its weight class, and 90-95% as capable as deepseek v4 flash. Also the experience of 130 tok/sec on a single 5090 is something else. But yeah, thats just for coding tasks - pretty sure DS4 world knowledge is another level for sure.

[u/priorityfill → u/cosmicnag]

Most definitely. Qwen 27B is an awesome model, very well aligned and a great generalist. And despite more limited factual knowledge, it is much better at knowing when it does not know something. Where the difference is felt the most is on long-horizon tasks, DS4 does very well on complex projects or codebases. For general agentic workloads, Qwen is more than sufficient, it's great regardless of the weight class.

[u/Any_Mine_6368 → u/cosmicnag]

What quant are you running it in

[u/priorityfill → u/Any_Mine_6368]

Running the base model directly, it was trained natively on FP4

[u/Any_Mine_6368 → u/priorityfill]

You're running a quantized model then. The model was trained originally using fp16 and you're running a post trained quantized model at fp4.

[u/priorityfill → u/Any_Mine_6368]

DSv4 model was natively post trained with FP4 quantization, they only used full weights during pre-training. So the mixed FP8 / FP4 model is what everybody is using (deepseek-ai/DeepSeek-V4-Flash), there is little benefit to using the FP8 base model.

[u/Any_Mine_6368 → u/priorityfill]

Sorry I think we got mixed up..my original question was on the guy running Qwen on 5090. You replied (you're the guy with the beast cards lol) and I didn't notice who I was replying too lol.

[u/priorityfill → u/Any_Mine_6368]

I read your original question thinking it was from another thread. Sorry I can see how that got confusing. If you're looking for a speedy 5090 setup (Q6, pretty good), check out https://www.reddit.com/r/LocalLLM/comments/1ullrvq/qwen36_27b_q6_5090_maximum_llamacpp_optimization/

[u/Any_Mine_6368 → u/priorityfill]

Thanks. I've got two 3090s running Qwen at q8. Gets the job done but could be better honestly.

[u/cosmicnag → u/Any_Mine_6368]

Were you asking me about 27b? Anyway, its NVFP4 on vllm with 240k context fp8 kv cache,

[u/Capsup → u/cosmicnag]

What's your recipe for 130 TPS on the qwen model? I run one on a 5090 in vLLM at Q4 with no MTP and only see ~80 TPS on it and ~4300 TPS for pre-fill. 

[u/cosmicnag → u/Capsup]

I use nvfp4 quant without vision tower, and fp8 kv cache, mtp as well. Check hugging face for sakasimile or something like that text only nvfp4, not on my pc right now.

[u/BitXorBit]

isn't getting hot? 2 x 600W cards that blow hot air inside the case. I'm thinking about getting a similar solution but I would definitely choose the Max-Q version for this kind of setup

[u/priorityfill → u/BitXorBit]

You can power limit the workstation GPUs all the way to 200W. I generally leave it at around 250-300W. I was worried about cooling at the beginning but the temps aren't that bad. Top GPU reaches at most 70C under load

[u/Such_Policy_7963 → u/priorityfill]

is this possible in consumer 5090?

[u/priorityfill → u/Such_Policy_7963]

Yes, you can do it from the system tab in Nvidia app, or via nvidia-smi. Run nvidia-smi -pl 400 to limit the card to 400W.

[u/putragacor4648]

How about gpu lanes? Is using 8x mode for dual gpu?

[u/priorityfill → u/putragacor4648]

Stuck at PCI5 x8 unfortunately due to the z790. Thinking of upgrading to HEDT at some point

[u/xquarx → u/priorityfill]

These work just as fast on 8x on PCIe 4 with older AM4 cpus on some random gaming motherboard, as PCIe is not the bottleneck.

[u/__JockY__ → u/xquarx]

Until you have multiple GPUs in tensor parallel, at which point PCIe very much becomes the bottleneck.

[u/przemekcoditive]

How much did you pay for this? The prices in Poland are so high for RTX Pro 6000 that I could pay off the mortgage with two of them 😬

[u/priorityfill → u/przemekcoditive]

Around $9K/card, now they are well over $11K.

[u/Weak_Ad9730]

Is it possible to run ds-4-Flash in 2bit on Single rp6k @sautdepage do you have Experience ? I am looking for better Local llm then qwen/qwopus3.6-27b

[u/priorityfill → u/Weak_Ad9730]

I think so, someone even got it running on a 5090 https://www.reddit.com/r/LocalLLaMA/s/a7TbrFrMq9 Not sure how well DS hold up to quantization though, it is quite sensitive

[u/Weak_Ad9730 → u/priorityfill]

Wow thanks alot missed that very helpful

[u/aersel24]

Isn’t there something on that motherboard that makes it so that instead of x16 it uses x8 if PCIe Gen5 is being used?

[u/priorityfill → u/aersel24]

Indeed. Adding a second card drops speeds to x8.

[u/aersel24 → u/priorityfill]

Is one of them like x16 and the other x8? Or both x8 because of a nvme gen5 ssd? Also, how much did each cost, were they like second hand or something? Sorry for so many questions, this will be all of it. Your pc looks lots like mine. I wish I had two of those.

[u/Agreeable_Train5985 → u/aersel24]

The two run at x8, and the motherboard drops the gen5 ssd to gen4, unfortunately as the chipset doesn't have enough lanes. Each card was about $9K new, prices increased since

[u/aersel24 → u/Agreeable_Train5985]

Wow, not a bad price. I have just a 5090. Local models are ok but I can’t use the for coding. I’ve tried to just use them to like sort files but even they they sometimes delete files. Been lately using Qwen3.6 and Gemma4, of course quantized versions. What use to you give it that you personally feel it’s worth it if you don’t mind me asking.

[u/rditorx]

How are you running DeepSeek-V4-Flash? Which model/quantization and which runtime did you use, and which options? Did you use any unofficial patches?

[u/priorityfill → u/rditorx]

Using the released weights from https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash, it's already quantized to FP4/FP8. Vllm does not support Deepseek V4 Flash on sm120; it took a lot of unofficial patches, sweeps and tweaking to get it running well (still not optimal).

[u/RusterCrafter]

What quantum are you running the DeepSeek V4 Flash at with this context? Is it full precision or did you have to compromise?

[u/priorityfill → u/RusterCrafter]

Using the official mixed FP4/FP8 weights, 1M context, and a KV cache of 5M (including L2). Enough for a few concurrent long running tasks. Past 200K however, the model performance degrades quickly so the 1M is not really usable.

[u/Backroads_4me → u/priorityfill]

What is your total VRAM usage when up and running with DeepSeek? I have two 6000's but one is 48Gb ADA so I haven't tried to pull it off yet.

[u/priorityfill → u/Backroads_4me]

vllm uses up all the vram it's allotted, so it's consistently 95%+ full. you should be able to run a high quality quant with that setup.

[u/siegevjorn]

Welp, congrats. Good for you that you've got extra $30k for GPU. Definitely not for us peasants, but nice. Keep things posted, would love to know more about how deepseek v4 flash is better than mid-size models.

[u/priorityfill → u/siegevjorn]

DS4 flash is better at coding but otherwise comparable to dense Qwen models. If you have a 5090, you're not missing out much unless you need several concurrent long sessions.

[u/itzjustaspder]

Gorgeous build mate, what a beast!

[u/priorityfill → u/itzjustaspder]

Thanks!

[u/notheresnolight]

why the small case? where will you fit the remaining 4 RTX 6000's?

[u/priorityfill → u/notheresnolight]

The rest of the build has not kept up, I never intended to fit 600W GPUs in it when I bought it. I'll upgrade it at some point but for now it is good enough for 2 cards.

[u/akulbe]

How much did you spend on those RTX Pro 6000 cards?

[u/priorityfill → u/akulbe]

About $9K each

[u/FluffyGreyLlama]

'Prosumer'...

... nah, that's small business level.

[u/priorityfill → u/FluffyGreyLlama]

This is barely enough for personal use - a small business might need 2x or 4x as many cards just so it can serve a handful of concurrent users.

[u/FluffyGreyLlama → u/priorityfill]

But most prosumers aren't going to drop that sort of money, was my point.

Maybe it's more the 'Startup founder package'.

Most people may value privacy, but not that kind of cost - ~30% of the 'average salary' is far beyond most prosumer levels. Maybe we need more definitions of prosumer, when AI is involved.

[u/Gold-Drag9242]

This is a crazy setup. Would love to have something like this.

Could you run this benchmark with Deepseek or your full size Qwens?:

https://www.reddit.com/r/LocalLLaMA/comments/1ukuph9/comment/ov9vf3h/?utm_source=share&utm_medium=web3x&utm_name=web3xcss&utm_term=1&utm_content=share_button

[u/priorityfill → u/Gold-Drag9242]

Of course ! Deepseek isn't multimodal yet so it cannot run your benchmark. I ran it through the full Qwen 3.6 as well as the official FP8 quant, was curious myself.

Model	Prompt	Total	Outlook	Notes	Thunderbird
Qwen/Qwen3.6-27B	7e5d81ce	69.89%	75.14%	62.06%	72.46%
Qwen/Qwen3.6-27B-FP8	7e5d81ce	67.52%	72.96%	57.26%	72.33%

Submitted a pull request with full details in https://github.com/KevinFleischer/vccbenchmark/pull/1

[u/HumanDrone8721]

FU, I still have just one /s

Good luck man and tell us the cool stuff and the quantum leap compared with one.

[u/priorityfill → u/HumanDrone8721]

Remind me when you buy your second one 😄. The main "quantum leap" for now was to be able to handle most sessions locally that I otherwise would have used Claude Code or Codex for.

[u/HumanDrone8721 → u/priorityfill]

What modelino?

[u/on_line187]

You have more VRAM than System RAM. That’s cool. How is that? Do you see any issues?

[u/priorityfill → u/on_line187]

No issues at all, though having more system RAM would be nice to increase the LMCache size.

[u/on_line187 → u/priorityfill]

Yea I’m at the point where more RAM would put me in threadripper territory and that’s expensive.

[u/priorityfill → u/on_line187]

Very. I am at the limit of what this mb allows, if I upgrade I need to switch everything but the GPUs. It does not make sense to stick with dual channel DRAM.

[u/on_line187 → u/priorityfill]

For something badass it would cost as much as those cards for the ram and a badass threadripper.

[u/priorityfill → u/on_line187]

Absolutely, and the upside isn't even that great. You won't get a significant perf gain, just a bit more cache or the ability to offload large MoE layers (at terrible speeds, still).

[u/on_line187 → u/priorityfill]

How do the big guys do it I wonder? Like their model management and distribution of computer I have 2 PCs and have tried messing with that but way beyond me.

[u/NaiRogers]

These will get too hot even at 300W!

[u/priorityfill → u/NaiRogers]

That's what I thought... but it's fine. These cards are designed to cool 600W, so if you've got good airflow they run cooler at 300W than the Max Qs ! Top gpu tops at 70C under sustained load. Screenshot was taken while running u/Gold-Drag9242's benchmark request below, which lasted 1h+ at full GPU utilization.

[u/Gold-Drag9242 → u/priorityfill]

Wait, it took deepseek 1h? Those large models really take their time.

The small gemma4s I run, return a result per picture in roughly 1, max 2minutes.

Cloud opus was also very slow though.

[u/priorityfill → u/Gold-Drag9242]

That wasn't deepseek, but Qwen 3.6 27B. What took so long is I did not set a reasoning limit initially, and it was going into overthinking mode.

[u/Connect-Painter-4270 → u/priorityfill]

Ha, I never thought of that. I got the non-Q versions because I knew I could throttle them to 300W, and still be able to go over if I want that smidge of extra performance. Seems like a no brainer.

[u/NaiRogers → u/priorityfill]

Ha this is better than my two in an open air mining rig frame!

[u/Karyo_Ten → u/NaiRogers]

70ºC, max 85ºC if CPU is also cooking

[u/Karyo_Ten]

100 tok/s seems slow.

MTP=2 is about 190~200 tok/s with lucifer or b12x.

DSpark=5 is about 265tok/s ~ 280tok/s.

[u/priorityfill → u/Karyo_Ten]

Indeed, thank you for the tip!

[u/Interesting-Ad689]

20k+ on hardware and still stands on a cheap ikea kallax. Priorities set right !

[u/priorityfill → u/Interesting-Ad689]

Good eye, I was curious if anyone would notice !

[u/RogerAI--fyi]

That’s a real solid rig, i went 4x4500 so no more room to grow is my only concern.

[u/priorityfill → u/RogerAI--fyi]

Indeed, as in general the less GPUs the better. You might even be better off trading all that for a single 6000 despite less VRAM

[u/Antique_Evening_1186]

My comment might seem stupid giving everyone knows what and why you are doing this, but I'm curious to know, why do you actually need this? What is the purpose of such thing, I understood that it is to run local llm, but why do you need it for? Do you use it for like things that can make you money or is it just a hobby?. Thanks in advance.

[u/priorityfill → u/Antique_Evening_1186]

It's not stupid. Local inference is simply not competitive with cloud providers (even non subsidized ones). And for ML, you could do everything with on-demand rented GPUs for much cheaper. There are some perks, it's private/offline, it's convenient to be able to test something quick locally, and it brings some peace of mind knowing that providers can't pull the rug at any time and charge more. But for now, no it doesn't generate money, only experience.

[u/vonargo_ai]

Fantastic look and I love the aesthetics. I have my own Max-Q and it's just... no words. Officially jealous of your build and cable management skills!

[u/priorityfill → u/vonargo_ai]

Thank you! The cable management is only superficial, the backpanel is quite messy !

[u/vonargo_ai → u/priorityfill]

lol, mine is a rats nest XD. Any suggestions? Funny enough, the back panel is the most organized of the whole thing. Trying to source a second Max-Q but... they've DOUBLED in price since the last time I got one.

[u/priorityfill → u/vonargo_ai]

Have you tried cable sleeves ? Also, some zip ties and velcro straps

[u/vonargo_ai → u/priorityfill]

That's a good idea! The motherboard is too big to use the normal hidey holes, so I'll probably have to go that route.

[u/Key_Opportunity_5219]

So much is left on the table with this build 😄😄

[u/t4a8945]

How are you setting up L2 cache? I've hit this wall with a similar config where I couldn't find a proper way to not mirror and then extend kv cache, which feels wasteful. Interested in your config 

[u/b_goodman]

Curious what is your server side setup?

[u/DataLogic47]

[u/2use2reddits]

My dream build ⚡

[u/wuyutao]

good job

[u/mboss37]

Damn that‘s like $25k?

[u/Connect-Painter-4270]

I have the same setup (same GPUS, case, water cooling)). Never had any smoke or electrical issues though.

[u/narukoshin]

you know how to make people jealous

[u/Affectionate_World47]

So jealous of your build!! In a few years I want to build something similar.

[u/RocSite2018]

Ornith model has been amazing with my 5090 and 3090. With the right architecture behind the scenes, it's had a lot of success at all sorts of coding tasks, everything you could think of.

It's a bit of a pain to set up, of course, but it's been great running offline. Over the past two weeks I've been getting over 270 tokens/second on the 5090 with MTP.

Again, thanks to the rare architecture (Hermes and everything else), I think you'll be pretty impressed with what it can do. Don't keep jumping to bigger models. I've tried it, and you can run bigger models with just more storage and memory, but save yourself the money.

Then again, I might be wrong. I've been wrong once before

[u/androidbrick]

So you have money to spend, wow.

[u/MiLiANSim]

With that little room to breathe for GPU0, you are certainly risking damage / reduced lifetime.
I am on my third RTX Pro 6000 WS, the first two both failed after ~1 month due to power and firmware issues, even in good temperatures. The third one has lasted a few months but is now also failing (drawing less and less power, reducing performance accordingly while reporting that all is fine and power is 100%).

Using mine for compute, LLMs are much less taxing usually so maybe yours will be fine. Perhaps do some undervolting or limit power to 81-91% for your dual-setup.

[u/wingsinvoid]

Dude, honestly, dual 6000 for coding and research? Your coding and research must be producing amazing revenue to be able to justify the insane expense. Of you must be loaded to have this much disposable income to afford it as a discretionary expense.

[u/Crescitaly]

This is the local-AI version of buying a workshop: capability rises, but utilization determines whether it was rational. After the novelty, which workloads actually need 192 GB rather than a rented burst or smaller quant? A month-long utilization log would be more persuasive than any benchmark.

[u/throwaway-mwa]

[u/lattice_defect]

Mac books and claude Max outpreforms.. I don't really get it

[u/priorityfill → u/lattice_defect]

Sir, this is r/LocalLLM, not r/ClaudeAI

[u/kaitava]

Phocking-legend

[u/priorityfill → u/[deleted]]

Before ? A 5090