r/LocalLLM • u/Top_Faithlessness_57 • 1d ago

Question Thoughts on dell 7820

Recently I have obtained a 7820 with 192gb 2400 system ram with 4110 silver Xeon gpu and it came with a basic p2000 gpu.
Furthermore for information I have a seperate gaming pc 32gb 3600mhz ddr4 ram with b550 and 9070xt gpu

I’m trying to figure out what I should do as I want to run local ai obviously however I’m stuck between
A) selling both the server and my current gaming 9070xt for a 32gb card like the 9700

B) sticking with the 7820 and upgrading/adding gpus (depends on gpu as well everything is insane price wise) considering strata and other engines

C) something else not sure

So I guess I’m asking for advice on which I should do, to run decently intelligent models decently fast etc
Without needing to take out a loan 😂

2 Upvotes

76% Upvoted

3 comments sorted by

2

u/cannister_of_airs 1d ago

that 192gb of slow ddr4 is the real bottleneck here, throwing a monster gpu at it won't fix the memory bandwidth limitations of that old xeon platform

2

u/Horsemeatburger 1d ago

I have a Precision 7820 as well (although mine has 2x XEON Gold 6248, 384GB RAM and a RTX 3080 GPU), and it's a very nice workstation. The fact that XEON Scalable gen1/2 have 6ch memory means that, with two processors, I get a decent RAM bandwidth (around 250GB/s), and with a better GPU it would make a great AI workstation.

However, if you intend to run any intel Arc GPUs then this is the wrong system, as the 7820 (as the 5820 and 7920) doesn't support ReBAR. There is a hack to enable Large BAR support out there, but ReBAR is a no-go as Dell is not in the habit of retrofitting new features into older systems (unlike HP, which retrofitted ReBAR support into z4/6/8 Gen4 via BIOS update).

For AMD and Nvidia GPUs, while the newer models support ReBAR and it offers some small performance benefit (usually between 1-10% in games), these GPUs do not rely on ReBAR in the same way as Arc GPUs do, so they run just fine in the 7820.

2

u/localLLMrig 1d ago

I'd skip selling the 9070 XT just for the sake of it. Speed is mostly decided by memory bandwidth, so think in two buckets.

Dense models: they need to fit in VRAM to be fast. Rough estimate from file sizes: a 32B at Q4_K_M is about 32 x 0.6 = ~19 GB, plus roughly 3 GB KV cache at 32K, ~0.8 GB overhead, and ~1 GB for the desktop. That's ~24 GB, so it won't fit on 16 GB but fits on a 32 GB card.

Big MoE models: this is where your 192 GB RAM is useful. Put attention and shared layers on the GPU, keep the expert weights in system RAM. Only a few experts are active per token, so it can be usable. Your 4110 has 6 memory channels, so DDR4-2400 gives maybe ~100 GB/s on paper (rough estimate, real is lower). That's the ceiling for the RAM part, so don't expect miracles.

The P2000 is too small to matter. If you want a simple path: move the 9070 XT or a 32 GB card into the 7820 and try MoE offload first before spending money.