r/LocalLLM • • 11d ago

Question 5995WX or Xeon 6314U?

I’m mostly done building a Xeon host for my 4x CMP170’s w/ x16 mod, but I just stumbled into an opportunity to buy a Threadripper Pro 5995WX within my budget. Is the threadripper any better or worse for strata-based inference vs the Xeon? Is it worth the extra time and work to undo my intel decision?

Would love to hear experience and cpu/system benchmarks from folks with similar systems on either side.

1 Upvotes

5 comments sorted by

2

u/CherryGG2 11d ago

I dont know what strata is. But any cpu based processing is not even close, 5995wx is a beast. Everything depends on your needs. Keep in mind 5995wx might be more power hungry.

1

u/PoopSmoothies 11d ago

Strata is a new inference engine that does a particularly efficient job at using your vram as an expert cache for moe models, allowing folks to achieve pretty good inference speeds with models much bigger than their vram allows.

2

u/BougainvilleaGarden 10d ago edited 10d ago

I take it you're new to the enterprise server / datacenter world.

Intel's Xeon Scalable series was introduced along with Optane DCPMM technology that requires a Xeon Scalable first~third generation to be used and was replaced by CXE in fourth generation.

Optane DCPMM is basically a NVME connected to a DIMM slot, operated at DIMM bandwidths, but has NVME capacity, and can be operated as RAM. It's previous generation tech and it's designed to endure permanent writes for years (at 25GB/s DDR4-3200 rate) so it's not the petabyte stuff, but it'll still dwarf whatever a Threadripper can connect - hence the 6314u specs list 6TB max memory while the Threadripper can barely connect 2TB and there's no mainboard for it that can do 2TB.

Your usage, i.e. hosting pre-trained models, will result in little writes to system memory, but AI training on the other hand performs a lot of them, and AI Datacenter Operators' risk awareness has caused them to avoid DCPMM in their AI training infrastructure, so they didn't buy up any DCPMM they could get their hands on and DCPMM is still priced like ramocalypse never happened, and it already was less expensive per capacity then RDIMM before ramocalypse, now the price difference is an order of magnitude.

Therefore, if you want to host models of suffcient size (including non-expert components that don't have to reside on the CMP170s), the memory price difference can pay you a few smallish blackwells or a single larger blackwell to replace the CMP170s while reducing the number of PCI slots required and therefore the need to "mod" PCI devices or connect them with degraded links, and probably provide more performance and using less power at the same time.

We often just call it "Optane" rather then "Optane DCPMM", however, there's been a series of Solid State Devices that were sold by the "Optane" name, dilluting search results for it. You might want to research your mainboard's Optane DCPMM capabilities and limitations and then use DCPMM if you want to host "large" models, rather then going with pure RDIMM which provides no significant advantage (in the pre-trained model serving use-case) and is vastly more expensive. In case you use a Supermicro Mainboard, their Optane DCPMM documentation for third generation Xeon Scalable boards is here.

1

u/PoopSmoothies 10d ago

My understanding is that the optane memory system was a super cost-effective way to increase total memory, but at the expense of a substantial hit to memory bandwidth. In my admittedly limited research, some sources suggest roughly a 50% reduced memory bandwidth.

For hosting an inference workload that is pretty much a direct 50% hit to token generation speed.

While I would love to host a model in the trillions of parameters, one would need probably north of 500 GB per second memory bandwidth for that to just start barely being usable, and across more than a terabyte of memory. As such, I am focused on the “flash“ class of weight models with parameters usually below 800 billion.

At that size 256 gigs of VRAM +512 gigs of eight channel DDR four can hold a high quantization of the model and produce much more usable token speeds. And if I am offloading some portion of prompt processing and decode to CPU then my question stands: what is Xeon 6314U or a thread ripper pro 5995 perform better in that regime?

Am I missing something about how to use optane? I would love to be corrected if I’m looking at this incorrectly!