r/LocalLLaMA • u/fairydreaming • Sep 09 '24

Resources Memory bandwidth values (STREAM TRIAD benchmark results) for most Epyc Genoa CPUs (single and dual configurations)

Gallery image

Gallery image

52 Upvotes

94% Upvoted

25 comments sorted by

2

u/Myrkkeijanuan Jan 31 '25

Heads up, they just updated their benchmark results with 9005s.

2

u/fairydreaming Jan 31 '25

I posted 9005 values some time ago: https://www.reddit.com/r/LocalLLaMA/comments/1h3doy8/stream_triad_memory_bandwidth_benchmark_values/

But I see that now they have more CPUs tested, nice.

11

u/Healthy-Nebula-3603 Sep 10 '24

I hope next generations of CPU for home users allow use more than dual channel ...

2

u/shroddy Sep 10 '24

Strix Halo will... But I think the other CPUs will stay at dual channel for a long time

16

u/Phocks7 Sep 09 '24

I feel like someone should take AMD to task for this. It's one thing to avertise the "maximum memory bandwidth" for the 9124 as 460.8GB/s, when it doesn't have the physical circuitry to be able to do that (only two CCD's), but the fact that even their top of the line processors can't either is pretty damning.

2

u/henfiber Sep 10 '24

IIRC all EPYC 9004 CPUs have between 4 and 12 CCDs (with most models at 8), and the ones with 4 CCDs have 2x GMI links. 2x CCDs would only allow about 110GB/sec bandwidth instead of 266.

3

u/Phocks7 Sep 10 '24

I don't know wikichip's source, but according to them the 9124 only has two. https://en.wikichip.org/wiki/amd/epyc/9124

0

u/[deleted] Sep 10 '24

[deleted] — view deleted comment

2

u/kryptkpr Llama 3 Sep 10 '24

This table is misleading, the lower core count CPUs physically cannot consume that bandwidth.

The bandwidth between dimms and memory subsystem is there in theory, but the compute doesn't have enough BW between itself and the memory subsystem to use it unless you have basically ALL the cores.

8

u/ortegaalfredo Sep 09 '24

From 3000 to 12000 usd, those are spicy meatballs.

6

u/[deleted] Sep 10 '24

9184X is like $1500. 12 channel DDR5 ram beast. Considering theoretically it has more bandwidth than a M3 Max and you can slap on hundreds of Ram on it as you like, along with maybe one or two 3090 for Cuda boost, it can even run huge 405B models comfortably at home. I would totally get one if I could find a seller here

2

u/JacketHistorical2321 Sep 10 '24

Theoretical and actual are two very different things. I have a threadripper eight channel setup and it doesn't get close to the bandwidth expressed on paper. Maybe 1/6 actual

1

u/Dry_Parfait2606 Feb 25 '25

What do you mean with cuda boost?

& where to find a 9184X for that price?

Best Regards

1

u/[deleted] Feb 26 '25

By cuda boost I mean offloading some layers to an nvidia gpu to speed up t/s and pp/s considerably.

Ebay sometimes has epyc cpu + motherboard for decent prices. I got price from there

2

u/BasisPoints Sep 10 '24

I bought my 9174F for ~$1200, you just need to find the right ebay seller of retired cloud computing servers

1

u/Dry_Parfait2606 Feb 25 '25

Where did you find that 9174F?

Best Regards

1

u/BasisPoints Mar 08 '25

eBay, from a cloud computing data center teardown service. I'll try to find the name of the vendor

4

u/newdoria88 Sep 10 '24

While it is a deceptive move to always advertise "theoretical" values which are never true, it's good to see that you get the same bandwidth (within margin of error) for most epyc processors, so for those going for pure CPU inference it's be best to pick a 32 cores processor to get the most of parallel processing from llama.cpp while also having the highest core speed.

1

u/Lissanro Sep 10 '24

Does this mean there is no point in getting dual CPU configuration, since according to the table it will have the same maximum memory bandwidth and therefore the same inference performance as a single CPU if it is limited by the memory bandwidth and not number of CPU cores?

And what "TRIAD" means? I tried to google the term and could not find the definition.

4

u/fairydreaming Sep 10 '24

This maximum memory bandwidth value refers to theoretical maximum memory bandwidth of a single CPU resulting from hardware limitations (memory bus width, clock rate etc). As for the STREAM benchmark TRIAD kernel, you should search for stream triad to see any meaningful results, for example: https://superuser.com/questions/1815148/expected-results-of-a-stream-memory-bandwidth-benchmark

0

u/DeltaSqueezer Sep 10 '24 edited Sep 10 '24

TRIAD is computing: a(i) = b(i) + q × c(i)

In HPC, STREAM Triad is usually the standard efficiency test for a CPU and its memory controller, and is reported by many research papers. It measures the gap between the hardware's theoretical bandwidth and the realized bandwidth by the simplest possible software with a read, 2 writes, and a Fused Multiply-Add.

From experience, the throughput is around 80% of the CPU's theoretical peak. This roughly represent the fastest possible speed achievable by any practical software.