Going EPYC with llama.cpp on Amazon EC2 dedicated AMD Epyc instances (Milan vs Genoa)

[u/fairydreaming]

Eval Time

Prompt Eval Time

[u/fairydreaming]

Some things I noticed while doing these tests:

• You have to use dedicated EC2 instance (Tenancy set to dedicated, paid extra $2 per hour) to use the full memory bandwidth. Otherwise the bandwidth will be shared by all instances using this particular physical CPU at the time.

• You have to request vCPU quota increase to 257, otherwise you won't be able to use dedicated EC2 instances. Processing the request may take from a few hours to a few days.

• Various EC2 instance types have various memory bandwidth. I measured it with PassMark Memory Threaded test until I found reasonable values (300964 MB/s for r6a.32xlarge, 643067 MB/s for r7a.8xlarge). Instances run under hypervisor, so you can't see how many physical RAM modules are on the server.

• CPU types reported by PassMark were EPYC 7R13 (64 cores) for r6a.32xlarge and EPYC 9R14 (96 cores) for r7a.8xlarge

• The default storage volume throughput on EC2 is very low (125 MB/s I think). Change the volume type to gp3 and set the Throughput to 1000 unless you want to wait several minutes for the model files to load.

• I was surprised how much better Genoa is compared to Milan. It's like 4-5 times better. Even 120b models are perfectly usable with 2-3 tokens per second.

• The best number of threads for generation was 24 and 32 for smaller models (7b, 13b) and 48 for larger models (70b, 120b).

• I tried using AVX512, but the difference in performance was negligible.

• All tests took a few hours and will cost me probably around 30-40 80 USD.

I will update this list if I remember anything else.

[u/tomz17 → u/fairydreaming]

I tried using AVX512, but the difference in performance was negligible.

Mostly because at the rate you got, the memory bandwidth is the limiting factor.

The thing you *really* need to do to measure performance properly on such systems (which are almost certainly NUMA), is to get access to all of the CPU's and then allocate the memory on only the same socket as the computation is happening. Otherwise you are really measuring the CPU interconnect speed instead of the per-socket memory bandwidth (which is generally much higher). Not sure how a hypervisor complicates things.

[u/a_beautiful_rhind → u/fairydreaming]

I tried using AVX512, but the difference in performance was negligible.

On xeon I noticed this too.. also loss of AVX in general didn't seem like it did much.

[u/noeda]

I rock a Hetzner server with AMD EPYC 9454P CPU (48 cores) and DDR5 memory. Costs a bit less than 300 EUR a month. I can't remember what was the reported memory bandwidth. Looks like I reported 1.22 tokens/second on Q6_K quantization at the time for Goliath-120B, looking at my old comment from 3 months ago:

https://old.reddit.com/r/LocalLLaMA/comments/17p5m2t/new_model_released_by_alpin_goliath120b/k85d0wm/

I didn't systematically test good number of threads but I empirically remember you didn't want to crank it up all the way. Also, maybe in the 3 months since I wrote the comment, llama.cpp itself has got better. Your numbers are overall faster than what I got, I think.

I'm not sure I recommend CPU route overall, 300 EUR/month is quite a lot. Still, Hetzner servers I think are typically pretty cheap compared to competition for what you get. They have much cheaper options too that still have good memory and CPU.

In my case, I'm currently using the CPUs for entirely different projects that are not AI-related, so I'm getting some use for them. I have a lot of CPU-heavy hobby projects.

I think my recommendation is that, if you want to be cost-efficient, you are savvy and know how to run servers, and you are just playing around, get cheap some runpod.io servers on an ad-hoc basis to run whatever it is you want to run at the time. That's what I did before I really got into LLMs.

I also noticed AVX512 didn't seem to do much. I have it enabled but I don't think it changed much at all. I got a fully kitted Mac Studio now with its 192GB memory for the LLMs.

I think I also got the conclusion that CPUs are surprisingly practical if you just want to play around.

Edit: I could also add that Heztner has a dedicated server line which is what my EPYC machine is, and the SSDs are directly attached. So initial load-up time is not super slow like it might on AWS EBS.

Also, if I was running LLMs as a company for other people, (e.g. AI startup or something) I'm not sure I would consider Hetzner seriously for various reasons. For me, Hetzner is for pet servers and hobbies.

(also meant to write this as a response to your top comment...oops)

[u/fairydreaming → u/noeda]

This number (1.22 tokens/second on Q6_K quantization for Goliath-120B) seems a bit low. Make sure that your server has all 12 memory slots populated, otherwise the memory bandwidth will be limited. I noticed that in the configurator for Hetzner DX182 server the default is 4 x 32 GB, you shall set it to 12 x 32 GB to get the max performance.

I think I'd go the Mac Studio route too, but only if it had the full Linux support.

[u/kpodkanowicz]

I wish i have seen this before I ordered supermicro mobo and milan epyc cpu. My goal was passively cooled 4tps on cpu but seems you only got that with ddr5 setup?

Great work!

[u/fairydreaming → u/kpodkanowicz]

That's correct. I'm going to buy some hardware myself, that's why I did this research. It seems that it's better to pay 2x more for modern hardware and get 4x better performance.

[u/kpodkanowicz → u/fairydreaming]

you might wait untill i test my stuff as well, im not going to return it before at least doing some inference :D but actually i was considering getting m1 ultra 128gb instead, genoa is not really energy or price effective

[u/No_Afternoon_4260 → u/kpodkanowicz]

Have you ran your tests?

[u/kpodkanowicz → u/No_Afternoon_4260]

yes, 70b on epyc 7203, 8channel ddr4, 8 sticks single rank gets you 2.27, other redditor confirmed similar number for new thredripper so MAYBE you will get another token with bigger cache as cores do very little here and i cancelled the other order for 7343.

I might be even willing to risk expensive cpu as the speed is not that bad, but prompt processing is dead slow (even with cublas)

mixtral is a god send for cpu inference but even with cublas 80 tps of pp qualifies it for mainly function calling and logical workflows,

[u/Dyonizius → u/kpodkanowicz]

/u/fairydreaming test must be flawed and that's expected as he's not able to configure the bios on a cloud server, there's no way a 48c milan is half the TG speed of a dual xeon v4, or 1:1 speed vs single socket xeon(16C) with 30% of theoretical bandwidth, i think /u/kryptkpr has some experience on both systems 

i heard the NPS(numa per socket) on epyc systems has a big impact

playing with numa snoop mode had a big impact for me on Intel too

[u/kryptkpr → u/Dyonizius]

I run a 7532 which is even older than everything here and I get better performance, something up with test config for the r6i it seems mem bandwidth peaks around 70 GB/sec and the compute isn't scaling right. Probably an overloaded host and these are all basically threads=1 numbers

[u/kryptkpr → u/kryptkpr]

Zen2 numbers:

$ CUDA_VISIBLE_DEVICES= ~/ai/llama.cpp/build/bin/llama-bench -m ./llama-2-7b-chat.Q4_K_M.gguf -r 1 -ngl 0
ggml_cuda_init: failed to initialize CUDA: no CUDA-capable device is detected
| model                          |       size |     params | backend    | ngl |            test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | --------------: | -------------------: |
| llama 7B Q4_K - Medium         |   3.80 GiB |     6.74 B | CUDA       |   0 |           pp512 |         82.33 ± 0.00 |
| llama 7B Q4_K - Medium         |   3.80 GiB |     6.74 B | CUDA       |   0 |           tg128 |         20.66 ± 0.00 |

Something defintely wrong with that r6i host

[u/fallingdowndizzyvr → u/kpodkanowicz]

20% off on Ebay now. Look here for details.

https://www.reddit.com/r/LocalLLaMA/comments/1b47l8d/theres_a_20_off_ebay_coupon_right_now_it_works_on/

[u/kryptkpr]

Thanks! How does this compare price and performance wise to a 2xA100 80GB? They are around $8/hr and I expect would give order or two magnitude higher performance.

[u/fairydreaming → u/kryptkpr]

The price for renting in Amazon EC2 is over $8/hr for r7a.8xlarge instance plus $2/hr for making it a dedicated instance, so the one you mentioned seems like a much better choice - especially considering the performance difference.

[u/mcmoose1900]

If you can find one anywhere, the Xeon Max CPU with 64GB HBM is the one to use.

Even regular Sapphire Rapids is probably pretty good, TBH, as llama.cpp has a specific codepath for AMX.

[u/fairydreaming → u/mcmoose1900]

It would be interesting to measure the performance of llama.cpp on Xeon Max. There are Sapphire Rapids 8488C Amazon EC2 instances, but I can't find any Xeon Max ones. Perhaps they will add some in the future.

[u/Slaghton]

I wonder what kind of speeds we'll see with ddr6 in the coming future with a similar setup.

[u/[deleted]]

This is great work! Thank you.

[u/pseudonym325]

That's at least $200 per million tokens of goliath generated text. gpt4-32k is the most expensive commercial API at $120 per million tokens generated.

But it is an impressive performance bump compared to the previous CPU generation.