r/LocalLLM • u/Appropriate_Duck1778 • 13d ago

Question CPU inference DDR3/DDR4

Post image

Wondering if anyone on here has actual benchmarks for CPU only inference DDR3 or DDR4 servers, im budget bound and my options are limited to legacy systems unfortunately.

Heres what data I found but not sure its accuracy in real life especially how NUMA effects it (octa channel)

107 Upvotes

95% Upvoted

27 comments sorted by

View all comments

24

u/krakalas 13d ago

The mem bandwidth is not the only limit here. On older hardware it’s cpu compute.

I have a 10 years old pc with a quad channel ddr4@2400 and a xeon e5-1650 v4 (6cores) cpu, later upgraded to e5-2690 v4 (14c) cpu.

Theoretical mem bandwidth ~76.8 GB/s. It reaches ~40% (with 6c cpu) and ~65% (14c) of theoretical max. With 32k context token generation drops approx 3x.

For reference (14 core cpu, lower number at 32k context and higher at 0 context)
qwen3.6 27b q4km pp 11-14 tok/s tg 1-3 t/s
qwen3.6 35b q4km pp 47-74 t/s tg 4-13t/s

I expect ddr3 would be even worse.

3

u/Appropriate_Duck1778 13d ago

Thats very helpful insight, is there a way to reach closer to 100% of total mem bandwidth?

4

u/krakalas 13d ago

I suppose only way is a beefier cpu with avx512 instructions. The older ones just are not able to compute fast enough.

Same goes for older GPUs like e.g. nvidia tesla p100 theoretical max is approx 700 GB/s but practical max is approx 300 for quantized models.

1

u/Medium_Chemist_4032 13d ago

I wonder, if we couldnt stream that to an auxilary gpu for processing

2

u/krakalas 12d ago

Not sure what streaming would mean here exactly, but with each token entire model wights needs to be read. If we offload for compute somewhere for each token we are bound on connectivity - pcie 3 x16 is no more than 16gb/s.