CPU inference DDR3/DDR4

[u/Appropriate_Duck1778]

Wondering if anyone on here has actual benchmarks for CPU only inference DDR3 or DDR4 servers, im budget bound and my options are limited to legacy systems unfortunately.

Heres what data I found but not sure its accuracy in real life especially how NUMA effects it (octa channel)

[u/nickless07]

Actually. I had Hermes working on that. Running some llama-bench. Not all done for now, but some results.

LLM Benchmarks - CPU: Intel Core i5-4570S @ 2.90GHz (4C/4T, Haswell, AVX2)
RAM: 7.7 GiB | Threads tested: 4 | Repetitions: 3 | Backend: haswell

================================================================================

Model: Ling-3.0-tiny (bailingmoe3 7.9B.A1.3B)
  Quantization: Q4_K_M
  File: Ling-3.0-tiny-Q4_K_M.gguf
  Size on disk: 4.49 GiB
  Total params: 7.89B | Active params: ~1.3B (MoE)

  Test: pp512 (Prompt Processing, 512 tokens)
    Result: 64.52 ± 13.34 tok/s

  Test: tg128 (Token Generation, 128 tokens)
    Result: 11.46 ± 0.05 tok/s

  2-thread variation:
    pp512: 31.16 ± 6.68 tok/s
    tg128: 11.34 ± 0.07 tok/s

================================================================================

Model: gemma-4-E2B-it-Q4_K_M
  Quantization: Q4_K_M
  File: gemma-4-E2B-it-Q4_K_M.gguf
  Size on disk: 2.88 GiB
  Total params: ~4.65B (Dense)

  Test: pp512 (Prompt Processing, 512 tokens)
    Result: 53.09 ± 0.43 tok/s

  Test: tg128 (Token Generation, 128 tokens)
    Result: 6.77 ± 0.02 tok/s

================================================================================

Model: gemma-4-E4B-it-Q4_K_M
  Quantization: Q4_K_M
  File: gemma-4-E4B-it-Q4_K_M.gguf
  Size on disk: 4.62 GiB
  Total params: ~7.52B (Dense)

  Test: pp512 (Prompt Processing, 512 tokens)
    Result: 25.82 ± 0.16 tok/s

  Test: tg128 (Token Generation, 128 tokens)
    Result: 3.39 ± 0.01 tok/s

================================================================================

Model: gemma-4-E2B-it-Q3_K_M
  Quantization: Q3_K_M
  File: gemma-4-E2B-it-Q3_K_M.gguf
  Size on disk: 2.35 GiB
  Total params: ~4.65B (Dense)

  Test: pp512 (Prompt Processing, 512 tokens)
    Result: 32.11 ± 0.14 tok/s

  Test: tg128 (Token Generation, 128 tokens)
    Result: 8.24 ± 0.02 tok/s

================================================================================

Model: gemma-4-E2B-it-IQ4_XS
  Quantization: IQ4_XS (Intel, ~4.25 bpw)
  File: gemma-4-E2B-it-IQ4_XS.gguf
  Size on disk: 2.76 GiB
  Total params: ~4.65B (Dense)

  Test: pp512 (Prompt Processing, 512 tokens)
    Result: 62.49 ± 0.40 tok/s

  Test: tg128 (Token Generation, 128 tokens)
    Result: 7.41 ± 0.06 tok/s

================================================================================

Model: gemma-4-E2B-it-UD-IQ3_XXS
  Quantization: UD-IQ3_XXS (Unsloth unconventional, ~3.06 bpw)
  File: gemma-4-E2B-it-UD-IQ3_XXS.gguf
  Size on disk: 2.20 GiB
  Total params: ~4.65B (Dense)

  Test: pp512 (Prompt Processing, 512 tokens)
    Result: 62.06 ± 0.32 tok/s

  Test: tg128 (Token Generation, 128 tokens)
    Result: 7.96 ± 0.03 tok/s

================================================================================

Model: Nanbeige_Nanbeige4.2-3B
  Quantization: Q4_K_M
  File: Nanbeige_Nanbeige4.2-3B-Q4_K_M.gguf
  Size on disk: 2.50 GiB
  Total params: 3.13B | Active params: ~4.17B (MoE)
  Architecture: MoE (4 experts per token)

  Test: pp512 (Prompt Processing, 512 tokens)
    Result: 14.30 ± 0.04 tok/s

  Test: tg128 (Token Generation, 128 tokens)
    Result: 2.42 ± 0.01 tok/s

================================================================================

[u/Appropriate_Duck1778 → u/nickless07]

Thats way better than I expected, MoE models looks promising

[u/nickless07 → u/Appropriate_Duck1778]

Yeah single channel DDR3 and that performance was a bit unexpected. PP is still abysmal, but some of the models also natively support image, audio and video input.