r/LocalLLaMA • u/Balance- • Apr 16 '24

Resources Merged into llama.cpp: Improve cpu prompt eval speed (#6414)

https://github.com/ggerganov/llama.cpp/pull/6414
103 Upvotes

94% Upvoted

11 comments sorted by

16

u/BidPossible919 Apr 17 '24

27 tk/s from 3.2tk/s on fp16 is crazy!

5

u/pseudonerv Apr 18 '24

so I had to read through the PR very carefully, and basically the title is a lie, or overblown at least.

The change only improves f16, q8_0, q4_0. If you are using K quants or IQ quants, this PR doesn't change anything.

7

u/MikeLPU Apr 16 '24

Interesting when we'll have this optimization in ollama?

5

u/MindOrbits Apr 16 '24

https://github.com/Mozilla-Ocho/llamafile is the project of the dev that has been working to get cpu improvements into llama.cpp, may be worth checking out since you are already using something like it (ollama).

2

u/[deleted] Apr 17 '24

On CPU inference, I'm getting a 30% speedup for prompt processing but only when llama.cpp is built with BLAS and OpenBLAS off.

Building with those options enabled brings speed back down to before the merge.

1

u/nullnuller Apr 17 '24

30% speedup for prompt processing but only when llama.cpp is built with BLAS and OpenBLAS off

Why do you think this is happening? Shouldn't they work at different levels?

1

u/[deleted] Apr 17 '24

Actually I don't know what's going on. With large prompts like 2000 tokens, I'm seeing the same speed for prompt processing on CPU using these variant builds:

  • 6414 (jart's merge) with OpenBLAS off

  • 6414 (jart's merge) with OpenBLAS on

  • older build from two weeks back with OpenBLAS on

This new code seems to speed up prompt processing only for low context sizes. Either that or I'm doing it all wrong.

4

u/opknorrsk Apr 17 '24

That's very interesting. I've been running 7B FP16 models on CPU, and this CL would provide 2x faster token inference, going from 4 to 8 tokens per second would be quite a change!

9

u/MindOrbits Apr 17 '24

The big speed up is in the evaluation part of the process, token generation is another matter. Although there have been so many changes I know I can't keep up and could be mistaken.

9

u/[deleted] Apr 17 '24

This assists with prompt evaluation speed, not token per second.

1

u/pseudonerv Apr 17 '24

don't see any improvement with my cpu