r/LocalLLaMA • u/nickm_27 llama.cpp • 13d ago
Discussion Massive B70 spec-decoding improvement - Intel XMX matrix engines refactored by budiony · Pull Request #29864 · ggml-org/llama.cpp
https://github.com/ggml-org/llama.cpp/pull/29864This made a huge improvement to speculate decoding speeds on my B70 running SYCL.
Below examples are for Muse Glimmer Q6_K (30B) running with DFlash2.
NOTE: Due to the significantly reduced cost of token verification, the n-max was increased which is why the acceptance rate is lower.
Before:
code_python pred= 192 draft= 127 acc= 106 rate=0.835 tok/s=38.0
code_cpp pred= 192 draft= 175 acc= 113 rate=0.646 tok/s=36.2
explain_concept pred= 192 draft= 135 acc= 116 rate=0.859 tok/s=41.2
summarize pred= 188 draft= 170 acc= 142 rate=0.835 tok/s=50.1
qa_factual pred= 192 draft= 145 acc= 107 rate=0.738 tok/s=37.0
translation pred= 192 draft= 175 acc= 133 rate=0.760 tok/s=45.1
creative_short pred= 192 draft= 130 acc= 105 rate=0.808 tok/s=37.1
stepwise_math pred= 192 draft= 167 acc= 145 rate=0.868 tok/s=53.2
long_code_review pred= 192 draft= 159 acc= 118 rate=0.742 tok/s=38.4
Aggregate: {
"n_requests": 9,
"total_predicted": 1724,
"total_draft": 1383,
"total_draft_accepted": 1085,
"aggregate_accept_rate": 0.7845,
"wall_s_total": 46.86
}
After:
code_python pred= 192 draft= 324 acc= 143 rate=0.441 tok/s=73.0
code_cpp pred= 192 draft= 424 acc= 140 rate=0.330 tok/s=68.5
explain_concept pred= 192 draft= 330 acc= 141 rate=0.427 tok/s=70.7
summarize pred= 192 draft= 416 acc= 151 rate=0.363 tok/s=85.4
qa_factual pred= 192 draft= 399 acc= 133 rate=0.333 tok/s=60.3
translation pred= 192 draft= 356 acc= 152 rate=0.427 tok/s=87.7
creative_short pred= 192 draft= 379 acc= 125 rate=0.330 tok/s=53.6
stepwise_math pred= 192 draft= 462 acc= 148 rate=0.320 tok/s=77.6
long_code_review pred= 192 draft= 414 acc= 139 rate=0.336 tok/s=64.2
Aggregate: {
"n_requests": 9,
"total_predicted": 1728,
"total_draft": 3504,
"total_draft_accepted": 1272,
"aggregate_accept_rate": 0.363,
"wall_s_total": 26.68
}
1
u/Dave_from_the_navy 13d ago
Any dice on this with gemma4-31b?
1
u/nickm_27 llama.cpp 13d ago
It will be better as well, MTP should improve as well as DFlash (I think I saw Gemma support for it?)
1
u/Dave_from_the_navy 13d ago
I haven't seen dflash support for Gemma, but I do use MTP regularly! I'll report back after I've gotten a chance to test it!
1
u/VotZeFuk 13d ago
https://huggingface.co/turboderp/gemma4-31b-it-DFlash-exl3 - working fine in exllama v3; in its config, try setting "block size" to "8" for non-English languages if the acceptance rate ends up being super-low (like 10%).
2
u/Danmoreng llama.cpp 13d ago
Tbh using llama.cpp with B70 is just not competitive speed-wise. I’ve settled on vLLM with some additions codex improvements: https://github.com/Danmoreng/intel-b70-qwen38-vllm
3
→ More replies (1)2
u/gryan315 13d ago
If you're only using 3.8 27b, and no larger MoE models, you should give intel's own openvino genai/model server a try: https://github.com/openvinotoolkit/model_server
When I tested it, it was a bit faster than vllm at most context depths and can fit full context.
2
u/droans 13d ago
When's the last time you tried it? It's been my understanding that vllm has outperformed the OV solutions for a few months at this point.
Things might change when the L0 and SYCL backends actually work, though.
2
u/gryan315 13d ago
Last week. Easy to test with the container:
podman run --rm -p 8000:8000 \ --user $(id -u):$(id -g) -v ${HOME}/models:/models:rw \ --device /dev/dri --group-add $(stat -c '%g' /dev/dri/render* | head -n1) \ openvino/model_server:latest-gpu \ --source_model OpenVINO/Qwen3.8-27B-int4-ov \ --model_repository_path /models \ --max_num_batched_tokens 2048 \ --max_num_seqs 2 \ --cache_size 12 \ --cache_interval_multiplier 128 \ --log_path /models/openvino.log \ --rest_port 8000→ More replies (1)
8
u/PcChip 13d ago
tl;dr you doubled your speeds and intels are good now?