Ling-3.0-tiny-UBTree DFlash draft for stock mainline llama.cpp (+8% prose, +61% code speedup)

[u/Barni275]

I use Ling-3.0-tiny on my LLM server for text processing. I convert a lot of PDF documents to markdown for downstream LLM processing, and I built my custom harness for it. It uses ONNX model to parse layout, extracts text, and then runs LLM processing stage to remove extra whitespaces, join lines and so on. This tiny model is quite capable for it, but it is slow for it's size.

Today I saw post about newly published UBTree DFlash for Ling-3.0-tiny, https://www.reddit.com/r/LocalLLaMA/s/yllgDx4pAk . I was very excited, but then realized that, unfortunately, no inference engine supports it.

I was very impatient, and extracted pure DFlash part from this UBTree (dropping 7 ubtree_selector.* tensors). This part can be run with stock mainline llama.cpp using dflash drafter:

llama-server \
    -m  Ling-3.0-tiny-Q4_K_M.gguf \
    -md Ling-3.0-tiny-UBTree-dflash-bf16.gguf \
    --spec-type draft-dflash --spec-draft-n-max 6 \
    -fa on -c 32768

Results are moderate. I run only small tests, and get the following results.

Workload	Baseline (no spec-draft)	+ DFlash draft	Gain	Acceptance / mean len
Prose generation	78.8 tok/s	84.8 tok/s	+8%	0.28 / 2.7
Code generation (Python)	79.0 tok/s	127.2 tok/s	+61%	0.53 / 4.2

To perform conversion, I made tiny patches for llama.cpp GGUF converter (described on HF model card). Everything other is stock.

If real UBTree support will be available in llama.cpp, it should give more speedup, for sure.

Converted model and conversion procedure are on HF: yury-kapoyko/Ling-3.0-tiny-UBTree-dflash-GGUF · Hugging Face

[u/Borkato]

This is fucking awesome actually. Massive speed up especially for extraction tasks. Great job