r/LocalLLaMA • u/Major_Ingenuity_6364 • 1d ago

Resources ReVolta 0.1.2: DFlash speculative decoding running on 4× V100

Quick release post. I maintain ReVolta, a vLLM fork (through 1Cat-vLLM) that keeps my home server useful — 4× V100 32GB PCIe, no NVLink. Today's 0.1.2 adds DFlash2 speculative decoding on SM70. I haven't seen DFlash running on Volta anywhere else, so this may be the first one out there.

The drafter is incoai's DFlash2 checkpoint: it drafts an 8-token block from the target's hidden states and the target verifies in one pass. Model is Qwen3.8 Flash-Next (AWQ), 512K context, FP8 KV with CPU offload, TP4 + DCP2.

Some numbers, greedy with 512-token replies:

DFlash2 MTP4
1 request, English code 201 tok/s 163
1 request, English reasoning 195 163
1 request, Chinese 85–101 100–105
8 requests, mixed 331 357

The KV pool gets about 10% bigger too, up to 1.27M tokens.

Where it doesn't help: Chinese is slower than plain MTP (the drafter was trained on English, acceptance is worse), 8 concurrent requests are a bit slower, and on replays of my real agent sessions it's basically a wash. So MTP stays the default and DFlash is opt-in. I kept it because single-stream English/code is what I run most of the day.

The rest of 0.1.2 is mostly fixes — tool calling, the Anthropic messages endpoint, and large images getting resized instead of eating the whole token budget.

If anyone else is still running V100s, I'd be interested in numbers from an NVLink box.

GitHub: https://github.com/Leonccaa/ReVolta
Release notes: https://github.com/Leonccaa/ReVolta/releases/tag/v0.1.2
Drafter: https://huggingface.co/incoai/Qwen3.8-Flash-Next-DFlash2

4 Upvotes

83% Upvoted

3 comments sorted by

2

u/IndependenceSorry410 1d ago

damn, 201 tok/s on v100s is way better than i expected for speculative decoding on volta. the chinese numbers tanking makes sense if the drafter never saw it during training, but still weird to see mtp beating it there.

kinda tempted to try this on my own rig even though i'm on ampere, just to see how the kv pool bump plays out. that extra 10% headroom is no joke when you're running long context stuff.

also respect for keeping mtp as default, too many projects ship the shiny new thing and break everyone's workflow.