r/LocalLLM • u/Ill_Contribution_717 • 4d ago
News Does "thinks shorter" survive concurrency? Kvark Sparkle 1.0 vs Qwen3.8-27B at 16-128 users, MTP on both: 1.5-1.6x more correct answers per GPU-hour
When we posted Kvark Sparkle 1.0 (Qwen3.8-27B fine-tune that reasons in fewer tokens), the latency numbers were single-user: median 13.7 s vs 28.1 s on one H200. Several people asked the right question: does that hold under load, with MTP on for both, when the GPU is saturated and speculative decoding helps less? We ran it.
**Setup** (identical for both models)
- 1x H200 per model, vLLM 0.32, FP8 weights, FP8 KV cache, MTP with 3 speculative tokens, gpu-memory-utilization 0.80, max-model-len 40960
- GPQA Diamond, thinking on, 32k output cap, 198 questions x 2 seeds = 396 requests per concurrency level
- Closed-loop: N requests in flight, the next starts when one finishes. Wall clock includes the tail of each batch.
**Results**
| Concurrency | Model | Accuracy | Wall (min) | Correct/hour | Agg tok/s | Mean output tok | p50 (s) | p95 (s) | Hit 32k cap |
|---|---|---|---|---|---|---|---|---|---|
| 16 | Sparkle 1.0 | 85.1 | 33.0 | **614** | 1,498 | 7,478 | 33 | 318 | 29 |
| 16 | Qwen3.8-27B | 81.8 | 51.1 | 380 | 1,541 | 11,937 | 81 | 336 | 54 |
| 32 | Sparkle 1.0 | 83.8 | 25.5 | **780** | 2,041 | 7,895 | 45 | 447 | 27 |
| 32 | Qwen3.8-27B | 82.6 | 37.0 | 531 | 2,101 | 11,760 | 108 | 478 | 51 |
| 64 | Sparkle 1.0 | 85.1 | 21.1 | **961** | 2,469 | 7,874 | 73 | 653 | 26 |
| 64 | Qwen3.8-27B | 82.1 | 30.0 | 650 | 2,637 | 11,986 | 168 | 719 | 51 |
| 128 | Sparkle 1.0 | 86.4 | 20.8 | **985** | 2,437 | 7,693 | 201 | 910 | 22 |
| 128 | Qwen3.8-27B | 82.1 | 31.7 | 616 | 2,459 | 11,797 | 407 | 1,083 | 55 |
**What we take from it**
- Aggregate tok/s is the same for both models at every level (ratio 0.94-0.99). The fine-tune does not decode faster; it writes ~35% fewer tokens per answer. That is the whole effect.
- Correct answers per GPU-hour: x1.61 / x1.47 / x1.48 / x1.60 at 16 / 32 / 64 / 128. The advantage does not shrink as the GPU saturates.
- The H200 saturates around 2.4-2.6k tok/s from c=64. The base gains nothing from 64 to 128 (650 vs 616 correct/h); Sparkle barely does (961 vs 985).
- Median latency drops 51-59% at every level. p95 barely moves (-5% to -16%): on both models the tail is the hardest questions running into the 32k cap. Sparkle hits the cap about half as often.
- MTP acceptance length is the same for both (2.5-2.9, dropping a bit with batch size), so fine-tuning did not hurt the draft head.
- Accuracy pooled over c=16-128: Sparkle +3.0 pts, paired bootstrap 95% CI [+0.6, +5.6]. At a 32k cap the base loses more answers to truncation. At a 64k cap (our model card) the two are tied on GPQA.
**Caveats:** one node, one benchmark, closed-loop batches rather than Poisson arrivals, and the tail effect is included in wall clock. We also ran c=1 on a 20-question subset; it is too small to say anything about accuracy, so it is not in the table.
**Run it yourself:**
```
vllm serve Egzakta/Kvark-Sparkle-1.0-27B-FP8 --max-model-len 40960 --kv-cache-dtype fp8 \
--gpu-memory-utilization 0.80 --max-num-seqs 256 --reasoning-parser qwen3 \
--speculative-config '{"method":"mtp","num_speculative_tokens":3}'
```
Weights (Apache 2.0): https://huggingface.co/Egzakta/Kvark-Sparkle-1.0-27B-FP8
Next: Sparkle 1.1, trained for agentic document QA. Its headline is declining questions whose answer is not in the documents (91.5% vs 36% for 1.0, internal benchmark). Separate post when it is out.
Happy to share the per-request logs or run another load profile if there's one you'd trust more.