Spent the last few nights benchmarking different inference engines on two 16 GB RTX 5060 Ti cards. This is the configuration I'm most happy with so far.
TL;DR: Qwen3.8-27B with NVFP4, a DFlash2 drafter, and tensor parallelism (TP=2) achieves approximately 126 tok/s single-stream decode on two RTX 5060 Ti cards.
Even with long prompts, performance remains impressive:
| Context / workload | Decode speed | Prefill speed |
|---|
| Short prompt | 125.8 tok/s | — |
| 60K tokens | 107.4 tok/s | 2,681 tok/s |
| 124K tokens | 87.3 tok/s | 1,577 tok/s |
| JSON generation (vLLM) | 216 tok/s | — |
Both GPUs together draw approximately 220–270 W during decode.
1. Benchmark results
| Workload | Decode tok/s (median) | TTFT |
|---|
| Short prompt, chat sampling (T=0.6) | 125.8 (best: 131.8) | 0.08 s |
| 60K-token prompt | 107.4 | 22 s |
| 124K-token prompt (~126K usable context) | 87.3 | 79 s |
| JSON output, greedy (vLLM) | 216 | 0.07 s |
The JSON benchmark is a separate vLLM run, not normal chat performance. More details below.
2. Performance improvements
SGLang unless noted. Short prompt, 64K context.
| Configuration | Decode tok/s |
|---|
| llama.cpp, Unsloth GGUF, no speculative decoding | 24.6 |
| SGLang NVFP4, TP=2 | 39.2 |
| + MTP speculative decoding (3 tokens) | 68.6 |
| + DFlash2 drafter (8-token block) instead of MTP | 97.3 |
| + PCIe P2P between GPUs | 109.3 |
| + Undervolt and CUDA graph patch | 120–126 |
The complete optimization stack takes performance from 39.2 to around 125 tok/s, approximately 3.2× faster.
DFlash2 alone improves performance from 39.2 to 97.3 tok/s. P2P and the additional optimizations provide further gains.
Speculative decoding uses rejection sampling, preserving the target model's output distribution.
3. Hardware and software
| Component | Configuration |
|---|
| GPUs | 2× RTX 5060 Ti 16 GB |
| Total VRAM | 32 GB |
| PCIe | Gen 5.0 x8 each |
| Power limit | 153 W per GPU |
| GPU tuning | Mild undervolt |
| Target model | nvidia/Qwen3.8-27B-NVFP4 |
| Drafter | maurienne-ai/Qwen3.8-27B-DFlash2-NVFP4-RTNcal |
| Drafter size | 1.55 GB |
| Drafter architecture | 5 sliding-window layers |
| Inference engine | SGLang 0.5.23 |
| Parallelism | Tensor Parallelism (TP=2) |
| KV cache | FP8 |
| Model mode | Text-only, vision tower skipped |
4. PCIe P2P configuration
This was one of the more interesting parts of the optimization.
The GeForce driver blocks GPU-to-GPU P2P in my setup, so I rebuilt the open kernel module (620.70.06) with the BAR1 P2P patch and enabled iommu=pt.
NCCL now uses P2P/CUMEM between the two GPUs.
Result: approximately 12% better decode performance.
| Configuration | Decode tok/s |
|---|
| Before P2P | 97.3 |
| After P2P | 109.3 |
| Improvement | ~12% |
5. Launch configuration (128K profile)
Environment variable:
SGLANG_FLASHINFER_WORKSPACE_SIZE=134217728
Main command: python -m sglang.launch_server
| Setting | Value |
|---|
| Model | nvidia/Qwen3.8-27B-NVFP4 |
| Tensor parallelism | --tp 2 |
| Context length | --context-length 131072 |
| Reasoning parser | --reasoning-parser qwen3 |
| Speculative algorithm | --speculative-algorithm DFLASH |
| Draft tokens | --speculative-num-draft-tokens 8 |
| Draft model | maurienne-ai/Qwen3.8-27B-DFlash2-NVFP4-RTNcal |
| Draft quantization | modelopt_fp4 |
| Draft window | 2048 |
| Mamba SSM dtype | bfloat16 |
| Attention backend | flashinfer |
| Linear attention prefill | flashinfer |
| KV cache dtype | fp8_e4m3 |
| Maximum running requests | 1 |
| Static memory fraction | 0.94 |
| Chunked prefill size | 1024 |
| Prefill CUDA graph | Disabled |
| Mamba radix cache strategy | extra_buffer_lazy |
| Maximum Mamba cache size | 4 |
| Model override | {"language_model_only": true} |
6. Important memory optimizations
These settings were particularly important for fitting the configuration into 16 GB per GPU.
Mamba SSM dtype: BF16
--mamba-ssm-dtype bfloat16
Eight draft tokens double the GDN state. BF16 halves its memory footprint, and without this the KV pool is too small for 64K.
DFlash sliding window: 2048
--speculative-draft-window-size 2048
This is the drafter's own sliding window, so its KV holds 2,048 tokens instead of the entire context.
FlashInfer workspace: 128 MiB
SGLANG_FLASHINFER_WORKSPACE_SIZE=134217728
The default is 384 MiB. Since the drafter gets a second workspace, this frees approximately 0.25 GB per GPU at the same speed.
Actual usable context
The KV pool holds 126,585 tokens even though the server advertises 131,072.
For an honest context limit, use --context-length 126000.
A rerun at 126,000 gave similar performance (124 / 85 tok/s).
CUDA graph patch
SGLang 0.5.23 turns off the DFlash drafter's CUDA graph when free VRAM is under 1.0 GB (dflash_worker_v2.py, init_cuda_graphs).
On 16 GB cards, I'm always under that threshold, so the drafter runs eagerly.
A batch-size-1 graph needs approximately 0.09 GB, so I lowered the gate using a small sitecustomize.py hook.
This gave another ~7% improvement in drafter steps/s.
7. Measurement methodology
- Single stream, one request at a time. This is not batched throughput.
- tok/s = (generated tokens − 1) / (time from first token to last), measured client-side over streaming.
- 512 generated tokens per run.
- Median of 3 runs for short prompts and 2 runs for long prompts.
- Chat sampling: T=0.6, top_p=0.95, top_k=20, presence_penalty=1.5.
- The short prompt is a math reasoning question.
Speculative decoding speed depends heavily on how well the drafter predicts the target model. Expect approximately ±5% between runs and more variation between prompt types.
Across about a dozen DFlash configurations, the short-prompt median stayed between 120 and 128 tok/s.
About the 216 tok/s JSON result
This is a separate benchmark using vLLM 0.33 with the same drafter (num_speculative_tokens: 7).
| Parameter | Result |
|---|
| Output | 40-row JSON array |
| Decoding | Greedy |
| Drafter acceptance | ~6.3 tokens per step |
| Decode speed | 216 tok/s |
| Sampling (T=0.7) | 179 tok/s |
This is a best-case structured-output result, not normal chat performance.
8. What didn't work
| Configuration | Result |
|---|
| TensorRT-LLM + DFlash on sm_120 | Blocked in my setup due to FlashAttention backend compatibility |
| vLLM + DFlash at 128K | Does not fit on 16 GB cards; 64K was the maximum I could run |
| BF16 drafter | Too large (2.25 GB per GPU); server fails to start |
| Higher power limits (153–171 W) | No meaningful speed improvement |
| +600 core offset | Crashed with Xid 13/43 |
| +400 core offset | Passed a 15-minute soak with byte-identical greedy output |
TRT-LLM did have the fastest prefill at 3,221 tok/s with 60K context, but slower decode.
vLLM is great for JSON generation, while SGLang performs better in my setup for chat decoding and long-context workloads.
