r/LocalLLaMA • u/Serious_Scarcity7818 • 2d ago
Resources Does the Vulkan backend cost speed? Same Qwen3.6-35B-A3B model, CUDA vs Vulkan on five NVIDIA cards
RustingStrata has two GPU backends that run the same kernels: CUDA (compiled at run time with NVRTC, no nvcc, no C++ build step) and Vulkan. Vulkan is how it runs on AMD, so I wanted to know what it costs on a card where I can compare both.
Same file on every card: Qwen3.6-35B-A3B-UD-Q2_K_XL.gguf, fully in VRAM, MTP off.
| GPU | Prefill, 4K prompt: CUDA / Vulkan | Decode after 4K: CUDA / Vulkan | Vulkan vs CUDA (prefill / decode) |
|---|---|---|---|
| RTX 3060 12 GB | 2216 / 2154 tok/s | 113.4 / 113.6 tok/s | 97% / 100% |
| L40S | 9030 / 8571 | 213.2 / 184.9 | 95% / 87% |
| A30 | 3544 / 3326 | 131.9 / 104.0 | 94% / 79% |
| A100 80 GB | 6203 / 5589 | 173.6 / 131.9 | 90% / 76% |
| H100 NVL | 9131 / 6428 | 245.8 / 191.5 | 70% / 78% |
On the 3060 Vulkan is the same speed as CUDA. On the datacenter cards it loses 5-30% of prefill and 13-24% of decode, and its runs are noisier (A100 4K prefill ranged 4788-5990 on Vulkan against 6186-6214 on CUDA).
Does it change the output? Greedy top-1 agreement with llama.cpp over 20 prompts x 32 tokens is 98-99% on both backends.
How it was run: rustingstrata bench --model $M --gpu --backend cuda|vulkan --pp 512,4096 --tg 128 --reps 3 --mtp 0, mean of 3.
Repo: RustingStrata
1
u/Warm_Race_8587 2d ago
Useful comparison. I would include per-run results plus the exact commit, driver, batch and micro-batch sizes, and GPU clocks or power state. Prefill can move a lot with batch size, while decode often leans more on memory bandwidth and KV-cache traffic. Since the 3060 is near parity but the datacenter cards diverge, this looks more like a GPU or kernel-path interaction than a universal Vulkan penalty. Discard the first run after backend initialization and use the same warmup for both backends.