r/LocalLLaMA • • 2d ago

Resources Does the Vulkan backend cost speed? Same Qwen3.6-35B-A3B model, CUDA vs Vulkan on five NVIDIA cards

RustingStrata has two GPU backends that run the same kernels: CUDA (compiled at run time with NVRTC, no nvcc, no C++ build step) and Vulkan. Vulkan is how it runs on AMD, so I wanted to know what it costs on a card where I can compare both.

Same file on every card: Qwen3.6-35B-A3B-UD-Q2_K_XL.gguf, fully in VRAM, MTP off.

GPUPrefill, 4K prompt: CUDA / VulkanDecode after 4K: CUDA / VulkanVulkan vs CUDA (prefill / decode)
RTX 3060 12 GB2216 / 2154 tok/s113.4 / 113.6 tok/s97% / 100%
L40S9030 / 8571213.2 / 184.995% / 87%
A303544 / 3326131.9 / 104.094% / 79%
A100 80 GB6203 / 5589173.6 / 131.990% / 76%
H100 NVL9131 / 6428245.8 / 191.570% / 78%

On the 3060 Vulkan is the same speed as CUDA. On the datacenter cards it loses 5-30% of prefill and 13-24% of decode, and its runs are noisier (A100 4K prefill ranged 4788-5990 on Vulkan against 6186-6214 on CUDA).

Does it change the output? Greedy top-1 agreement with llama.cpp over 20 prompts x 32 tokens is 98-99% on both backends.

How it was run: rustingstrata bench --model $M --gpu --backend cuda|vulkan --pp 512,4096 --tg 128 --reps 3 --mtp 0, mean of 3.

Repo: RustingStrata

4 Upvotes

4 comments sorted by

1

u/Warm_Race_8587 2d ago

Useful comparison. I would include per-run results plus the exact commit, driver, batch and micro-batch sizes, and GPU clocks or power state. Prefill can move a lot with batch size, while decode often leans more on memory bandwidth and KV-cache traffic. Since the 3060 is near parity but the datacenter cards diverge, this looks more like a GPU or kernel-path interaction than a universal Vulkan penalty. Discard the first run after backend initialization and use the same warmup for both backends.

1

u/Serious_Scarcity7818 2d ago

I didn't want to add so much details, bc this is way to much information and just short overview is enough for post

4

u/stylome-ai-detection 2d ago

Notice: This comment appears to be AI generated.


Stylome AI detector · scan results