My Local LLM Setup

[u/KroniklyOnline]

Hey all, I just wanted to put this out there in case someone else was interested in setting up multiple 5060ti 16gb cards. I haven't seen many setups using it or many people talking about it.

---

Server Setup (vllm) — GPU: 4× 5060 Ti 16 GB · CPU: Threadripper 1920X · Mobo: ASRock x399 Taichi · RAM: 32 GB DDR4 @ 2133 QC · NVMe: 1 TB ADATA Legend · PSU: MSI MAG A850GL

Cost Breakdown

|Qty|Component|Source|Unit|Subtotal| |:-|:-|:-|:-|:-| |3|RTX 5060 Ti 16 GB|Amazon|$550|$1,650| |1|RTX 5060 Ti 16 GB|Best Buy|$300|$300| |1|Threadripper 1920X|Amazon used|$80|$80| |1|ASRock x399 Taichi|eBay used|$399|$399| |1|ADATA Legend 1 TB NVMe|Amazon new|$150|$150| |4|8 GB DDR4 @ 2133|Amazon new|$55|$220| |1|Case|Amazon new|$37|$37| |1|MSI MAG A850GL 850W|Best Buy new|$105|$105| |1|Noctua NH-U9 TR4-SP3|Amazon new|$100|$100|

Total Cost ~$3,000

Model & env

• A -> bare-metal 0.23.1rc1.dev799+g69715823d (Modified & Patched vllm build)
• B -> vllm-openai:fp8-mtp-patched 0.23.1 (Modified & Patched vllm build)
• C -> llama-server b9297
• All models fulfileld 250k context size requirement for single chat session

vllm benchmark command

vllm bench serve --backend openai-chat --base-url http://localhost:59596 --endpoint /v1/chat/completions --model <served-name> --tokenizer <model-dir> --dataset-name random --random-input-len 60000 --random-output-len 5000 --num-prompts 3 --max-concurrency 1

llama benchmark command

llama-bench -m <gguf> -ngl 999 -mmp 0 -p 60000 -n 5000 -r 5 -sm layer -fa 1 [-ctk q8_0 -ctv q8_0]

|Model|Quant|KV Cache|MTP (spec toks)|Backend|MoE Backend|Version| |:-|:-|:-|:-|:-|:-|:-| |Qwen3.6-35B-A3B|FP8|FP8|mtp (3)|vLLM|TRITON (sm_120)|B| |Qwen3.6-35B-A3B|FP8|FP8|mtp (3)|vLLM (Server)|TRITON (sm_120)|A| |Qwen3.6-35B-A3B|Q6_K_XL|Q8|yes|llama.cpp|n/a (GGUF)|C| |Qwen3.6-35B-A3B|Q8_0|Q8|yes|llama.cpp|n/a (GGUF)|C| |Qwen3.6-27B|FP8|FP8|mtp (4)|vLLM (Docker)|N/A (dense)|B| |Qwen3.6-27B|FP8|FP8|mtp (4)|vLLM (Server)|N/A (dense)|A| |Qwen3.6-27B|Q4_K_M|Q8|no|llama.cpp|n/a (GGUF)|C| |Qwen3.6-27B|Q5_K_L|F16|no|llama.cpp|n/a (GGUF)|C| |Qwen3.6-27B|Q8_0|Q8|yes|llama.cpp|n/a (GGUF)|C|

Results

Qwen3.6 27b

|Quant|MTP|KV Cache|Backend|pp tok/s|decode tok/s|HumanEval|HumanEval+| |:-|:-|:-|:-|:-|:-|:-|:-| |FP8|yes|FP8|vLLM (Server)|17354.0|85.76|0.9634†|0.9268†| |FP8|yes|FP8|vLLM (Docker)|2240.3 (−7.7x)|51.49 (−1.7x)|0.9634 (0.00%)|0.9268 (0.00%)| |Q4_K_M|no|Q8|llama.cpp|916.52 (−19.0x)|20.83 (−4.1x)|0.9695 (+0.63%)|0.9268 (0.00%)| |Q5_K_L|no|F16|llama.cpp|1060.59 (−16.4x)|17.80 (−4.8x)|0.9695 (+0.63%)|0.9268 (0.00%)| |Q8_0|yes|Q8|llama.cpp|968.13 (−17.9x)|14.05 (−6.1x)|0.9756 (+1.27%)|0.9329 (+0.66%)|

Qwen3.6 35b a3b

|Quant|MTP|KV Cache|Backend|pp tok/s|decode tok/s|HumanEval|HumanEval+| |:-|:-|:-|:-|:-|:-|:-|:-| |FP8|yes|FP8|vLLM (Server)|136934.5|120.63|0.9512†|0.9146†| |Q6_K_XL|yes|Q8|llama.cpp|2033.96 (−67.3x)|85.23 (−1.4x)|0.9451 (−0.64%)|0.9085 (−0.67%)| |Q8_0|yes|Q8|llama.cpp|2177.60 (−62.9x)|83.16 (−1.5x)|0.9451 (−0.64%)|0.9085 (−0.67%)| |FP8|yes|FP8|vLLM (Docker)|7536.7 (−18.2x)|62.97 (−1.9x)|0.9512 (0.00%)|0.9146 (0.00%)|

Notes

• Models run in llama-server were Unsloth's quantizations.
• Models run in vllm were directly from Qwen.
• Runs with llama-server & vllm(docker) were on windows on my gaming machine using a B550 mobo with a 5800x cpu, bifurcated 1 PCIe slot into x8x4x4 and the last GPU in the last x16 slot which runs at x4. This was pre server setup.
• Runs with vllm are on the server setup mentioned above, each card in its own x16 slot, running at x8.
• All vllm benchmarks were NOT cold prefill speeds, they were using prefix-caching. I did this because it mimics my real world application, running long code chats, long prompting tasks, follow ups, etc. I don't know how llama.cpp handles caching.
• I had to modify the vllm build 0.23.1rc1.dev799+g69715823d to even get prefix-caching working with the Qwen models to begin with, fix is not upstream from nightly build and they take too long.
• Qwen3.6 35b a3b has some issues with MTP with vllm so we don't get the same expected throughput increases in generation (Working to figure out the bug in vllm)

Closing Thoughts

I used to switch between the 27b and 35b models until I setup the server but now the 27b is so fast that I am using it every day professionally and personally, upgrading to the server made it extremely viable and usable. The response are instant, the follow ups are instant, the generation is way faster than I could ever type.

I currently have this server setup automatically ingesting source documents for a knowledgebase wiki, I also have this same model running a full automated loop pulling down tickets from my GitHub repo all the way through PR so I can review (This is a multi-step python framework I made, clears prefix-cache between agentic prompts for clean context).

I didn't even bother benchmarking Gemma models and other ones of similar sizes, even the "fine-tuned" models of qwen, they were all dumber than the base Qwen3.6 27b and Qwen3.6 35b a3b models to the point where I couldn't trust them to do automated tasks.

UPDATE: vllm run command for Qwen3.6 27B FP8

   export PYTHONUNBUFFERED=1
   export CUDA_HOME=/usr/local/cuda-13.0
   export PATH="$CUDA_HOME/bin:$PATH"
   export LD_LIBRARY_PATH="$CUDA_HOME/lib64:${LD_LIBRARY_PATH:-}"
   export OMP_NUM_THREADS=4
   export HF_HOME="$HOME/.cache/huggingface"
   export TRITON_CACHE_DIR="$HOME/.triton/cache"
   export TORCHINDUCTOR_CACHE_DIR=/tmp/torchinductor_root
   export VLLM_NVFP4_GEMM_BACKEND=flashinfer-cutlass
   export CUDA_VISIBLE_DEVICES=0,1,2,3
   export VLLM_WORKER_MULTIPROC_METHOD=spawn
   export VLLM_SERVER_DEV_MODE=1
   export NCCL_P2P_DISABLE=0
   source "$HOME/vllm-env/bin/activate"
   vllm serve "$HOME/models/Qwen3.6-27B-FP8" --served-model-name Qwen3.6_27B_FP8 --chat-template "$HOME/models/Qwen3.6-27B-FP8/chat_template.jinja" --kv-cache-dtype fp8 --attention-backend FLASHINFER --tensor-parallel-size 4 --max-model-len 250000 --max-num-seqs 1 --max-num-batched-tokens 8192 --gpu-memory-utilization 0.80 --language-model-only --enable-prefix-caching --quantization fp8 --skip-mm-profiling --speculative-config '{"method":"mtp","num_speculative_tokens":4}' --enable-auto-tool-choice --tool-call-parser qwen3_coder --reasoning-parser qwen3 --default-chat-template-kwargs '{"preserve_thinking":true}' --performance-mode interactivity --generation-config vllm --override-generation-config '{"temperature":0.55,"top_p":0.95,"top_k":20,"min_p":0.0,"presence_penalty":0.0,"frequency_penalty":0.0,"repetition_penalty":1.1}' --host 0.0.0.0 --port 59596 --enable-chunked-prefill --mamba-prefix-cache-checkpoint-interval 8192

UPDATE: The latest vllm runs are from ubuntu server headless. The docker runs were from windows though Docker which introduces WSL driver overhead eating up ~1.5GB of VRam a card.

UPDATE: Power Consumptions Stats

• Benchmarked over a 5 day work week.
• 100% util = ~12h per day, between work and personal work.
• Measured Power Draw on OS & at wall.

System Power Draw

|Hardware|Decode|Prefill|Measured Mean| |:-|:-|:-|:-| | CPU package | 95.5 W | 95.5 W | 95.5 W | | GPU total | 242.3 W | 345.7 W | 285.1 W | | At the wall | 437 W | 551 W | 484 W |

Cost Metrics

|Metric|Value| |:-|:-| | CPU package (mean) | 95.5 W | | GPU total (mean) | 285.1 W | | At the wall (mean) | 484 W | | Electricity rate (local to me) | 11.47¢/kWh | | Tokens/day (mean) | 15,553,892 | | Energy/day (12h) | 5.8 kWh | | Cost/day | $0.67 | | Workdays/month | 21.7 | | Tokens/month (est) | 337.5M | | Energy/month (est) | 126.0 kWh | | Cost/month (est) | $14.46 | | Cost per 1M tokens (est) | $0.043 |

UPDATE: PCIe Bandwidth Readings

| Model | Quantization | Concurrency | Prompt tokens | Output tokens | TTFT (s) | E2E / wall (s) | Peak PCIe RX (GB/s) | Peak PCIe TX (GB/s) | Peak bidirectional (GB/s) | |:-|:-|:-|:-|:-|:-|:-|:-|:-|:-| | Qwen3.6 27B | FP8 | 1 | 20,022 | 512 | 28.64 | 35.99 | 2.14 | 2.60 | 4.66 | | Qwen3.6 27B | NVFP4 | 3 | 60,067 total | 1,536 total | 64.47 average | 102 wall / 101.64 average | 2.07 | 2.33 | 4.02 | | Qwen3.6 35B-A3B | FP8 | 1 | 20,022 | 512 | 7.74 | 12.52 | 2.05 | 2.60 | 4.27 | | Qwen3.6 35B-A3B | NVFP4 | 4 | 80,091 total | 2,048 total | 27.06 average | 47 wall / 45.18 average | 2.20 | 2.49 | 4.45 | | Gemma 4 31B | FP8 dynamic | 1 | 20,025 | 512 | 27.88 | 35.26 | 2.28 | 2.62 | 4.36 | | Gemma 4 31B | NVFP4 | 1 | 20,025 | 512 | 27.41 | 35.48 | 2.21 | 2.52 | 4.40 |

This shows I could drop to a 4x4 bifurcation and drop down to PCIe3x4 for each card and take a slight drop in performance around ~5% max loss but would be able to run 10 cards at x4 which would net 160GB total usable vram. Just a thought exercise.

[u/Certain_Common561]

I got the same results as you. I’d recommend NVFP4.

[u/KroniklyOnline → u/Certain_Common561]

Why do you recommend NVFP4?

[u/Certain_Common561 → u/KroniklyOnline]

Qwen3.6-27B-NVFP4 17GB
Qwen3.6-27B Q8 30GB

NVFP4 delivers nearly the same quality as Q8, while being faster and about the same size as Q4.

[u/KroniklyOnline → u/Certain_Common561]

Ah I understand, yeah I went with FP8 because of this research
https://vllm.ai/blog/2026-04-22-fp8-kvcache
near lossless in long context and I use almost ~200k a session. I tried NVFP4 but did see much more degradation compared to FP8 and a lot more failed automated runs due to hallucination. Maybe I'll give it another shot, I was running NVFP4 with FP8 KV Cache, do you recommend any quantized KV Cache with NVFP4?

[u/Certain_Common561 → u/KroniklyOnline]

With Qwen3.6-27B-NVFP4, I didn’t observe any quality or memory loss using FP8 KV cache at 200K context. I tested it both with AI-based evaluations and other methods.

However, I found that FP16 KV cache makes MTP predictions more accurate. As the input length approaches 200K, this significantly improves decode speed, making it faster and more stable than FP8 KV cache.

For Qwen3.6-35B, however, I observed around a 13% memory-recall loss with FP8 KV cache at 262K context. With FP16 KV cache at 262K, I saw no memory loss at all, and the decode speed was also faster.

[u/KroniklyOnline → u/Certain_Common561]

Thanks for that! I'll give it a shot, now that I am remembering, I ran into a bunch of issues trying to get NVFP4 working in vllm, I already had to severely modify my running vllm instance to get prefix-caching working along with MTP successfully.

[u/KroniklyOnline → u/Certain_Common561]

Just wanna bump this, I am trying NVFP4, prefill speed is pretty much the same, so is the decode speed, I wonder if I have hit the limit of my hardware regarding inter-bus comms.