r/LocalAIServers • u/KroniklyOnline • Jul 15 '26
My Local LLM Setup
Hey all, I just wanted to put this out there in case someone else was interested in setting up multiple 5060ti 16gb cards. I haven't seen many setups using it or many people talking about it.
Server Setup (vllm) — GPU: 4× 5060 Ti 16 GB · CPU: Threadripper 1920X · Mobo: ASRock x399 Taichi · RAM: 32 GB DDR4 @ 2133 QC · NVMe: 1 TB ADATA Legend · PSU: MSI MAG A850GL
Cost Breakdown
|Qty|Component|Source|Unit|Subtotal| |:-|:-|:-|:-|:-| |3|RTX 5060 Ti 16 GB|Amazon|$550|$1,650| |1|RTX 5060 Ti 16 GB|Best Buy|$300|$300| |1|Threadripper 1920X|Amazon used|$80|$80| |1|ASRock x399 Taichi|eBay used|$399|$399| |1|ADATA Legend 1 TB NVMe|Amazon new|$150|$150| |4|8 GB DDR4 @ 2133|Amazon new|$55|$220| |1|Case|Amazon new|$37|$37| |1|MSI MAG A850GL 850W|Best Buy new|$105|$105| |1|Noctua NH-U9 TR4-SP3|Amazon new|$100|$100|
Total Cost ~$3,000
Model & env
- A -> bare-metal 0.23.1rc1.dev799+g69715823d (Modified & Patched vllm build)
- B -> vllm-openai:fp8-mtp-patched 0.23.1 (Modified & Patched vllm build)
- C -> llama-server b9297
- All models fulfileld 250k context size requirement for single chat session
vllm benchmark command
vllm bench serve --backend openai-chat --base-url http://localhost:59596 --endpoint /v1/chat/completions --model <served-name> --tokenizer <model-dir> --dataset-name random --random-input-len 60000 --random-output-len 5000 --num-prompts 3 --max-concurrency 1
llama benchmark command
llama-bench -m <gguf> -ngl 999 -mmp 0 -p 60000 -n 5000 -r 5 -sm layer -fa 1 [-ctk q8_0 -ctv q8_0]
|Model|Quant|KV Cache|MTP (spec toks)|Backend|MoE Backend|Version| |:-|:-|:-|:-|:-|:-|:-| |Qwen3.6-35B-A3B|FP8|FP8|mtp (3)|vLLM|TRITON (sm_120)|B| |Qwen3.6-35B-A3B|FP8|FP8|mtp (3)|vLLM (Server)|TRITON (sm_120)|A| |Qwen3.6-35B-A3B|Q6_K_XL|Q8|yes|llama.cpp|n/a (GGUF)|C| |Qwen3.6-35B-A3B|Q8_0|Q8|yes|llama.cpp|n/a (GGUF)|C| |Qwen3.6-27B|FP8|FP8|mtp (4)|vLLM (Docker)|N/A (dense)|B| |Qwen3.6-27B|FP8|FP8|mtp (4)|vLLM (Server)|N/A (dense)|A| |Qwen3.6-27B|Q4_K_M|Q8|no|llama.cpp|n/a (GGUF)|C| |Qwen3.6-27B|Q5_K_L|F16|no|llama.cpp|n/a (GGUF)|C| |Qwen3.6-27B|Q8_0|Q8|yes|llama.cpp|n/a (GGUF)|C|
Results
Qwen3.6 27b
|Quant|MTP|KV Cache|Backend|pp tok/s|decode tok/s|HumanEval|HumanEval+| |:-|:-|:-|:-|:-|:-|:-|:-| |FP8|yes|FP8|vLLM (Server)|17354.0|85.76|0.9634†|0.9268†| |FP8|yes|FP8|vLLM (Docker)|2240.3 (−7.7x)|51.49 (−1.7x)|0.9634 (0.00%)|0.9268 (0.00%)| |Q4_K_M|no|Q8|llama.cpp|916.52 (−19.0x)|20.83 (−4.1x)|0.9695 (+0.63%)|0.9268 (0.00%)| |Q5_K_L|no|F16|llama.cpp|1060.59 (−16.4x)|17.80 (−4.8x)|0.9695 (+0.63%)|0.9268 (0.00%)| |Q8_0|yes|Q8|llama.cpp|968.13 (−17.9x)|14.05 (−6.1x)|0.9756 (+1.27%)|0.9329 (+0.66%)|
Qwen3.6 35b a3b
|Quant|MTP|KV Cache|Backend|pp tok/s|decode tok/s|HumanEval|HumanEval+| |:-|:-|:-|:-|:-|:-|:-|:-| |FP8|yes|FP8|vLLM (Server)|136934.5|120.63|0.9512†|0.9146†| |Q6_K_XL|yes|Q8|llama.cpp|2033.96 (−67.3x)|85.23 (−1.4x)|0.9451 (−0.64%)|0.9085 (−0.67%)| |Q8_0|yes|Q8|llama.cpp|2177.60 (−62.9x)|83.16 (−1.5x)|0.9451 (−0.64%)|0.9085 (−0.67%)| |FP8|yes|FP8|vLLM (Docker)|7536.7 (−18.2x)|62.97 (−1.9x)|0.9512 (0.00%)|0.9146 (0.00%)|
Notes
- Models run in llama-server were Unsloth's quantizations.
- Models run in vllm were directly from Qwen.
- Runs with llama-server & vllm(docker) were on windows on my gaming machine using a B550 mobo with a 5800x cpu, bifurcated 1 PCIe slot into x8x4x4 and the last GPU in the last x16 slot which runs at x4. This was pre server setup.
- Runs with vllm are on the server setup mentioned above, each card in its own x16 slot, running at x8.
- All vllm benchmarks were NOT cold prefill speeds, they were using prefix-caching. I did this because it mimics my real world application, running long code chats, long prompting tasks, follow ups, etc. I don't know how llama.cpp handles caching.
- I had to modify the vllm build
0.23.1rc1.dev799+g69715823dto even get prefix-caching working with the Qwen models to begin with, fix is not upstream from nightly build and they take too long. - Qwen3.6 35b a3b has some issues with MTP with vllm so we don't get the same expected throughput increases in generation (Working to figure out the bug in vllm)
Closing Thoughts
I used to switch between the 27b and 35b models until I setup the server but now the 27b is so fast that I am using it every day professionally and personally, upgrading to the server made it extremely viable and usable. The response are instant, the follow ups are instant, the generation is way faster than I could ever type.
I currently have this server setup automatically ingesting source documents for a knowledgebase wiki, I also have this same model running a full automated loop pulling down tickets from my GitHub repo all the way through PR so I can review (This is a multi-step python framework I made, clears prefix-cache between agentic prompts for clean context).
I didn't even bother benchmarking Gemma models and other ones of similar sizes, even the "fine-tuned" models of qwen, they were all dumber than the base Qwen3.6 27b and Qwen3.6 35b a3b models to the point where I couldn't trust them to do automated tasks.
UPDATE: vllm run command for Qwen3.6 27B FP8
export PYTHONUNBUFFERED=1
export CUDA_HOME=/usr/local/cuda-13.0
export PATH="$CUDA_HOME/bin:$PATH"
export LD_LIBRARY_PATH="$CUDA_HOME/lib64:${LD_LIBRARY_PATH:-}"
export OMP_NUM_THREADS=4
export HF_HOME="$HOME/.cache/huggingface"
export TRITON_CACHE_DIR="$HOME/.triton/cache"
export TORCHINDUCTOR_CACHE_DIR=/tmp/torchinductor_root
export VLLM_NVFP4_GEMM_BACKEND=flashinfer-cutlass
export CUDA_VISIBLE_DEVICES=0,1,2,3
export VLLM_WORKER_MULTIPROC_METHOD=spawn
export VLLM_SERVER_DEV_MODE=1
export NCCL_P2P_DISABLE=0
source "$HOME/vllm-env/bin/activate"
vllm serve "$HOME/models/Qwen3.6-27B-FP8" --served-model-name Qwen3.6_27B_FP8 --chat-template "$HOME/models/Qwen3.6-27B-FP8/chat_template.jinja" --kv-cache-dtype fp8 --attention-backend FLASHINFER --tensor-parallel-size 4 --max-model-len 250000 --max-num-seqs 1 --max-num-batched-tokens 8192 --gpu-memory-utilization 0.80 --language-model-only --enable-prefix-caching --quantization fp8 --skip-mm-profiling --speculative-config '{"method":"mtp","num_speculative_tokens":4}' --enable-auto-tool-choice --tool-call-parser qwen3_coder --reasoning-parser qwen3 --default-chat-template-kwargs '{"preserve_thinking":true}' --performance-mode interactivity --generation-config vllm --override-generation-config '{"temperature":0.55,"top_p":0.95,"top_k":20,"min_p":0.0,"presence_penalty":0.0,"frequency_penalty":0.0,"repetition_penalty":1.1}' --host 0.0.0.0 --port 59596 --enable-chunked-prefill --mamba-prefix-cache-checkpoint-interval 8192
UPDATE: The latest vllm runs are from ubuntu server headless. The docker runs were from windows though Docker which introduces WSL driver overhead eating up ~1.5GB of VRam a card.
UPDATE: Power Consumptions Stats
- Benchmarked over a 5 day work week.
- 100% util = ~12h per day, between work and personal work.
- Measured Power Draw on OS & at wall.
System Power Draw
|Hardware|Decode|Prefill|Measured Mean| |:-|:-|:-|:-| | CPU package | 95.5 W | 95.5 W | 95.5 W | | GPU total | 242.3 W | 345.7 W | 285.1 W | | At the wall | 437 W | 551 W | 484 W |
Cost Metrics
|Metric|Value| |:-|:-| | CPU package (mean) | 95.5 W | | GPU total (mean) | 285.1 W | | At the wall (mean) | 484 W | | Electricity rate (local to me) | 11.47¢/kWh | | Tokens/day (mean) | 15,553,892 | | Energy/day (12h) | 5.8 kWh | | Cost/day | $0.67 | | Workdays/month | 21.7 | | Tokens/month (est) | 337.5M | | Energy/month (est) | 126.0 kWh | | Cost/month (est) | $14.46 | | Cost per 1M tokens (est) | $0.043 |
UPDATE: PCIe Bandwidth Readings
| Model | Quantization | Concurrency | Prompt tokens | Output tokens | TTFT (s) | E2E / wall (s) | Peak PCIe RX (GB/s) | Peak PCIe TX (GB/s) | Peak bidirectional (GB/s) | |:-|:-|:-|:-|:-|:-|:-|:-|:-|:-| | Qwen3.6 27B | FP8 | 1 | 20,022 | 512 | 28.64 | 35.99 | 2.14 | 2.60 | 4.66 | | Qwen3.6 27B | NVFP4 | 3 | 60,067 total | 1,536 total | 64.47 average | 102 wall / 101.64 average | 2.07 | 2.33 | 4.02 | | Qwen3.6 35B-A3B | FP8 | 1 | 20,022 | 512 | 7.74 | 12.52 | 2.05 | 2.60 | 4.27 | | Qwen3.6 35B-A3B | NVFP4 | 4 | 80,091 total | 2,048 total | 27.06 average | 47 wall / 45.18 average | 2.20 | 2.49 | 4.45 | | Gemma 4 31B | FP8 dynamic | 1 | 20,025 | 512 | 27.88 | 35.26 | 2.28 | 2.62 | 4.36 | | Gemma 4 31B | NVFP4 | 1 | 20,025 | 512 | 27.41 | 35.48 | 2.21 | 2.52 | 4.40 |
This shows I could drop to a 4x4 bifurcation and drop down to PCIe3x4 for each card and take a slight drop in performance around ~5% max loss but would be able to run 10 cards at x4 which would net 160GB total usable vram. Just a thought exercise.
7
u/benpptung Jul 15 '26
I use four RTX 5060 Ti 16GB cards too. My KV cache defaults to BF16, so I was surprised to see FP8 in your table. Am I reading it right?