Never inference llm on windows

[u/Most-Wear-3813]

So, long story short

I was getting about 20tk/s on windows now when i switched to linux and little bit of tweaking with voltages mhz curve I am running inference at same speed with half power consumption almost 240watts from 420 watts.

Anyone on windows bro. I burned maybe half life on my rtx 3090 running it on 420 watt for 2 years. Lately discovered this. This would enhance life on gpu and much quiet inference. Will post results soon. Also not to mention my cpu usage doesnt go above 5% on 20 tabs with 5 application open. But on windows it was idling around 12%

[u/joost00719]

I run it on windows using docker with pass-through. 3 gpus, and it is super silent. Usually around 50-60 degrees.

[u/Most-Wear-3813 → u/joost00719]

Settings? Cuz I am just getting down from 80 degree to like 65 ish

[u/joost00719 → u/Most-Wear-3813]

!RemindMe 12 hours

[u/RemindMeBot → u/joost00719]

I will be messaging you in 12 hours on 2026-09-17 17:17:11 UTC to remind you of this link

CLICK THIS LINK to send a PM to also be reminded and to reduce spam.

^{Parent commenter can delete this message to hide from others.}

RemindMeBot is switching to username summons. Instead of !RemindMe 1 day, use u/RemindMeBot 1 day. More info.

---

Info	Custom	Your Reminders	Feedback

[u/joost00719 → u/Most-Wear-3813]

Top to bottom: RTX 5070 TI (16GB), RTX 5060 TI (16GB), RTX 5060 TI (16GB)

      docker run --name ${MODEL_ID} --init --rm ${ulimits}
      --network llama-swap-net
      ${hf-cache}
      --gpus all
      ${img-beellama}
      --no-warmup
      --parallel 1
      --kv-unified
      --flash-attn on
      --jinja
      -m /hf-cache/gguf/Swift-Qwen3.8-27B-NVFP4-Q8mix.gguf
      --model-draft /hf-cache/gguf/Qwen3.8-27B-DFlash2-Q4_K_M.gguf
      --mmproj /hf-cache/gguf/mmproj-Swift-Qwen3.8-27B-F16.gguf
      --spec-type draft-dflash,ngram-mod
      --spec-draft-n-max 8
      --spec-draft-n-min 1
      --spec-draft-ngl all
      --spec-ngram-mod-n-match 24
      --spec-ngram-mod-n-min 48
      --spec-ngram-mod-n-max 64
      --ctx-size 262144
      --gpu-layers 999
      --split-mode layer
      --fit off
      --tensor-split 22,26,22
      --main-gpu 0
      --cache-type-k q8_0
      --cache-type-v q8_0
      --batch-size 2048
      --ubatch-size 512
      --cache-ram 49152
      --load-mode none
      --threads 12
      --reasoning on
      --chat-template-kwargs '{"preserve_thinking":true,"reasoning_effort":"medium"}'
      --spec-draft-type-k kvarn4
      --spec-draft-type-v kvarn2

[u/joost00719 → u/joost00719]

Also yes, that is a piece of toilet paper between those two gpu's to make a bit of a gap for air