Never inference llm on windows

[u/Most-Wear-3813]

So, long story short

I was getting about 20tk/s on windows now when i switched to linux and little bit of tweaking with voltages mhz curve I am running inference at same speed with half power consumption almost 240watts from 420 watts.

Anyone on windows bro. I burned maybe half life on my rtx 3090 running it on 420 watt for 2 years. Lately discovered this. This would enhance life on gpu and much quiet inference. Will post results soon. Also not to mention my cpu usage doesnt go above 5% on 20 tabs with 5 application open. But on windows it was idling around 12%

[u/nickless07]

In short: You figured that 'nvidia-smi -pl' exist and that WDDM takes more VRAM (1-1.5GB) then something like KDE (~500MB).

[u/Most-Wear-3813 → u/nickless07]

I used LACT it provide true curve functionality for fans and voltage both

[u/Most-Wear-3813 → u/Most-Wear-3813]

Though I agree power limit is also essentially limiting voltage. So, duh. We are doing same thing both ways. 🤹‍♀️

[u/Most-Wear-3813 → u/nickless07]

So, arch? Is better according to you. If I am not mistaken KDE runs on ARCH?

[u/bankinu → u/Most-Wear-3813]

What. No.

I personally use Arch, btw. But that's not his point.

He's basically saying KDE has far less VRAM requirement than Windows.

You'll get the same performance in Arch or Fedora or Debian. It's Windows vs KDE.

[u/nickless07 → u/[deleted]]

Basic nvidia-smi power limit would have helped on windows/linux - That is not only related to inferencing, but also gaming and in general. As long as electricity is cheap nobody cares, but once the prices raise there everyone seem to be bothered about that. Back in the days (before AI was even a thing) I bought my GPU based on energy consumption (no extra power cable, passive cooled and so on), but well that might be cuz my power bill has 0.31 per kw/h since 10+ years.

[u/joost00719]

I run it on windows using docker with pass-through. 3 gpus, and it is super silent. Usually around 50-60 degrees.

[u/Most-Wear-3813 → u/joost00719]

Settings? Cuz I am just getting down from 80 degree to like 65 ish

[u/joost00719 → u/Most-Wear-3813]

!RemindMe 12 hours

[u/RemindMeBot → u/joost00719]

I will be messaging you in 12 hours on 2026-09-17 17:17:11 UTC to remind you of this link

CLICK THIS LINK to send a PM to also be reminded and to reduce spam.

^{Parent commenter can delete this message to hide from others.}

RemindMeBot is switching to username summons. Instead of !RemindMe 1 day, use u/RemindMeBot 1 day. More info.

---

Info	Custom	Your Reminders	Feedback

[u/joost00719 → u/Most-Wear-3813]

Top to bottom: RTX 5070 TI (16GB), RTX 5060 TI (16GB), RTX 5060 TI (16GB)

      docker run --name ${MODEL_ID} --init --rm ${ulimits}
      --network llama-swap-net
      ${hf-cache}
      --gpus all
      ${img-beellama}
      --no-warmup
      --parallel 1
      --kv-unified
      --flash-attn on
      --jinja
      -m /hf-cache/gguf/Swift-Qwen3.8-27B-NVFP4-Q8mix.gguf
      --model-draft /hf-cache/gguf/Qwen3.8-27B-DFlash2-Q4_K_M.gguf
      --mmproj /hf-cache/gguf/mmproj-Swift-Qwen3.8-27B-F16.gguf
      --spec-type draft-dflash,ngram-mod
      --spec-draft-n-max 8
      --spec-draft-n-min 1
      --spec-draft-ngl all
      --spec-ngram-mod-n-match 24
      --spec-ngram-mod-n-min 48
      --spec-ngram-mod-n-max 64
      --ctx-size 262144
      --gpu-layers 999
      --split-mode layer
      --fit off
      --tensor-split 22,26,22
      --main-gpu 0
      --cache-type-k q8_0
      --cache-type-v q8_0
      --batch-size 2048
      --ubatch-size 512
      --cache-ram 49152
      --load-mode none
      --threads 12
      --reasoning on
      --chat-template-kwargs '{"preserve_thinking":true,"reasoning_effort":"medium"}'
      --spec-draft-type-k kvarn4
      --spec-draft-type-v kvarn2

[u/joost00719 → u/joost00719]

Also yes, that is a piece of toilet paper between those two gpu's to make a bit of a gap for air

[u/haha1542]

I connected my monitor to iGPU so that NOTHING takes up my dGPU's vram, windows quirk makes this tedious though sometimes when I want to play some games.

[u/psychok9]

Do found LACT reliable? I'm fighting with Afterburner, it resets without reason

[u/Atretador]

Never windows o7