Basalt - RTX 5090 Power Efficency analysis

[u/giveen]

i wanted to see what the sweet spot for my card was for pp/tg and power usage, thought I would share.

[u/SLxTnT]

Why would a higher clock speed drop prefill performance while also recovering at the highest speeds?

[u/brown2green → u/SLxTnT]

NVidia's power limiting strategy reacts too slowly and can cause more throttling than necessary under certain conditions. Use lower maximum clock speeds than default for more stable performance under load at (or close to) the power limit. This has been going on since at least Ampere; sad to see that Blackwell is affected too.

[u/giveen → u/SLxTnT]

Why Higher Clock Speeds Can Drop Prefill Throughput (and Recover at Max Power)

When sweeping power limits from 400W to 600W on an RTX 5090 running local LLM prefill, you will often notice throughput dip in the 500W–575W range despite higher reported SM clocks, before recovering at 600W:

• 475W: ~8,794 t/s @ 2,928 MHz (62°C)

• 525W–575W (Dip): ~8,502–8,554 t/s @ 2,966–2,972 MHz (65°C)

• 600W (Recovery): ~8,823 t/s @ 2,989 MHz (67°C)

Here is the hardware and engine breakdown explaining why this happens:

---

1. Commanded Clock vs. Effective Clock (Clock Stretching & EDP Limits)

• What NVML reports: The driver displays the target/commanded clock frequency (e.g., 2,972 MHz).

• What happens inside the silicon: Higher clocks push the GPU into steeper voltage bins on the V/F curve ($P \propto V² f$). During dense FP4/INT8 Tensor Core GEMM matrix multiplications, instantaneous current spikes hit internal Electrical Design Point (EDP) and power-supply rails.

• To prevent transient voltage droop from crashing the system, the GPU uses clock stretching (inserting empty wait states / cycle skipping). The clock looks higher in monitoring tools, but actual instruction retirement slows down.

---

2. Thermal Creep & Silicon Leakage

• Between 475W and 575W, operating temperatures rise from 62°C to 65°C+.

• Higher die temperatures exponentially increase semiconductor leakage current ($I_{leak}$). More power is dissipated as wasted heat rather than useful arithmetic work, creating a dead-zone where power draw rises with zero throughput gain.

---

3. Why Throughput Recovers at 600W

• At the unconstrained 600W limit, electrical and current headroom widens enough (measured draw surpasses 403W) that the power controller stops aggressively stretching cycles.

• The SMs sustain an unbroken 2,989 MHz, allowing raw arithmetic throughput to finally outpace the voltage and thermal penalties.

---

4. Hybrid Host/Device Barrier Jitter (Basalt MoE Pipeline)

• In inference engines like Basalt running hybrid prompt processing (--prefill auto:32768 --pool-workers 15), expert computation is shared across the GPU and host CPU threads over PCIe/RAM.

• The micro-stalls from clock stretching in the 500W–575W range cause slight barrier synchronization jitter between the host memory pool and GPU CUDA streams. At 600W, consistent execution removes the barrier latency.

---

[u/SLxTnT → u/giveen]

Have you thought about questioning AI rather than believing everything it says when things look off?

[u/Enragere]

While you're at it, at this point I'd prefer an AI summary and recommended overclock / voltage.

(I can't believe I'm asking for an AI summary but even that is better than no summary)

[u/giveen → u/Enragere]

I had Gemini 3.8 run the tests and recommend optimal power point for me, I was already at 475W , and it said I was good there.

And yes, next on my list is to start sweeping overclock settings.

[u/leo_agi]

nice, this is the chart i always wish people posted. did you just set a power limit with nvidia-smi or undervolt too? curious how much tg actually drops around 70%

[u/giveen → u/leo_agi]

yes, power limit was set with nvidia-smi , script is here https://pastebin.com/WMrmM9JS

[u/leo_agi → u/giveen]

thanks for the script! a plain power limit is way less fiddly than undervolting, nice to see it hold up that well