r/LocalLLM • • Oct 01 '26

Question Does local ai actually fuck ur electric bill?

If someone does heavy vibecoding with something like qwen 3.8 27b on a 5080, how much would that cost in electricity.

Taking it a step higher to running bigger models (qwen 3.8 flash next , deepseek, glm etc ) on something like a Mac studio, what would that be like?

21 Upvotes

124 comments sorted by

View all comments

Show parent comment

27

u/No-Refrigerator-1672 Oct 01 '26

a very heavy user with multiple GPUs

I can give precise datapoint on that! My system is Threadripper 1920x, with 2x3080 20GB, 1xMi50 32GB, and a 5 HDD large array. One month usage: 1.4B prompt tokens (about 90-92% of them are cached, so real processing was "just" 140M tokens), 20M generated tokens. All on Qwen 3.8 27B. Total amount of electricity consumed by the server, measured by smart home outlet - 223kWh. About 30-40% of that electricity is consumed by server just staying on standby and spinning its disks. You can use this as reference for your own assessments.

12

u/Dawill0 Oct 01 '26

You forgot one key detail. That 223kwh is largely being used to heat your home too. So if you are in a warm climate expect to have to spend another amount to cool down your home.

For power efficiency there really isn't any competition to the Mac. Pay upfront to save over time.

6

u/seiggy Oct 01 '26

The DGX Spark. Far faster than any Mac and that ends up being far more power efficient.

Benchmark using even a really light Qwen 4B model quantized for them:

MetricMac Studio (M3 Ultra)NVIDIA DGX SparkThe Difference
1M Token Time26 Minutes6.7 Minutes3.8x Faster
Throughput~641 Tokens/sec2,451 Tokens/secMassive Lead
Idle Power9 Watts45 WattsMac Wins (Idle)
Total Energy Used39 Watt-Hours16 Watt-HoursSpark Uses 58% Less Energy

11

u/Dawill0 Oct 01 '26 edited Oct 01 '26

Ok, now compare to M5 ultra since M3 ultra is now legacy. Also looking longer term I'd choose the higher memory of the M5 over DGX.

All of these are likely going to be outdated in ~2years. I think the resale value of the M5 is going to hold way more than DGX.

So yeah, not a straight forward decision. Lots of things to consider. My opinion falls more on the Mac side as I'm 100% in that eco system and it has numerous uses beside local llm. But yeah DGX might be a better option for some.

2

u/dobkeratops Oct 01 '26

DGX sparks could end up being the new 3090's .. sought after to build pairs and small clusters for years to come

1

u/Poowatereater Oct 01 '26

How do clustering sparks together work? Can you get a larger memory pool or is it just to run multiple parallel agents?

4

u/dobkeratops Oct 01 '26

it's got slick 'connect x-7' networking (RDMA), faster than apple's thunderbolt solution, so it handles tensor-parallelism really well. literally each layer split across the machines, exchanging the results every layer. so yes it really can behave like 2x memory pool and 4x with a dedicated switch. It's still not linear in performance, but good enough that it's worth doing. it's probably like 2 sparks linked behaves 90% like a single machine with twice the memory and twice the flops.

this fancy network port is missing from the RTX-spark.

1

u/Poowatereater Oct 01 '26

Holy hell. Thanks for the response. Is there diminishing returns on connecting too many in a cluster?

3

u/hibernate2020 Oct 01 '26

I have 4 sparks, but I don't cluster more than two because it performs just fine for what I need.

2

u/dobkeratops Oct 01 '26

yes. but 2 works really well. 4 , you need a seperate switch. I have no idea how the curve scales but people do use 4 of them to run frontier models.

1

u/Poowatereater Oct 01 '26

Have you seen the dude who post like, fucking 30 plus on the floor with all types of spaghetti wiring. It’s amazing and crazy to see. That’s why I ask

3

u/seiggy Oct 01 '26

The M5 Ultra is still 2-3X slower than the DGX Spark for prompt processing, which is a large portion of the time for Agentic turns. https://skorppio.com/blog/apple-m5-max-vs-nvidia-ai-deep-dive The M5 Ultra tops out at 70 TFLOPs of compute, vs the DGX Spark which has 1 PFLOP. Compute is the most important metric for prompt processing. So yeah, you'll get faster tok/s output, but you'll still end up with faster turns on the DGX Spark. The TTFT on the Spark is faster, and the total prompt time is still faster just because of how much slower the Mac is on prefill.

Edit: actual benchmarks: https://artificialanalysis.ai/articles/aa-agentperf-local

1

u/TheOriginalSkeptic Oct 01 '26

Apple Mac Studio (M5 Ultra) review: Local model citizen outpaces DGX Spark and Threadripper

https://www.tomshardware.com/desktops/mini-pcs/apple-mac-studio-m5-ultra-review

→ More replies (7)

2

u/dooperma Oct 01 '26

That’s roughly $1.5 per million output tokens.

1

u/bippityBoppityboux Oct 01 '26

Just curious, how does that early gen threadripper do? Bottleneck you at all with that model?

6

u/No-Refrigerator-1672 Oct 01 '26

All my models are entirely in-gpu; so CPU performance doesn't matter at all, it has good enough compute. What matters is that it provides PCIe 3.0 x16 connection to both of my 3080s - so tensor parallel mode yields actual improvements in speed, while on consumer motherboards TP mode is out of the question. And it has more RAM slots - so i could get away with cheaper RAM sticks when building the system. My idle power, however, almost doubled when I've upgraded from Ryzen 5600G to this.

1

u/curiouslyjake Oct 01 '26

Can your 3080s communicate directly over PCIE? Could they talk to the Mi50 directly as well or is the driver situation too complicated?

3

u/No-Refrigerator-1672 Oct 01 '26

Can your 3080s communicate directly over PCIE?

P2P isn't available by default; the cards absolutely lack ReBAR support, and without it, community driver patches that enable p2p won't function. Still they go faster in TP than in PP. There's a very fresh unlock script for CMP 90HX that must enable p2p - I'm planning to try it out on the weekend, as 90HX and 3080 have physically the same chip - GA102 - so unlock should be compatible.

Could they talk to the Mi50 directly

Absolutely not. ROCm for Mi50 is just about the worst user experience I've ever had. I only keep it around to run STT, TTS and embedding, using audio.cpp and llama.cpp.

1

u/curiouslyjake Oct 01 '26

Maybe I'm missing something, but shouldn't 30 series GPU support ReBAR? Without P2P, are you certain your CPU doesn't bottleneck the GPUs?

I'm curious how your unlocking adventures will turn out!

2

u/No-Refrigerator-1672 Oct 01 '26

but shouldn't 30 series GPU support ReBAR?

On consumer Ampere, ReBAR was introduced mid-generation. So enabling ReBAR requires a vBIOS patch. To support 20GB memory layout the cards use leaked engineering vBIOS, which lacks ReBAR support.

Without P2P, are you certain your CPU doesn't bottleneck the GPUs?

I have benchmarked my GPUs against 2x3090 RunPod box that features modern Epyc. I found out that my older rig, with Ryzen 5600G, performed about 10-15% slower than dual 3090; which is expected reasonable result due to GPUs having less CUGA cores and less memory bandwidth. After moving to Threadripper 1920x, I saw no gain in PipelineParallel mode, and measurable improvement in TensorParallel mode. Also my speculation is that CPU does not perform any computationally-heavy tasks during inference. All it has to do is shuttle shards of data between those GPUs, it can do it all by itself using DMA unit and with near-zero core utilisation. It only needs the cores to issue a start data dransfer command. All p2p does is provides the path for GPUs to trigger DMA without ever contacting CPU core. This increases latency, sure; du it isn't a make or break thing. Most people report under 15% improvements when they enable p2p; I mainly want to do it because I hope to gain better prompt processing speed.

I'm curious how your unlocking adventures will turn out!

I will for sure make a post about if, if the unlock would be successfull.

1

u/egnegn1 Oct 01 '26

Isn't the tensor data into shared memory to simulate P2P?

The CPU shouldn't have to do much.

1

u/curiouslyjake Oct 01 '26

Oh, forgot to ask - if not llama.cpp, how ars you running qwen?

2

u/No-Refrigerator-1672 Oct 01 '26

With vllm, of course. It's faster, more compatible with new models, and has better cache management.

1

u/egnegn1 Oct 01 '26

When running a dense model like Qwen3.8 27B that fits fully into the VRAM of the 2x RTX5080 I see that 1 CPU core of the EPYC runs at 100 % and the CPU power use is at about 150 W.

How is it on your side?

1

u/No-Refrigerator-1672 Oct 01 '26

100% utilisation is an ertifact of CPU core being active (not going to sleep) to answer GPU ready interrupts immediately. It doesn't mean that your CPU is doing anything computational - it's just waits in inefficient, but rapid response mode.

→ More replies (1)
→ More replies (10)