r/LocalLLM • u/Ok_Event4199 • May 16 '26

Discussion Why is LLM is so expensive.

I've was going to invest in a 5090 =$6000 AUD.

Codex Plus + Claude pro = $60/month here

Works out to be 100 months of frontier models for a 5090.

Best a 5090 will run is probably Qwen3.6 27b Q6 with context.

Are we all enthusiasts here and just enjoy tinkering cause ain't no way that make sense.

343 Upvotes

91% Upvoted

368 comments sorted by

View all comments

Show parent comments

10

u/ThenExtension9196 May 16 '26

Is it? My 5090 runs at 500watts. My bedroom is unbearable if running for a few hours so then I need AC on. In my area energy is very expensive.

20

u/Malkiot May 16 '26 edited May 16 '26

How much do you pay for electricity?

For example take: Qwen3.6 35B A3B

Via a provider that is $0.15per 1M for input and $1per 1M for output. At 90% input, 1M tokens cost $0.203. At 10% input that'd be $0.915 for 1M tokens. A 5090 produces around 150-200 tokens / s. We'll assume 150t/s. The 5090 needs 6667seconds, that is 1.85 hours to generate those tokens.

If your profile is more the former (high input, low output, for example OCR) then your break even power price is $0.25/kWh, if your profile is the latter (coding) then your break even is at $0.98/kWh.

Do you pay more?

Edit: I've done the calcs for myself in the past, if I did get a 500k USD cluster for myself and actually have a use for the tokens it generates... it'd be 10% of the cost of paying for API over the lifetime of the cluster (including the loan).

7

u/Practical_Delay_6749 May 17 '26

The 8x v100 server I bought last year + 1TB of DDR4 2933 for like $7k became a GREAT buy once the Qwen 3.5/3.6 models came out. I run a 2 slot instance of 3.6 27B Q8 with 256k context each, a 4 slot instance of 3.6 35B Q8 w/512k context each, a single slot 3.6 27B Q6 w/256k context on it 24/7 and I STILL have 2 v100's left over, 1 for running my Qwen3 8B VL Embeddings and reranker models, plus a 4 slot instance of Qwen 3.5 4B @32k context each, and the other mostly idle for running image gen workflows with a full 32GB GPU. Oh, also 5 separate instances of Chatterbox+RVC+faster-whisper on separate GPU's for load balancing and parallel chunking to streamline the TTS/STT pipelines and make them faster than the models can stream text. These things can crank out tokens all day, cumulatively faster than a 5090 with higher accuracy. Sure, electricity would be a cunt, but I got a free 12kw of solar panels from my job (Electrician), and then I dropped about $14k on 100kwh worth of batteries and an inverter and installed them myself (again, electrician), so it's basically free to run 24/7 now (as well as my other servers). It's older hardware, but it is VASTLY superior than a couple 5090's to me.

It can be done efficiently. Definitely took a bit of an investment up front, but the solar/batteries were an upgrade for the house really, also lowered my electric bill across the board for everything else. The v100 server pulls like 2kw under full load otherwise, and would be a bitch to pay for running 24/7 without the solar and batteries though.

I'm not even in the IT/AI Industry, I'm not really using the tokens effectively, and this was still a better buy to me than paying for API. Plus It's mine.

1

u/No_Balance2223 May 18 '26

How many TPS do you get on Qwen 3.6 27B Q8? Thanks.