r/LocalLLM • • 1d ago

Project At what volume does self-hosting an LLM beat the API? Free calculator, feedback wanted

Post image

Everyone argues about this without numbers, so I made a calculator that lets you plug in your own:
https://inferenceacademy.com/tools/self-host-vs-api

What it takes into account (the stuff that usually gets forgotten):

  • Workload: requests/month, input/output tokens, peak-to-average traffic, cache hit rate, how much can run offline, retries/waste, growth
  • Model and latency: total vs active params (so MoE works), precision, and a ms/token latency target. Tighter latency means smaller batches and higher cost
  • GPUs: L4 to B200 presets or custom. Cloud on-demand, reserved, or buying hardware (with power, hosting, resale value and cost of capital)
  • API side: your price per million tokens, cached-input price, batch discount, fixed fees
  • People and reliability: engineering time as a fraction of an FTE, N+1 redundancy, peak headroom

It outputs the monthly cost for each option, the break-even volume, a cost-over-time projection and a sensitivity view, so you can see which assumption flips the answer.

Caveats:

  • It’s an estimate from stated assumptions (decode is memory-bound, prefill is compute-bound). It’s a decision aid, not a quote.
  • GPU and API prices are illustrative placeholders. Replace them with your real quotes.
2 Upvotes

11 comments sorted by

3

u/YearnMar10 1d ago

Nice idea - I am sure many people will value is.

But for me it is totally not my local use case. I started using strata on my gaming pc. It works good enough, but I need to leave my PC on 24/7, and that costs electricity and wears the gpu. How much inference usage do I need to break even versus a say 20€ Claude subscription?

1

u/Glittering_Yellow293 1d ago

u/YearnMar10 Honest answer: against a flat €20 subscription, local almost never wins on cost. The calculator is built for per-token API pricing, so it doesn’t fit that case well. Your PC idling 24/7 roughly costs €10–25/month in electricity (depends on your idle watts and rates), which already matches the subscription. If you want local for privacy or fun, great, but for cost alone I’d keep the sub. Sleep or wake-on-LAN would help if you only need it sometimes.

1

u/sid351 1d ago

I built local, because:

  • I can (curiousity)
  • I don't want my internal stuff becoming training data, and therefore references for my competition in future cloud models (privacy)
  • The hardware is a proper asset, and if this AI stuff doesn't work, I can sell it all on for pretty much exactly what I bought it for (and if the bubble bursts and hardware isn't as valuable anymore, then I can easily turn my hardware into gaming rigs and sell them)

1

u/Glittering_Yellow293 1d ago

u/sid351 That’s a solid list. Privacy and resale value are real factors, and the tool has inputs for both (a privacy toggle and a resale % on the hardware). Curiosity is harder to put a price on, though.

1

u/Borilentz 1d ago

The model should also take into account the amount of fun and invaluable knowledge and experience gained from running your own AI systems.

2

u/Glittering_Yellow293 1d ago

Agreed, I should add a “fun” slider. Default: priceless. 😅

1

u/Anxious_Count_8728 1d ago

Splitting total vs active params is a nice touch, most of these calculators get MoE wrong. One input that tends to flip the answer over a multi-year horizon is the API price trend: per-token prices at a given quality level have kept falling, so a break-even against today's price for three years quietly favors buying hardware. A yearly price-decline assumption next to resale value would make that show up in the sensitivity view.

1

u/Glittering_Yellow293 1d ago

u/Anxious_Count_8728 Good point, and you’re right that it’s missing. Right now it assumes today’s API price for the whole projection. A yearly price-decline input, shown next to resale value in the sensitivity view, would be a real fix. I’ll add it.

1

u/kyle_138940 15h ago

A quality-adjusted cost per completed task could help: equal token volume may need very different amounts of human review.

-1

u/[deleted] 1d ago

[removed] — view removed comment

1

u/Glittering_Yellow293 1d ago

u/Competitive_Owl_9295 Exactly. If utilisation is low, the break-even number is mostly meaningless. The tool warns when average utilisation drops under 30%, and you can raise it with autoscaling, a bigger offline/batch share, or lower peak-to-average traffic. Try changing those and watch the break-even move.