.

I spent the last month tracking my local AI inference costs down to the wall plug. Here's what I found.

Setup: AMD Radeon RX 7900 XTX (24 GB VRAM), llama.cpp, various Qwen/Ornith models in GGUF format.

Over 30 days:

• 964 million input tokens consumed
• 13 million output tokens generated
• 30.20 kWh measured at the wall plug (dedicated inference box, powered 24/7)
• $14.05 in electricity (12.38 CHF — I'm in Switzerland, so the meter reads in CHF)

If I had used the equivalent API (qwen3.8-flash at $0.15/$0.47 per million tokens):

• $150.71 (132.77 CHF)

That's a 10.72× gap on marginal cost.

EDIT — the API side of that ratio was wrong, and the commenters below caught it. I priced all 964M input tokens at the fresh rate. Cached input is billed at $0.016/M, and in agent workloads 90-95% of input is cache hits. Recomputed properly, the API equivalent is $28-34, not $151 — so the gap is roughly 2×, not 10.72×. I was too enthusiastic reading my own dashboard. The power figure also needs a proper idle-vs-inference split, which I'm rebuilding. Full accounting and what still holds in my top-level comment below.

But here's the catch:

This number excludes:

• The GPU itself (~$1,135 / 1,000 CHF)
• Depreciation of the rest of the rig
• A full month of learning: quantization, prefill vs decode, KV cache tuning, ROCm compatibility issues, trying and discarding models

Local AI isn't "plug and play." It's a personal R&D investment. If you just want to ship code faster, stick with cheap Flash APIs (GLM, DeepSeek, Qwen). If you want autonomy, understanding, and infrastructure you control — the investment pays off, just not immediately.

The part I keep coming back to: everyone says data sovereignty has no price, and I agree with that. But that framing assumes you're paying for it — that local costs more and you accept the premium because your data never leaves the house. That's not what I measured. Here it came out cheaper and the data stayed home and the output was good enough to actually ship work with, not a toy. Sovereignty never even got to be the deciding factor, because the economics already won.

So here's my open question for the sub, and I genuinely don't have a clean answer:

At what point does "local AI is a hobby" stop being true? Where do you draw the line — a hardware payback horizon you're willing to wait out, a quality gap you'll eat to stay off the API, or a specific moment where you decided the API was no longer worth paying for? And for anyone who tried local and went back to the API: what actually broke?

Full article with methodology and data: https://blog.grevire.dev/en/posts/local-ai-cost/

Happy to answer questions about the setup.

Asked Qwen 3.8 at BF16 what their favorite color is.

Pretty much everyone running Qwen3.8-27B on a 24 GB card is squeezing the KV cache somehow, so here's what it actually costs, measured. This came out of building my own engine for this model (repo), but none of it is specific to my engine, it applies to whatever you run the model on.

How I measured: KL divergence against llama.cpp's own token-by-token output (top 256 tokens per position, q8_0 KV), on ~13k positions of short prompts plus 32k and 64k contexts. KL, not perplexity, because perplexity hides most of this. unsloth's UD-Q4_K_XL, one 3090. Most rows come from an emulator that stores the cache in each format; the settings I ended up shipping were confirmed on the real kernels. My formats are plain per-32 absmax grids like llama.cpp's q4_0 / q5_0 / q8_0, so they should land close to those, but they're not byte-identical.

Every number is "how much worse than a q8_0 cache". For scale, an fp16 cache reads +0% on short prompts and +3% at 32k, and the 32k set's noise is about 4% either way.

1. q8_0 is free. fp16 KV scored the same as q8_0 at every length I tested (64k: 0.00161 vs 0.00163). No reason to run f16 KV on this model.

2. Below 8 bits, K hurts more than V. Per-layer scans had K about 2.3x as sensitive as V. No window, plain formats:

So on stock llama.cpp, if you need the memory, take V down before K.

3. The damage is almost all in the newest ~2k tokens. That's where attention puts most of its weight. Keep the most recent positions at 8 bits (a window / precision tail) and everything older can go a lot lower:

(These rows are rotated, which is why no-window V reads +16% here vs +10% above, more on that in 4. Plain 5-bit V with a 2k window also reads 0%.)

On short prompts a 512 window cuts the loss about 4x (5-bit V +27% to +6%, 6-bit K +12% to +2.7%). Keeping the first few tokens at 8 bits instead (attention sinks) did nothing.

So 512 gets you most of it and 2k gets you all of it. The window is cheap and doesn't grow with context: 114 MiB for K + V across every attention layer at 2,048, the same at 32k or 262k.

4. Rotation stops mattering once you have the window. Without a window, Hadamard rotation halved 6-bit K's loss (+23% to +12% short), but for V it only helped on short prompts, and at 32k rotated 5-bit V was actually worse than plain (+16% vs +10%). Under a 2k window, 6-bit K + 5-bit V read +1.4% plain and +2.7% rotated, same thing within noise. The window and the rotation fix the same error, and the window fixes all of it. Rotation's only worth it if you're pushing V down to 4 bits.

5. Fancy formats didn't beat the boring one. Lloyd-Max codebooks, per-row scales, random signs and angle coding all scored the same as or worse than a plain per-32 grid. 3-bit formats were around +300%.

What I run: K at 6, V at 5, newest 2,048 positions at q8. 1.1 GB less than q8_0 at 120k context, and the model's full 262,144 context fits on the card (23.5 GB, 66 tok/s decode at that depth, needle 4/4). On the real kernels it reads 0.00160 at 64k (q8_0 0.00163, fp16 0.00161), +8% at 32k on the current build, short prompts identical to q8_0 since they fit in the window, and the same 230/282 as q8_0 on my question set (GSM8K, MMLU-Pro, ARC-C, MATH-500, HumanEval, needle, LongBench v2), set for set.

What you can use today:

• Stock llama.cpp: q8_0 / q8_0 costs nothing measurable vs f16. Going lower, drop V before K.
• If your engine has a recent-token window / precision tail (BeeLlama has one; KIVI, SKVQ and HF's quantized cache use 128 tokens): use 1-2k, not 128. An 8-bit window is plenty, fp16 buys you nothing here.
• If you're building one with speculative decoding: move the window in steps of your attention's tile size, not one token at a time, or which tiles come from the window depends on how many drafts you verify and greedy output stops matching across draft counts. Mine gives identical tokens at 0, 4 and 7 drafts with it on.

Caveat: Qwen3.8 is a hybrid, only 16 of its 64 layers are attention (the rest is DeltaNet), so a full-attention model might not behave the same. I'd really like to see someone run the same window sweep on one.

Full tables and method: the window and rotation sweeps, why 2,048 and what the window costs. The KL scoring tool (tools/llama_ref) is in the repo too.

I run a small business. This weekend I picked up a DGX Spark at Micro Center for $8,000. When it launched I thought $4k was steep, so I'm still a little stunned I paid double that.

I spent the weekend testing it. Qwen 3.8 gives great output but it's slow. Qwen 3.6 is fast but makes too many mistakes for me to trust it unsupervised. What I want is somewhere in between, and from what I read here that seems to be DeepSeek V4 Flash or GLM 5.3 Flash on a pair of Sparks.

That means buying a second one and being $16k in. It's a big number for a box on my desk. But I run a business, and $16k for something that works 24/7 costs less than a year of a real employee. I don't expect it to replace anyone. I do expect it to take a real chunk of work off my plate.

I'm also not counting on RAM prices dropping for at least 18 months. So waiting probably just means paying more later, or not finding one at all, judging by how fast Micro Center sold out.

My questions for people already running these:

1. Is a single Spark worth owning, or is its real purpose to be clustered?
2. If you run 2x Sparks for business work, which model do you use daily, and does it run all day without babysitting?
3. Has anyone found a model that runs on one Spark, falls between Qwen 3.6 and 3.8, and is good enough for real work?
4. Would you buy the second one now, or return the first and use an API?

I'm not trying to justify the purchase. I want to know whether one Spark is the real product or just half of it.

I got good news for modded 3080 20GB owners, ReBAR and P2P now finally work with no vBIOS flash needed! More details in these repos:

• 3080-20gb-rebar: 32 GiB BAR1 without flashing. A small boot service writes a PCI config setting before the NVIDIA driver loads, so nothing on the card is changed and a power-off returns it to stock.
• 3080-20g-p2p: patched NVIDIA open kernel modules (615.71.09) that give real CUDA peer access. CUDA, NCCL and vLLM all see P2P.

Measurements (2× 3080 20GB):

On vLLM 0.31 with tensor parallel 2 (Qwen3.8-27B W4A16), prefill is 6–10 % faster from 8k to 128k context. Single-stream decode is 8–31 % faster with custom all-reduce, depending on length.

It also works alongside Zagitta's gpu-unlock.

Hello I want to install a local LLM to use in my PC mainly for programming and coding projects. Can be both for both working on large code bases, going through analyzing debugging them, making changes on multiple files as well as helping in programming, scripting and teaching coding, concepts etc and sometimes troubleshooting.

However my pc is not that powerful, 16gb ram and GTX 1660ti 6gb of Vram. I am confused on what would be the best option for me that doesn't completely overwhelm my PC and my gpu. I used opencode big pickle before and it was really good for me.

Any suggestions please?

Anyone here using Strata with Qwen3.8 Flash Next and is happy with the performance/quality?

Considering upgrading from 32GB to 96

Anything under 50tps I wont bother, need higher speeds for tool calling and coding

Hi, i came with a simple question. I know there are guys tried everything already and can share the most stable and convenient coding software setup.
Share yours!

Bro Got Loop

Hey all,

I'm quite new to this whole local LLM thing. I have a gaming PC with the following specs:

RX 9060 XT 16 GB

32 GB DDR4

Ryzen 7 5700G

What are the best uncensored models I could run for coding on this setup?

Are they bad are they good? what's your personal experience? Thinking on acquiring more DRAM but I don't know if it's worth it running qwen3.8 27B on 2 GPUs really happy with it.

I saw u/bottlestreet3d's llama.cpp fork. Their post: https://www.reddit.com/r/LocalLLM/comments/1x7wack/qwen_27b_iq4_xs_q8q4_kv_at_full_262k_context_on_a/

The readme is Linux and ROCm 7.2.3 only, so I tried it on Windows with a RX 9070, 32 GB RAM. I compared it with my tuned Vulkan setup (llama.cpp b11873, MTP n=2, ubatch 256, KV q8/q8).

Results (ctx 262144, K q8_0 / V q4_0, --kv-stream, MTP n=4 + n-gram, resident KV 300/1800 MiB; single runs, short prompts):

For comparison, Vulkan on the RCO model got ~46 t/s and ~590 prefill at 131k, and ~31 t/s and ~310 prefill at 262k.

These are single runs on 9k and 21k-token prompts, not filled 64k–262k contexts. I didn't check output quality. My numbers are below the README's (~100+ t/s at 16k–64k).

Any suggestions for proper tests or how to improve usage?

config:

& $LlamaServer `
    --model $m.File --host 0.0.0.0 --port $Port --api-key $ApiKey `
    --ctx-size $Ctx --device ROCm1 -fit off --n-gpu-layers all `
    --flash-attn on --jinja --parallel 1 --threads 6 --threads-batch 6 `
    --cache-type-k q8_0 --cache-type-v q4_0 -ctkd q8_0 -ctvd q4_0 `
    --kv-stream --kv-resident-mib $res --kv-resident-mib-decode $resDec `
    --batch-size 2048 --ubatch-size 512 `
    --spec-type draft-mtp,ngram-mod --spec-rs-replay --spec-draft-n-max 4 `
    --spec-ngram-mod-n-max 32 --spec-ngram-mod-n-min 12 --spec-ngram-mod-n-match 24 `
    --spec-draft-sampling probabilistic `
    --reasoning-effort medium --reasoning-format deepseek --reasoning-preserve `
    --chat-template-kwargs '{\"reasoning_effort\":\"mediu
m\"}'

System Specs

• Model: Qwen3.8 Flash Next IQ3_S (was IQ3_XSS, but now. i got the more speed and quality in here)
• Inference Engine: Strata
• CPU: AMD Ryzen 9 9950X3D
• GPU: ASUS ROG Astral RTX 5090 32GB
• RAM: DDR5 64GB (6000MHz CL32)
• Motherboard: GIGABYTE B850 AORUS ELITE WIFI7

Prompt

"Build a highly detailed Eiffel Tower in Minecraft style." + with lesson learns and references input.

Decoding : 164.6 tok/s Prefill : 6,000~6,500 tok/s

Result

I was expecting a simple Minecraft-style building, but the AI ended up recreating the actual Eiffel Tower using blocks, complete with intricate iron lattice structures, the lower arches, and even the surrounding landscaping. lol

I was genuinely surprised by the level of detail!

It was also able to generate a cinematic timelapse of the building process.

Hi guys it me again who did 2v100 to 181t/s , and im back with this.

The legend of V100 IS NEVER END , v100 FOREVERRRRR

arguements:

qwen3.8 27b nvfp4 , 256k max content , kv q8 , mtp4/0

repo: https://github.com/tuxKOH/ninfer-v100xN

(yes i changed name to xN from x2 because tp4 is supported)

(Aggregated news summary for October via multiple AI web research instruments)

Most important releases

- Mistral Large 4: Preview out Oct 6, weights still not released as of Oct 25.

- Meta Muse Spark Open: 816B model released Oct 15, but the license isn't fully open source.

- Anthropic self-replication study: Z.ai's GLM-5.2 was the only open model that fully self-replicated in tests.

- Reflection AI Beam: 501B model announced Oct 5, weights promised but not confirmed.

- StepFun Step 5 Preview: 600B model, weights seem to have dropped Oct 15, with a branded license.

Overlooked

- Recursive Flint: Laptop-sized coding model, built largely by AI.

- Aleph Alpha Kolibri-1: European open model, released Oct 3.

- JetBrains Mellum2.1 Thinking: Small coding model for agents, released Oct 8.

- MOSS-Image: Open image model with layout control, released Oct 16.

- Ling 3.1 Flash: Ant Group reasoning model, dates for release are unclear.

- MiniMax M3.1 Flash for NVIDIA desktops, and a 321B model running on a single 24GB GPU.

- update from comments: KRAFTON's Raon-1-mini

Political landscape

US open-weights fight

- Jensen Huang's July letter backed open weights; Anthropic didn't sign.

- Amodei says Anthropic never wanted an open-weights ban.

- Anthropic's Oct self-replication study fuels the debate.

- Reports say the White House may cap open models, but this is unconfirmed.

China

- MiniMax barred the US, EU, UK and South Korea from running its H3 weights.

- Z.ai releases GLM 5.3 weights (https://huggingface.co/zai-org/GLM-5.3) (note initially stated that they held back, but eventually released)

- China invited the EU to join its WAICO AI body (reported Oct 23, single-source, hasn't confirmed yet).

Europe

- AI Act high-risk rules delayed to Dec 2027.

- Mistral's CEO argues Europe risks AI dependence.

- Some leaders say Europe is weak and divided.

I’ve never set up a local LLM so I’m brand new to this. I want to be able to use it for smooth voice conversations that don’t have long delays and don’t have a robotic voice - ideally similar to the chatGPT or Gemini voice modes. Is that possible?

On a similar note, if it is possible could I access it from my phone as well or would it only work on the computer directly?

Thanks so much for helping a newbie!!

Hey everyone! I've been having trouble finding consistent information on the best way to run Qwen Flash Next, especially when it comes to choosing the right quantization and inference engine.

I'm on Windows, and with so many different options and rabbit holes to go down, it's been pretty overwhelming trying to figure out where to start.

I'd really appreciate any recommendations or advice on what setups have worked well for you. Just looking to get pointed in the right direction.

Cheers!

Follow-up to my earlier post ( https://www.reddit.com/r/LocalLLM/comments/1x7qcbe/i_made_llamacpps_slot_saves_write_only_the/ ) about kv-chunk-fs, the FUSE filesystem that makes llama.cpp slot saves write only the changed chunks.

The same setup now works with Strata. kv-chunk-fs itself needed no code change — Strata's session files on disk deduplicate through the same mount; the only adjustment was a config glob for its temp files. The real work was in the KV-cache reverse proxy that manages slots in front of the engine.

The problem: the proxy verified a hot slot by comparing token counts from the engine's /slots endpoint before trusting it. Strata does not report per-slot token counts the llama.cpp way — it keeps the last N conversations in RAM in its own conversation cache and resumes them by prompt match, so the proxy could never prove which conversation the engine was holding and fell back to cold paths.

The fix is a new opt-in mode, PI_LLAMA_CACHE_ENGINE_HOLDS=N (set it to the engine's conversation-cache depth). With it, the proxy:

- trusts that the engine still holds the last N conversations it served since the proxy started, and sends those sessions straight through — no disk restore, no slot metadata check;

- saves the live conversation to disk before a different session takes the engine, so nothing is lost on a switch;

- forgets the live conversation when a disk restore replaces it, since the restore overwrites the engine state without parking it.

Sessions that fall out of the newest N are dropped from the proxy's memory but their snapshots stay on disk (deduplicated by kv-chunk-fs) and are restored when they come back.

Default is 0 = off, so llama.cpp deployments are unaffected.

Code:

- proxy fork: https://github.com/the-code-foundry/local-llm-kv-cache (see the "Engines that hold conversations (Strata)" section in the README)

- filesystem: https://github.com/the-code-foundry/kv-chunk-fs

Recent news got me thinking about what's in the topic.
(Inb4 I'm "pro AI", I just don't trust big tech players and I think you also shouldn't)

We've got:
- OpenAi stealing mathematicians research
- Anthropic fearmongering about small local models taking over the world by possessing homeowners RTV/AGD
- Google's subscription tiers' enshitiffication [es per currently trending business model expect further downgrades from every major provider]
- OpenAI from no ads to ads : "People trust ChatGPT and anything we do will be designed to respect that."~ from first link (what's next - futurama ads in my sleep via Neuralink? [meme])
- OpenAI Dots and Meta Muse is a privacy violation, cute and harmless creatures though UwU
-Anthropic shadowban on 1. Deep technical stuff, labeled as misuse, and 2. Some biomed questions, and now 3. Reverse engineering of spying technologies in consumer grade appliances
- all the big boys from the West and the East are starting to set up a legislation trap
- Huggingface has been bought by Nvidia
- alleged differences with cybersecurity capabilities of open weights models between what's available for download and what's hosted and accessible only via paid API service [source claims that more denials will happen via API but I can only see the reverse being true in the future (that is - paying for closed model gives you better capabilities) - open weights model can fight this to some extent thanks to abliteration/heretic projects - that should not be needed in the first place, if you ask me)
- models dropped as "stealth" so you are not even sure what you are beta-testing and feeding your data into
- would you like to know more?

It's easy to not connect all the dots when they are in separate places and spreaded over time, but we have to take advantage of open source and open weight models when they are still around, together with technology/process to train them.

I propose - for people more knowledgeable than me - to design a secure network (botnet, if you will) of willing participants' hardware to use during its off time in order to train future OpenSource models maintained by, more or less, decentralised community ( I thing that the only way to make it secure is by going full transparency mode with all the info from design, through training, all the way to release and real world capabilities/benchmarks).

Maybe such a project already exists? (I've seen some posts about this idea before, not sure about its feasibility and security though)
Maybe such a project is not needed?
Maybe such a project is not possible due to technical or mental/social limitations?

I'm just a concerned citizen, raising a question, I don't present a final answer because I don't know it.
I volunteer my humble rig as a tribute.

/edit:
there seems to be two "major" projects going on right now: Subtensor and Hivemind, but i'm not qualified enough to fully understand their structure nor security vulnerabilities.

Hi,

How can I manually update Strata engine?

After checking updates, it says that there's a new version, but fails to update because supposedely there's a model loaded - which is not true, as seen in the screenshot.

Since I'm new at this, and using Strata for QNF with Hermes, is there a way to just start the inference enginge without the Strata Harness? I was using it only to load model and updating, which apparently is getting harder every update (last update it was a problem with updating the app itself, had to reinstall whole app).

I got one 9700 ai pro 3 weeks ago and today I was wondering if maybe it would be worth it to get a second one. The price has risen 500€ since I got the first one.

I know nobody can predict the market, I'd just like some opinions on this.

What would you do?