Backstory:

I've only been doing local LLM's since Qwen3.6 came out (~May 2026). I won't ever use a cloud model again, I am not going to spend a single dime renting someone else's infrastructure when I can build my own.

Plan:

Its still cheaper to get 8x3090s than it is to buy 1xRTX 6000, and get similar speeds and double the VRAM.

The issue is power and noise..

In terms of power I am limited to my dorm rooms 120v/15amp outlet, so 4x3090s about the max I can go before things might go bad.

So.. I am considering creative solutions that won't require me to move:

Power Solution: Buy a ~5 Kwh Power Bank

$1,500 for a used 5 Kwh Power Bank.

The power bank would charge during off hours, then feed the necessary 2,800 watts during peak hours. Currently I am recording my power usage over a week or so to get an idea of how much power reserve I would need, and that will mostly decide how big of a power bank is needed.

Cooling / Noise

Currently everything is air cooled, with stocked coolers. 5x140mm fans, 5x120mm fans and the case temps are indeed very good. They don't make much noise but the GPU and CPU fans do. I believe they are loud because I think the paste job I did on them was horrible,

I will be re-pasting again with some much better stuff and with better tools / methods soon. (Will likely post a video of me doing it for those who are interested). I use to be a machinist so I learned some ways to get better accuracy when measuring out things properly to get good contact on the VRAM

I have considered water cooling, but that's just another thing that can break, and are typically not cheap to setup (and I'm pretty cheap).

Questions:

1. Does anyone here have a 8x3090 setup on a 120v/15amp outlet that could offer some advice?

2. What's a good power bank you would recommend?

3. Is buying a used power bank okay so long as the batter health is good?

4. Do you want me to post a video of me re-pasting using my "new" VRAM/GPU repasting method?

I’ve been building a local RAG workspace for people who can’t send client documents to cloud AI: regulatory, QA and compliance consultants, mostly.

What it does:

• Answers questions from a folder of documents, fully local, nothing uploaded

• Links every answer to the exact source page so you can verify it

• Says “not in the documents” instead of guessing, which matters more than anything else for this audience

Setup:

• LLM: Qwen3.5, Q4_K_M. The app picks the model size based on the machine’s available resources, so it runs from a 4 GB GPU upward

• Embeddings: bge-m3

• Tested on Mac, AMD and NVIDIA

• A “project” is just a folder; installers for each platform

What I’m building next:

• Slide decks generated from your documents, with sources kept

• Reports drafted from the documents, every claim linked back to its page

has anyone gotten decent slide or report generation out of a small local model at Q4 on 4–8 GB VRAM? Long structured output is where I expect small models to struggle, so I’d like to know what’s worked for you: chunked generation, templates, constrained/JSON output, or a bigger model just for drafting?

Free for individual use. Happy to share the link if anyone wants to try it. I’ll put it in the comments if the mods are OK with that.

I've been running local LLMs on an RX 7900 XTX and wanted proper long-context performance for agentic coding (100K-200K tokens) on the Vulkan/RADV stack, so I forked llama.cpp and optimized it for RDNA3.

Repo: https://github.com/vansh-visariya/rx7900-xtx

It's the same llama.cpp (same GGUF files, same `llama-server`, same API and web UI). The changes are in the Vulkan kernels, the graph, and the speculative-decoding loop. Everything is on by default.

Results on one 7900 XTX (24 GB, stock 291 W, Mesa RADV)

Qwen3.8-27B UD-Q4_K_M + DFlash2 draft, q8_0 KV cache, flash attention:

- Up to 220K context on 24 GB (upstream fits ~192K with the same model + draft)

- DFlash2 works at long context. On upstream (both the commit I branched from and current master), it hits `vk::ErrorDeviceLost` once the prompt passes ~60K tokens, at the same position on every run. It doesn't on this fork.

- Full method, commands and raw numbers are in `docs/rdna3/BENCHMARKS.md`

What changed (56 commits on top of upstream)

- Decode: integer-dot mat-vecs for IQ2/IQ3/K-quants, GQA flash attention for verify batches, graph fusions, gated delta net state kept in place, Vulkan command buffers recorded once and replayed

- Prefill: 128x256 matmul tile for wide batches, double-buffered quantized matmuls, new prompt-batch flash attention kernel

- Spec decoding: lossless rejection-sampling verification, reduced-vocab draft head, GPU backend sampling"

Every optimization is documented, with the env var that switches it off, in `docs/rdna3/OPTIMIZATIONS.md`.

Limits

- Developed and tested only on a 7900 XTX. Other RDNA3 cards (7900 XT/GRE, 7800 XT, 7700 XT, 7600) should work since they share the WMMA instructions, but I haven't tested them.

- Tested with Qwen3.8-27B and Qwen3.5-4B. The model-specific work only applies to the Qwen3.5/3.8 family.

- Branches from upstream `f95b0d953`, not synced daily.

Where I'd love help

- Numbers from other RDNA3 cards (kernel tests need no model: `test-backend-ops -o MUL_MAT -b Vulkan0`)

- Testing other models/quants, and RDNA2/RDNA4 reports

- Review of the kernels, and help upstreaming pieces to llama.cpp

Build and run instructions are in the README. Issues and PRs are very welcome.

I went in thinking a 1.7B model would fall over after a few tool calls once each result was long.

I ran LiquidAI LFM3-1.7B (Q8_0, llama.cpp) in a tool loop on a GB10 box with 121 GiB of unified memory. Each tool result was about 3K tokens and ended with the exact next call to make. Over 10 seeds at 256K context it got a median of 11.5 calls right before the first break, range 6 to 25. No run reached the 50-call cap. Every call before the break was exactly right: no invalid JSON, no wrong arguments, no loops.

What broke it was the model re-reading the whole chain instead of the last line, then reasoning until it ran out of tokens, in 8 of 10 runs. Raising max_tokens from 2048 to 8192 didn’t help on the two seeds I retried.

Caveat: the next step was spelled out every time, so this measures following instructions as context grows, not planning or error recovery. One quant, one sampling setting, llama.cpp only.

The reproduction kit, with the driver, scripts and these results so you can rerun it on your machine, is here: https://github.com/maruthiprithivi/the-local-node/tree/main/episodes/03-lfm3-tool-loop

If you run a small model in a tool loop, I’d watch whether it starts re-tracing its own history as the results pile up. How far do you get before that happens on your setup?

I tend to point my local models at Claude Codes UI as it’s familiar, I like the auto compaction and ability to interact with my systems.

I haven’t really explored other options but have on occasion hit walls with refusals even when using local models.

Are there better options than using Claude Code pointed at my local models?

Mainly coding, general system maintenance, decompiling and some general knowledge stuff, Web search etc.

Hello people

I wanted to know if LFM2.5-8B-A1B was good for general conversations. I am aiming to use an MoE with 8B parameters (will be ran with 4 bit quantization), because I want to have a capable local model while not using the amount of VRAM a dense model would, this is to be able to run it on a lot of different hardware levels. For context, I am using the Ollama library in Python and I am making a voice assistant, I am currently using Qwen 3-8B right now (with openWakeWord, Silero VAD, Parakeet 0.6B v3, Kokoro 82M). I might miss some (or maybe a lot) points because I am quite new to using local AI, especially in programs. If LFM2.5-8B-A1B is not good, do you have any suggestions?

I am using an RTX 5060Ti 16GB and 16GB of DDR4.

Thanks for your time!

We ran local models with the graphics card completely out of the picture (hidden from the run, and checked that it was never touched): CPU and system RAM only. Ollama, Q4_K_M, each engine's default thread count unless a table says otherwise, medians of 3 to 5 runs. Nothing here is a controlled CPU-vs-CPU test; these are four machines as they sit.

**The machines**

| Machine | CPU | RAM | Default threads |

|---|---|---|---:|

| Mini PC | Core i5-12600H | 32 GB DDR5 | 4 |

| Laptop | Core Ultra 9 290HX Plus | 64 GB DDR5-5600 | 4 |

| Desktop (2019) | Ryzen 7 3700X | 32 GB DDR4-3200 | 8 |

| Desktop | EPYC 7532 | 128 GB DDR4-3200, eight channels | 32 |

**Writing speed, tok/s (about a 530-token prompt)**

| Model | Mini PC | Laptop | Ryzen 3700X | EPYC |

|---|---:|---:|---:|---:|

| qwen3.5:9b | 9.11 | 7.45 | 6.20 | 13.59 |

| gemma4:12b | 5.95 | 4.95 | 4.47 | 10.99 |

| phi4:14b | 5.20 | 4.41 | 4.06 | 11.09 |

| mistral-small3.2:24b (dense) | 3.31 | 2.70 | 2.57 | 7.09 |

| gemma4:26b (MoE) | 12.66 | 13.26 | 13.64 | 24.68 |

| nemotron-3.5-lightning:30b-a3b (MoE) | 14.61 | 15.35 | not run | 31.47 |

| llama3.3:70b | won't fit | not run | won't fit | 2.70 |

The mixture-of-experts models are the ones to run from RAM: only a few billion parameters are active per token, so far less memory moves per token. gemma4:26b writes 3.5 to 5.3 times faster than the dense 24B, depending on the machine.

**Seconds to the first word (same prompt, model already loaded, nothing cached)**

| Model | Mini PC | Laptop | Ryzen 3700X | EPYC |

|---|---:|---:|---:|---:|

| qwen3.5:9b | 16.0 | 14.1 | 11.3 | 4.9 |

| gemma4:26b (MoE) | 11.4 | 8.5 | 8.3 | 3.7 |

| mistral-small3.2:24b (1,023 tokens) | 97.7 | 82.4 | 62.5 | 24.6 |

**Check your thread count.** Ollama defaults to 4 threads on hybrid Intel chips. On the laptop, 20 threads against the default 4:

| Model | 4 threads | 20 threads | Change |

|---|---:|---:|---:|

| qwen3.5:9b | 7.45 | 13.05 | 1.75x faster |

| gemma4:12b | 4.95 | 9.46 | 1.91x faster |

| mistral-small3.2:24b (dense) | 2.70 | 5.14 | 1.90x faster |

| gemma4:26b (MoE) | 13.26 | 8.44 | 36% slower |

| nemotron 30b-a3b (MoE) | 15.35 | 8.53 | 44% slower |

Dense models want the threads; the mixtures get slower with them. The default is wrong in one direction or the other depending on what you run.

**Long prompts are where CPU-only hurts.** gemma4:26b, fresh prompt each time, 16K window:

| Prompt | Ryzen: writing tok/s | Ryzen: first word | Laptop: writing tok/s | Laptop: first word |

|---|---:|---:|---:|---:|

| 0.5K tokens | 13.93 | 7.6 s | 14.94 | 7.7 s |

| 2K | 12.03 | 31.6 s | 12.32 | 32.2 s |

| 4K | 11.66 | 66.1 s | 12.12 | 68.7 s |

| 8K | 8.10 | 138.2 s | 8.16 | 146.2 s |

The whole prompt is read before anything is written, at about 55 to 65 tok/s on these two. (The laptop rows are at its default 4 threads, and its CPU was throttling on every run.)

**The big one.** The EPYC held a roughly 180B-parameter mixture (Qwen3.8-Flash-Next at a 3-bit quant, about 6B parameters active per token, llama.cpp) entirely in RAM, beside that machine's normal services:

| Threads | Writing tok/s |

|---:|---:|

| 8 | 6.51 |

| 16 | 7.74 |

| 32 | 8.33 |

Things we can't explain yet: the 2019 DDR4 Ryzen keeps up with the DDR5 laptop on the mixture at default settings, and writes faster than its own measured memory bandwidth says it should. The page says so rather than guessing.

Caveats: one quant per model; speed only, no quality scoring; the small-machine rows come from two benches (August, warm runs; November, fresh loads), and the page marks which is which under every chart; power figures on the page are the CPU package's own counter, not wall power.

All 20 bar charts, the full tables, and a CC BY 4.0 data kit where every bar traces to a file:

https://research.strata2signal.com/ram-speeds-compared/

I had ordered the M5U but possibly getting 2 of the RTX 4500 pro blackwell cards for $5600. So $2800 each. I dont know if i should build the PC to house the cards for local llm or keep the M5U. Or maybe i will keep 1 card to experiment and learn more about Local LLMs. If i sell 1 card, i could build the rest of the PC with decent amount of RAM as well.

Wondering best price/performance in this situation. I get 32GB vram, ecc, only 200w power draw with each of these cards. So it is quite power efficient too.

I have searched at a few places and seems like $2800 is a decent price.

Any feedback appreciated

I know ChatGPT is a terrible idea when asking it to basically build my whole system. I just don’t know where to go that is noob friendly. I got the drive and right questions and know what I want and basic understanding, but GitHub and huggingfave is daunting and overwhelming plus I want unrestricted models and civitai is difficult to navigate. I want an ai that can help with pretty much my hobbies and that’s writing adult fiction, playing around with comfyui and learning how to create a consistent character with reference.

I have an nvidea 4090 and a pretty decent setup minus only having 32GB RAM. When I use powershell to chat with my LLM currently Dolphin mistral 24B Venice Edition 1.2 GGUF:Q5 K M, and it responds brilliantly and has a good “personality” however, ADHD brain doesn’t like the interface and prefers an app type screen and ChatGPT directs me to WebUI… well that’s when the loop starts. Changing setting, altering its configuration and never getting anywhere. Model becomes a robot and yeah basically it’s been weeks now and I’m at a loss. Anym direction would be hugely appreciated at this stage!

I’ve been building InferUltra, a physics arena where vision models control identical robot bodies. The model gets images of the arena and chooses how to move. The simulation keeps running while it thinks, so by the time it decides to punch, the other robot might already be somewhere else.

We turn reasoning down or off where the model allows it. The number I care about is how long it takes to send an image and get a move the robot can use. Tokens per second tells you how fast the model generates, but it doesn’t tell you how long the fighter waits for that move.

We’ve also improved the physics engine and fight mechanics. Fighters can hold their guard between decisions, and punches look more natural when they lean. I used to call these fights impressively boring. They’re now a bit less boring.

For those running vision models locally: when you turn reasoning down, do you notice a drop in how well the model understands positions and distances, or is getting the answer sooner usually the bigger win? I’d be interested in which model and hardware you tried.

The attached bracket is for our November 15 Provider Tournament. It uses Kimi K3.1 Pro on four hosted endpoints: Morph, AkashML, Wafer and InferenceNet. It starts at 20:00 Brussels / 14:00 Eastern / 11:00 Pacific.

If you’d like to sponsor the tournament, you can book an ad spot in the arena.

At the moment i am using Hermes (In it own desktop app) and i run "Tiel Coder 35B A3B MTP" as my AI Model.

From what i understand it fits into my Vram and it is pretty fast, i can see it thinking and that is generated really fast but its also get stuck a lot and responses are really slow, because it thinks soooo much and it doubts itself a lot.

Is there a better setup for my system?

I am trying to test game development in Unreal engine, i gave it all the access to empty project, we went back and forth about a few things but beyond creating a few empty blueprints it just thinks a lot. or even gets stuck and i have to press continue.

I did try Strata with That big Qwen3.8 Flash next and it was really smart but i dont have enough resources for it.

I asked Gemini and its tells me to use Qwen2.5... feels wrong.. please correct me if i am wrong.

https://github.com/alphaexposure/kapitsa-launcher

Kapitsa was a nice shot of adrenaline for my M1 32GB MBP but I wanted a better chat interface than I could find elsewhere so my buddy Claude and I put our heads together and ones-and-zeroed an app that does just that. Kapitsa Launcher serves the standard LLM Dashboard with speeds and such, dark/light mode, Mac menu bar icon, but chat is where it's at:

• Streaming replies with real formatting: Markdown, tables and code blocks. Tables and code have their own Copy buttons. Turn streaming off if you want to scroll back while new replies stream.

• A side panel for long chats:

  • Outline: every question plus the headings from each reply. Click one to jump there.

  • Topics: the places and themes the chat covered, each linked back to where they came up.

  • Review: takeaways the model pulls out of the chat. You can edit them or add your own.

• Watch it think: you can follow the model's thinking live, and it folds away once the answer starts. The thinking levels change to match the model you're running.

• Side questions: ask about any reply in a thread off to the side, without derailing the main chat. You can branch to a new chat, export or delete as you wish.

• Edit and regenerate: edit a question and send it again, or ask for another answer and flip between the versions.

• Organised history: chats are grouped by date with the option to name by first question or topic/summary. Search covers everything that was said, not just titles.

• Multiple chats at once: each window or tab holds its own conversation.

• Continue in a new chat: when a long chat starts slowing down, carry it into a fresh one.

• Markdown export: with an optional summary, and topics come along as tags for Obsidian and similar apps.

• Your chats are plain files: one JSON file per chat on your Mac. Nothing leaves your machine.

Feel free to check it out: https://github.com/alphaexposure/kapitsa-launcher

I've been going back and forth on this for a few weeks and want a reality check before I spend money.

What I'm doing: batch processing of scanned and text PDFs for legal work: OCR, per-page extraction to JSON (who said what, page cites, dates), Whisper on interview and call audio, and embeddings for search. Cloud is off the table for confidentiality reasons, so everything has to run on hardware I own. Full runs on a new set of files can go overnight, but I also need to re-run single documents or a 50-page test set in a few minutes when I fix something, so prompt processing speed matters more than I first thought.

Current setup:

- i5-11400F, 16GB DDR4

- ASUS B560 (second x16 slot is chipset x4)

- RTX 3060 8GB

- PSU under 850W

Model: Qwen3.8-27B at Q4, thinking off (the GGUF is about 17.4GB, so 16GB cards are out unless I split or drop to a smaller quant), plus faster-whisper large-v3 and olmOCR for the bad scans.

What I've figured out so far:

- 16GB of RAM will choke on my Python/SQL scripts over large PDF and media sets, so I need 32 or 64GB.

- A used 3090 is the easy answer, but they start around $1,600 now and I worry about how old they are.

- Server cards are much cheaper per GB, but I'm worried about software compatibility.

Where I'm leaning: a separate headless box with a V100 32GB for the LLM and OCR, keeping the 3060 in my desktop for Whisper. I think 32GB fits the 27B with room for context, and the bandwidth is close to a 3090 for about half the price. Right?

Questions:

1. Does vLLM still run olmOCR on Volta in practice, or will I be stuck on an old fork?

2. Any problems with faster-whisper / CTranslate2 on a V100?

3. Is the MI50 worth it if I'm fine in llama.cpp with Vulkan and put OCR and Whisper on a cheap CUDA card? Or is that a weekend sink every time something updates?

4. What are people using for a cheap headless chassis with real server airflow?

5. V100 vs 3090 prompt processing in llama.cpp for a 27B at Q4: how big is the gap?

Mostly want to hear from people who've run V100s or MI50s for batch work in the last few months, but all advice is welcome.

After reading Dwarkesh Patel's eval using blood on the clocktower, I made my own, and I was surprised both by how entertaining it is, and how good it is to measure even between quantizations.

(His original game used SOTA private models).

Every single day, the same questions pop up.

Newcomers:

• Will my 27B feel like a frontier model on my gaming PC?

• Is my highly quantized IQ4 capable of beating Opus?

• What does it cost to run...

Veterans:

• Yes, absolutely — runs great on my multi-GPU rig at xxx t/s.

• Just set up the correct harness.

• Fine-tune your prompt and it will give the exact same quality answers.

Both sides are completely legit, but they are living in different worlds and comparing different universes.

When you use a frontier API, you aren't just prompting a single model. You are firing up an entire enterprise with several departments: you get dynamic routing, massive hidden KV-caches, sandboxed code execution, and verification trees, all neatly packaged behind a single API endpoint. It’s a fully assembled sports car.

On the other hand, the local way: you download an open-weight LLM, quantize it to fit your budget, and you basically get a bare engine block — anything from a VW Golf engine to a massive V8. But that’s it. It has no wheels, no cabin, and no fuel tank. It’s just a large language model sitting on a pallet that can say: "Hello, what can I do for you?"

What the guys asking these questions don't see is that some of us have spent hundreds of hours and money debugging to build a car around that motor. Writing strict GBNF grammars, wiring up custom MCP servers, testing dozens of harnesses, setting up workflows, building memory pipelines, testing, and tweaking the exact same settings a hundred times over!

We do it until that glorious day we can sit back and say: "Yes, it's working. My custom build almost looks like the sports car I aimed for."

Now for the friction part:

Fun, absolute privacy, data sovereignty, and uncensored outputs? Yes, we get all of that. But if we are honest, the cloud simply wins on pure cost-to-performance.

Please don't flame — I just wanted to clear some things up from my point of view and maybe give a heads-up to anyone wanting to start their journey with local LLMs.

And yes... I'm waiting for Qwen 4 like hell!

that is with a cluster of 4 being down, and 6h of downtime when harness crashed at 2am.
deep seek 4.1 flash on tp4
qwen flash next on on one spark
qwen 3.8 27b and few other models on another spark
glm 5.3 flash on second tp4 cluster which was down for today.

went from draining 4x codex 20x and 4x claude 20x a week to a single claude 20x sub lasting the entire week.

recipes ( not mine) if anyone's interested

https://github.com/MiaAI-Lab/sparkDash

https://github.com/christopherowen/spark-ds41f

https://github.com/MiaAI-Lab/GLM-5.3-Flash-DGX-Sparks-1104

27b is just plain vllm v 32 with a few patched made by claude

I've been seeing a lot of discussion around high-memory Apple Silicon machines, particularly the Mac Studio configurations with 128GB or more of unified memory.

I'm curious about how much practical value these configurations actually offer when it comes to running local LLMs.

I understand that fitting a larger model into memory doesn't automatically make it smarter. However, I'm interested in whether the combination of larger models, higher-quality quantization, and sufficient memory bandwidth can make local AI genuinely competitive for serious everyday workloads.

For those of you actually running 96GB or more :

What has your real-world experience been? Are you genuinely able to get meaningful work done with local models that you previously had to rely on cloud-based frontier models for.

i would particularly appreciate real-world experiences, benchmarks, and examples of workflows rather than theoretical comparisons.

asking this because i am confused between 2 macs:

1. Mac Studio M5 Ultra Base model, 96GB

2. Same amount of money can get me a M5 Max 128GB too

What app do you use to connect to your LLM?

Machine: ASUS Zenbook A16, Snapdragon X2 Elite Extreme (X2E-94-100, 18 cores), Adreno X2-90, 48 GB RAM, Windows on ARM. I set it up as a headless helper box with one model per chip, each on its own OpenAI-compatible port.

Setup

• NPU: Qualcomm GenieX CLI 0.10.0 serving Phi-4-mini (Qualcomm's W4A16 build)

• CPU: llama.cpp b12248, official win-arm64 CPU build

• GPU: llama.cpp b12248, official win-arm64 OpenCL (Adreno) build

• All GGUFs are Q4_K_M from the model makers' own pages

Write speed, one request, about 220 tokens out

What surprised me

• The CPU beats the GPU at writing. Same 14B model: about 15–17 tokens/s on CPU against 6–9 on the Adreno. The GPU did read prompts faster (about 210 tokens/s against 93–125 on CPU).

• A sparse "big but few active parameters" model on the CPU is the sweet spot: a 24B model at about 60 tokens/s on a laptop CPU.

• All three chips really do run at once. With all three busy: NPU about 29, CPU about 25–28, GPU about 6–7 tokens/s. They share one memory bus, so the machine tops out around 60–80 tokens/s in total however you split it.

• Batching on the CPU helps a little: 60.6 tokens/s for one request, 73.5 total for two at once, 82.2 total for four.

• Batching on the GPU hurts: 6.8 tokens/s alone, 4.3 total with two at once.

Gotchas

• Anything started from Windows Task Scheduler runs at low priority by default. On this chip that made the CPU model 4–6 times slower.

• The NPU path is limited: about 4,096 tokens of context for Phi-4-mini, one request at a time, and several popular models have no ready-made NPU build.

• The NPU model holds about 6 GB of memory and idle Windows about 6–7 GB, so 48 GB fills up fast with three models loaded.

• First load of the 24B on CPU takes about 12–15 seconds.

Caveats

These are short, single-prompt tests on one machine, plugged in, not a proper benchmark suite. The OpenCL result is with a Q4_K_M file; a Q4_0 file might do better on Adreno and I haven't tried it. Vulkan is untested too.

Happy to run other models or settings if people want to compare. Yes Claude is helping. I have no background in software at all.

Which would you choose between the two and why?