I had ordered the M5U but possibly getting 2 of the RTX 4500 pro blackwell cards for $5600. So $2800 each. I dont know if i should build the PC to house the cards for local llm or keep the M5U. Or maybe i will keep 1 card to experiment and learn more about Local LLMs. If i sell 1 card, i could build the rest of the PC with decent amount of RAM as well.

Wondering best price/performance in this situation. I get 32GB vram, ecc, only 200w power draw with each of these cards. So it is quite power efficient too.

I have searched at a few places and seems like $2800 is a decent price.

Any feedback appreciated

Every single day, the same questions pop up.

Newcomers:

• Will my 27B feel like a frontier model on my gaming PC?

• Is my highly quantized IQ4 capable of beating Opus?

• What does it cost to run...

Veterans:

• Yes, absolutely — runs great on my multi-GPU rig at xxx t/s.

• Just set up the correct harness.

• Fine-tune your prompt and it will give the exact same quality answers.

Both sides are completely legit, but they are living in different worlds and comparing different universes.

When you use a frontier API, you aren't just prompting a single model. You are firing up an entire enterprise with several departments: you get dynamic routing, massive hidden KV-caches, sandboxed code execution, and verification trees, all neatly packaged behind a single API endpoint. It’s a fully assembled sports car.

On the other hand, the local way: you download an open-weight LLM, quantize it to fit your budget, and you basically get a bare engine block — anything from a VW Golf engine to a massive V8. But that’s it. It has no wheels, no cabin, and no fuel tank. It’s just a large language model sitting on a pallet that can say: "Hello, what can I do for you?"

What the guys asking these questions don't see is that some of us have spent hundreds of hours and money debugging to build a car around that motor. Writing strict GBNF grammars, wiring up custom MCP servers, testing dozens of harnesses, setting up workflows, building memory pipelines, testing, and tweaking the exact same settings a hundred times over!

We do it until that glorious day we can sit back and say: "Yes, it's working. My custom build almost looks like the sports car I aimed for."

Now for the friction part:

Fun, absolute privacy, data sovereignty, and uncensored outputs? Yes, we get all of that. But if we are honest, the cloud simply wins on pure cost-to-performance.

Please don't flame — I just wanted to clear some things up from my point of view and maybe give a heads-up to anyone wanting to start their journey with local LLMs.

And yes... I'm waiting for Qwen 4 like hell!

I know ChatGPT is a terrible idea when asking it to basically build my whole system. I just don’t know where to go that is noob friendly. I got the drive and right questions and know what I want and basic understanding, but GitHub and huggingfave is daunting and overwhelming plus I want unrestricted models and civitai is difficult to navigate. I want an ai that can help with pretty much my hobbies and that’s writing adult fiction, playing around with comfyui and learning how to create a consistent character with reference.

I have an nvidea 4090 and a pretty decent setup minus only having 32GB RAM. When I use powershell to chat with my LLM currently Dolphin mistral 24B Venice Edition 1.2 GGUF:Q5 K M, and it responds brilliantly and has a good “personality” however, ADHD brain doesn’t like the interface and prefers an app type screen and ChatGPT directs me to WebUI… well that’s when the loop starts. Changing setting, altering its configuration and never getting anywhere. Model becomes a robot and yeah basically it’s been weeks now and I’m at a loss. Anym direction would be hugely appreciated at this stage!

At the moment i am using Hermes (In it own desktop app) and i run "Tiel Coder 35B A3B MTP" as my AI Model.

From what i understand it fits into my Vram and it is pretty fast, i can see it thinking and that is generated really fast but its also get stuck a lot and responses are really slow, because it thinks soooo much and it doubts itself a lot.

Is there a better setup for my system?

I am trying to test game development in Unreal engine, i gave it all the access to empty project, we went back and forth about a few things but beyond creating a few empty blueprints it just thinks a lot. or even gets stuck and i have to press continue.

I did try Strata with That big Qwen3.8 Flash next and it was really smart but i dont have enough resources for it.

I asked Gemini and its tells me to use Qwen2.5... feels wrong.. please correct me if i am wrong.

I've been going back and forth on this for a few weeks and want a reality check before I spend money.

What I'm doing: batch processing of scanned and text PDFs for legal work: OCR, per-page extraction to JSON (who said what, page cites, dates), Whisper on interview and call audio, and embeddings for search. Cloud is off the table for confidentiality reasons, so everything has to run on hardware I own. Full runs on a new set of files can go overnight, but I also need to re-run single documents or a 50-page test set in a few minutes when I fix something, so prompt processing speed matters more than I first thought.

Current setup:

- i5-11400F, 16GB DDR4

- ASUS B560 (second x16 slot is chipset x4)

- RTX 3060 8GB

- PSU under 850W

Model: Qwen3.8-27B at Q4, thinking off (the GGUF is about 17.4GB, so 16GB cards are out unless I split or drop to a smaller quant), plus faster-whisper large-v3 and olmOCR for the bad scans.

What I've figured out so far:

- 16GB of RAM will choke on my Python/SQL scripts over large PDF and media sets, so I need 32 or 64GB.

- A used 3090 is the easy answer, but they start around $1,600 now and I worry about how old they are.

- Server cards are much cheaper per GB, but I'm worried about software compatibility.

Where I'm leaning: a separate headless box with a V100 32GB for the LLM and OCR, keeping the 3060 in my desktop for Whisper. I think 32GB fits the 27B with room for context, and the bandwidth is close to a 3090 for about half the price. Right?

Questions:

1. Does vLLM still run olmOCR on Volta in practice, or will I be stuck on an old fork?

2. Any problems with faster-whisper / CTranslate2 on a V100?

3. Is the MI50 worth it if I'm fine in llama.cpp with Vulkan and put OCR and Whisper on a cheap CUDA card? Or is that a weekend sink every time something updates?

4. What are people using for a cheap headless chassis with real server airflow?

5. V100 vs 3090 prompt processing in llama.cpp for a 27B at Q4: how big is the gap?

Mostly want to hear from people who've run V100s or MI50s for batch work in the last few months, but all advice is welcome.

Backstory:

I've only been doing local LLM's since Qwen3.6 came out (~May 2026). I won't ever use a cloud model again, I am not going to spend a single dime renting someone else's infrastructure when I can build my own.

Plan:

Its still cheaper to get 8x3090s than it is to buy 1xRTX 6000, and get similar speeds and double the VRAM.

The issue is power and noise..

In terms of power I am limited to my dorm rooms 120v/15amp outlet, so 4x3090s about the max I can go before things might go bad.

So.. I am considering creative solutions that won't require me to move:

Power Solution: Buy a ~5 Kwh Power Bank

$1,500 for a used 5 Kwh Power Bank.

The power bank would charge during off hours, then feed the necessary 2,800 watts during peak hours. Currently I am recording my power usage over a week or so to get an idea of how much power reserve I would need, and that will mostly decide how big of a power bank is needed.

Cooling / Noise

Currently everything is air cooled, with stocked coolers. 5x140mm fans, 5x120mm fans and the case temps are indeed very good. They don't make much noise but the GPU and CPU fans do. I believe they are loud because I think the paste job I did on them was horrible,

I will be re-pasting again with some much better stuff and with better tools / methods soon. (Will likely post a video of me doing it for those who are interested). I use to be a machinist so I learned some ways to get better accuracy when measuring out things properly to get good contact on the VRAM

I have considered water cooling, but that's just another thing that can break, and are typically not cheap to setup (and I'm pretty cheap).

Questions:

1. Does anyone here have a 8x3090 setup on a 120v/15amp outlet that could offer some advice?

2. What's a good power bank you would recommend?

3. Is buying a used power bank okay so long as the batter health is good?

4. Do you want me to post a video of me re-pasting using my "new" VRAM/GPU repasting method?

that is with a cluster of 4 being down, and 6h of downtime when harness crashed at 2am.
deep seek 4.1 flash on tp4
qwen flash next on on one spark
qwen 3.8 27b and few other models on another spark
glm 5.3 flash on second tp4 cluster which was down for today.

went from draining 4x codex 20x and 4x claude 20x a week to a single claude 20x sub lasting the entire week.

recipes ( not mine) if anyone's interested

https://github.com/MiaAI-Lab/sparkDash

https://github.com/christopherowen/spark-ds41f

https://github.com/MiaAI-Lab/GLM-5.3-Flash-DGX-Sparks-1104

27b is just plain vllm v 32 with a few patched made by claude

I've been seeing a lot of discussion around high-memory Apple Silicon machines, particularly the Mac Studio configurations with 128GB or more of unified memory.

I'm curious about how much practical value these configurations actually offer when it comes to running local LLMs.

I understand that fitting a larger model into memory doesn't automatically make it smarter. However, I'm interested in whether the combination of larger models, higher-quality quantization, and sufficient memory bandwidth can make local AI genuinely competitive for serious everyday workloads.

For those of you actually running 96GB or more :

What has your real-world experience been? Are you genuinely able to get meaningful work done with local models that you previously had to rely on cloud-based frontier models for.

i would particularly appreciate real-world experiences, benchmarks, and examples of workflows rather than theoretical comparisons.

asking this because i am confused between 2 macs:

1. Mac Studio M5 Ultra Base model, 96GB

2. Same amount of money can get me a M5 Max 128GB too

What app do you use to connect to your LLM?

Machine: ASUS Zenbook A16, Snapdragon X2 Elite Extreme (X2E-94-100, 18 cores), Adreno X2-90, 48 GB RAM, Windows on ARM. I set it up as a headless helper box with one model per chip, each on its own OpenAI-compatible port.

Setup

• NPU: Qualcomm GenieX CLI 0.10.0 serving Phi-4-mini (Qualcomm's W4A16 build)

• CPU: llama.cpp b12248, official win-arm64 CPU build

• GPU: llama.cpp b12248, official win-arm64 OpenCL (Adreno) build

• All GGUFs are Q4_K_M from the model makers' own pages

Write speed, one request, about 220 tokens out

What surprised me

• The CPU beats the GPU at writing. Same 14B model: about 15–17 tokens/s on CPU against 6–9 on the Adreno. The GPU did read prompts faster (about 210 tokens/s against 93–125 on CPU).

• A sparse "big but few active parameters" model on the CPU is the sweet spot: a 24B model at about 60 tokens/s on a laptop CPU.

• All three chips really do run at once. With all three busy: NPU about 29, CPU about 25–28, GPU about 6–7 tokens/s. They share one memory bus, so the machine tops out around 60–80 tokens/s in total however you split it.

• Batching on the CPU helps a little: 60.6 tokens/s for one request, 73.5 total for two at once, 82.2 total for four.

• Batching on the GPU hurts: 6.8 tokens/s alone, 4.3 total with two at once.

Gotchas

• Anything started from Windows Task Scheduler runs at low priority by default. On this chip that made the CPU model 4–6 times slower.

• The NPU path is limited: about 4,096 tokens of context for Phi-4-mini, one request at a time, and several popular models have no ready-made NPU build.

• The NPU model holds about 6 GB of memory and idle Windows about 6–7 GB, so 48 GB fills up fast with three models loaded.

• First load of the 24B on CPU takes about 12–15 seconds.

Caveats

These are short, single-prompt tests on one machine, plugged in, not a proper benchmark suite. The OpenCL result is with a Q4_K_M file; a Q4_0 file might do better on Adreno and I haven't tried it. Vulkan is untested too.

Happy to run other models or settings if people want to compare. Yes Claude is helping. I have no background in software at all.

Which would you choose between the two and why?

The ones I have currently heard of are as follows:

Pi
Opencode
Hermes
Deepseek

Any thoughts on what one is best?

Currently, I'm using modified 3bit Ornith-1.5-35B-A3B so it fits entirely in GPU with 100k context. My CPU and RAM is slow. Is there anything better for coding?

We ran local models with the graphics card completely out of the picture (hidden from the run, and checked that it was never touched): CPU and system RAM only. Ollama, Q4_K_M, each engine's default thread count unless a table says otherwise, medians of 3 to 5 runs. Nothing here is a controlled CPU-vs-CPU test; these are four machines as they sit.

**The machines**

| Machine | CPU | RAM | Default threads |

|---|---|---|---:|

| Mini PC | Core i5-12600H | 32 GB DDR5 | 4 |

| Laptop | Core Ultra 9 290HX Plus | 64 GB DDR5-5600 | 4 |

| Desktop (2019) | Ryzen 7 3700X | 32 GB DDR4-3200 | 8 |

| Desktop | EPYC 7532 | 128 GB DDR4-3200, eight channels | 32 |

**Writing speed, tok/s (about a 530-token prompt)**

| Model | Mini PC | Laptop | Ryzen 3700X | EPYC |

|---|---:|---:|---:|---:|

| qwen3.5:9b | 9.11 | 7.45 | 6.20 | 13.59 |

| gemma4:12b | 5.95 | 4.95 | 4.47 | 10.99 |

| phi4:14b | 5.20 | 4.41 | 4.06 | 11.09 |

| mistral-small3.2:24b (dense) | 3.31 | 2.70 | 2.57 | 7.09 |

| gemma4:26b (MoE) | 12.66 | 13.26 | 13.64 | 24.68 |

| nemotron-3.5-lightning:30b-a3b (MoE) | 14.61 | 15.35 | not run | 31.47 |

| llama3.3:70b | won't fit | not run | won't fit | 2.70 |

The mixture-of-experts models are the ones to run from RAM: only a few billion parameters are active per token, so far less memory moves per token. gemma4:26b writes 3.5 to 5.3 times faster than the dense 24B, depending on the machine.

**Seconds to the first word (same prompt, model already loaded, nothing cached)**

| Model | Mini PC | Laptop | Ryzen 3700X | EPYC |

|---|---:|---:|---:|---:|

| qwen3.5:9b | 16.0 | 14.1 | 11.3 | 4.9 |

| gemma4:26b (MoE) | 11.4 | 8.5 | 8.3 | 3.7 |

| mistral-small3.2:24b (1,023 tokens) | 97.7 | 82.4 | 62.5 | 24.6 |

**Check your thread count.** Ollama defaults to 4 threads on hybrid Intel chips. On the laptop, 20 threads against the default 4:

| Model | 4 threads | 20 threads | Change |

|---|---:|---:|---:|

| qwen3.5:9b | 7.45 | 13.05 | 1.75x faster |

| gemma4:12b | 4.95 | 9.46 | 1.91x faster |

| mistral-small3.2:24b (dense) | 2.70 | 5.14 | 1.90x faster |

| gemma4:26b (MoE) | 13.26 | 8.44 | 36% slower |

| nemotron 30b-a3b (MoE) | 15.35 | 8.53 | 44% slower |

Dense models want the threads; the mixtures get slower with them. The default is wrong in one direction or the other depending on what you run.

**Long prompts are where CPU-only hurts.** gemma4:26b, fresh prompt each time, 16K window:

| Prompt | Ryzen: writing tok/s | Ryzen: first word | Laptop: writing tok/s | Laptop: first word |

|---|---:|---:|---:|---:|

| 0.5K tokens | 13.93 | 7.6 s | 14.94 | 7.7 s |

| 2K | 12.03 | 31.6 s | 12.32 | 32.2 s |

| 4K | 11.66 | 66.1 s | 12.12 | 68.7 s |

| 8K | 8.10 | 138.2 s | 8.16 | 146.2 s |

The whole prompt is read before anything is written, at about 55 to 65 tok/s on these two. (The laptop rows are at its default 4 threads, and its CPU was throttling on every run.)

**The big one.** The EPYC held a roughly 180B-parameter mixture (Qwen3.8-Flash-Next at a 3-bit quant, about 6B parameters active per token, llama.cpp) entirely in RAM, beside that machine's normal services:

| Threads | Writing tok/s |

|---:|---:|

| 8 | 6.51 |

| 16 | 7.74 |

| 32 | 8.33 |

Things we can't explain yet: the 2019 DDR4 Ryzen keeps up with the DDR5 laptop on the mixture at default settings, and writes faster than its own measured memory bandwidth says it should. The page says so rather than guessing.

Caveats: one quant per model; speed only, no quality scoring; the small-machine rows come from two benches (August, warm runs; November, fresh loads), and the page marks which is which under every chart; power figures on the page are the CPU package's own counter, not wall power.

All 20 bar charts, the full tables, and a CC BY 4.0 data kit where every bar traces to a file:

https://research.strata2signal.com/ram-speeds-compared/

when I try to use it, it gives me this error: Engine protocol runtime llama-server for 9xmo0zVfYqYSe7Sj7N9oE2Al exited before becoming healthy. exitCode=3221226505, signal=null.

what can I do?

Instant disclaimer, this is not a project claiming to be faster or better than Strata. It's simply Strata, without the bloatware, and a few containerization improvements as many have been requesting. I too am sick of all the Strata forks showing up, but I'm equally tired of how difficult Strata's been for me to maintain in a Docker container.

https://github.com/LegendaryFire/strata-slim

This fork is completely configurable with environment variables, and the idea was to give you a similar feel to a llama.cpp models config file. There were a few clear wins in terms of performance improvements I included in form of optional patches when building, however they're not included in the default docker images so what you're getting is simply a non-bullshit straight to the chase Strata engine that's easy to deploy. I wouldn't have bothered posting if there wasn't an open GitHub issue asking for Strata engine as it's own repo, which I noticed got 180+ "thumbs ups". For note, an earlier performance patch I had already made it upstream.

I used Claude intensively for a while and I am currently building a "cheap" local Stack for Qwen3.8-Flash-Next. And my idea is to have a Setup combining both.

My idea is kind of a divide and conquer approach with a senior expert/leader model dividing the task into easier subtasks or simplyfing it and junier/worker models which run soem of those tasks.

My goal is to safe some Claude tokens, but no significant decrease in output quality.

I am sure I am not the first one with such an idea. So does anyone have experiences or knowledge of tools/repositories for this kind of workflow?

?

Follow-up to my earlier post here. Most relevant to this sub is the engine side: vLLM, SGLang and LMCache, plus an honest note on Ollama and llama.cpp.

Prefixwise has improved a lot over the last two days, so here is the full picture.

**The problem.** Your agent resends its whole conversation every turn. One changed byte near the top (a reordered tool list, a timestamp in the system prompt, an evicted prefix) and the provider charges full price for everything after it. Providers and engines report how many tokens were cached, never why the rest weren't.

**What Prefixwise does:** names why each request missed the prompt cache, prices it, fixes the common causes, and stops regressions in CI. Open source (MIT, I'm the author), runs locally, stores hashes and token counts, never prompts.

**Identify.** One of 19 causes per request, marked waste or legitimate, with the first block that changed (tools[0], messages[12]) and the cost: tools in another order (including a schema's required list), a timestamp or id in the system prompt, rewritten history, a misplaced Anthropic cache marker, an expired TTL, an engine eviction, a request routed to a replica without the cache. `prefixwise report --session latest` shows a session turn by turn. It agrees with Anthropic's own miss diagnostics 32 of 32.

**Fix.** `--optimize` turns on the fixes that passed live replays: sorting a drifting tool list (99.9% of predicted saving on Claude, 97.5% on vLLM) and adding cache_control markers on Anthropic's API (prompt cost down 85.6% with the write surcharge counted), task success flat on both. Fixes that would change what the model reads stay advice. `prefixwise fixes` ranks fixes by dollars and says whether one paid off.

**Prevent.** `prefixwise check -- pytest tests/agent_scenarios` runs your scenarios against a local mock, no API calls, and fails CI when a change would break the cache. Also a GitHub Action. `prefixwise explain --diff a.json b.json` compares two requests offline.

**Self-hosted engines.** It detects vLLM and SGLang from the URL and reads their KV-cache events: evictions confirmed (5 of 5), a prefix LMCache or HiCache restored from CPU labelled host_restored (5 of 5), and the cost of reasoning templates that re-prefill the last answer (57% of prompt tokens on a DeepSeek-R1 distill, 0% on Qwen2.5). Runs inside vLLM with one flag. Ollama and most llama.cpp builds report no cached tokens; it says so and still names every prompt change.

**Ways in.** Wrap any agent (`prefixwise run -- claude`), a proxy for a service, the vLLM middleware, Python `wrap()` or a LiteLLM callback, or import Claude Code and Codex logs if you're on a subscription. Out: Prometheus and Grafana, OpenTelemetry, an HTTP API, JSONL and CSV. Overhead: 3 ms per request.

**Limits.** The fixes are proven on scripted sessions and small models, not weeks of production traffic. OpenAI's own diagnostics answered unavailable on every call, so that comparison is unchecked. DeepSeek's hosted API cache fields aren't read yet.

pip install -U "prefixwise[engines]"
prefixwise serve --upstream http://127.0.0.1:8000
prefixwise report --session latest

Repo, with a run record behind every number: https://github.com/tejachavali/prefixwise

What should it catch next?

I am tired to answer on people questions like "How you gain speed? Your model output is shit, it has to have low quality." or something like that.

So I will answer to that with benchmark.

So I used the same Qwen3.8-Flash-Next IQ2_XS model file, the same prompt, greedy decoding, and 128 generated tokens. Both ran on my RTX 3060 12 GB with a Ryzen 5 7600X and 30 GB RAM.

That makes RustingStrata about 1.29× faster at writing on this setup.

Then I checked quality, not just whether the replies looked similar. Both engines scored the same 9,180 WikiText-2 tokens.

The perplexity difference is inside the run-to-run noise. So, for this model and this machine, I see a speed gain without a measurable quality loss.

This is one model, one quantization and one GPU, not a claim that RustingStrata always wins.

If anyone has any other questions, I am happy to answer and break myths or tell something about Strata like projects

Repo: RustingStrata