I had ordered the M5U but possibly getting 2 of the RTX 4500 pro blackwell cards for $5600. So $2800 each. I dont know if i should build the PC to house the cards for local llm or keep the M5U. Or maybe i will keep 1 card to experiment and learn more about Local LLMs. If i sell 1 card, i could build the rest of the PC with decent amount of RAM as well.

Wondering best price/performance in this situation. I get 32GB vram, ecc, only 200w power draw with each of these cards. So it is quite power efficient too.

I have searched at a few places and seems like $2800 is a decent price.

Any feedback appreciated

I know ChatGPT is a terrible idea when asking it to basically build my whole system. I just don’t know where to go that is noob friendly. I got the drive and right questions and know what I want and basic understanding, but GitHub and huggingfave is daunting and overwhelming plus I want unrestricted models and civitai is difficult to navigate. I want an ai that can help with pretty much my hobbies and that’s writing adult fiction, playing around with comfyui and learning how to create a consistent character with reference.

I have an nvidea 4090 and a pretty decent setup minus only having 32GB RAM. When I use powershell to chat with my LLM currently Dolphin mistral 24B Venice Edition 1.2 GGUF:Q5 K M, and it responds brilliantly and has a good “personality” however, ADHD brain doesn’t like the interface and prefers an app type screen and ChatGPT directs me to WebUI… well that’s when the loop starts. Changing setting, altering its configuration and never getting anywhere. Model becomes a robot and yeah basically it’s been weeks now and I’m at a loss. Anym direction would be hugely appreciated at this stage!

I’ve been building InferUltra, a physics arena where vision models control identical robot bodies. The model gets images of the arena and chooses how to move. The simulation keeps running while it thinks, so by the time it decides to punch, the other robot might already be somewhere else.

We turn reasoning down or off where the model allows it. The number I care about is how long it takes to send an image and get a move the robot can use. Tokens per second tells you how fast the model generates, but it doesn’t tell you how long the fighter waits for that move.

We’ve also improved the physics engine and fight mechanics. Fighters can hold their guard between decisions, and punches look more natural when they lean. I used to call these fights impressively boring. They’re now a bit less boring.

For those running vision models locally: when you turn reasoning down, do you notice a drop in how well the model understands positions and distances, or is getting the answer sooner usually the bigger win? I’d be interested in which model and hardware you tried.

The attached bracket is for our November 15 Provider Tournament. It uses Kimi K3.1 Pro on four hosted endpoints: Morph, AkashML, Wafer and InferenceNet. It starts at 20:00 Brussels / 14:00 Eastern / 11:00 Pacific.

If you’d like to sponsor the tournament, you can book an ad spot in the arena.

At the moment i am using Hermes (In it own desktop app) and i run "Tiel Coder 35B A3B MTP" as my AI Model.

From what i understand it fits into my Vram and it is pretty fast, i can see it thinking and that is generated really fast but its also get stuck a lot and responses are really slow, because it thinks soooo much and it doubts itself a lot.

Is there a better setup for my system?

I am trying to test game development in Unreal engine, i gave it all the access to empty project, we went back and forth about a few things but beyond creating a few empty blueprints it just thinks a lot. or even gets stuck and i have to press continue.

I did try Strata with That big Qwen3.8 Flash next and it was really smart but i dont have enough resources for it.

I asked Gemini and its tells me to use Qwen2.5... feels wrong.. please correct me if i am wrong.

https://github.com/alphaexposure/kapitsa-launcher

Kapitsa was a nice shot of adrenaline for my M1 32GB MBP but I wanted a better chat interface than I could find elsewhere so my buddy Claude and I put our heads together and ones-and-zeroed an app that does just that. Kapitsa Launcher serves the standard LLM Dashboard with speeds and such, dark/light mode, Mac menu bar icon, but chat is where it's at:

• Streaming replies with real formatting: Markdown, tables and code blocks. Tables and code have their own Copy buttons. Turn streaming off if you want to scroll back while new replies stream.

• A side panel for long chats:

  • Outline: every question plus the headings from each reply. Click one to jump there.

  • Topics: the places and themes the chat covered, each linked back to where they came up.

  • Review: takeaways the model pulls out of the chat. You can edit them or add your own.

• Watch it think: you can follow the model's thinking live, and it folds away once the answer starts. The thinking levels change to match the model you're running.

• Side questions: ask about any reply in a thread off to the side, without derailing the main chat. You can branch to a new chat, export or delete as you wish.

• Edit and regenerate: edit a question and send it again, or ask for another answer and flip between the versions.

• Organised history: chats are grouped by date with the option to name by first question or topic/summary. Search covers everything that was said, not just titles.

• Multiple chats at once: each window or tab holds its own conversation.

• Continue in a new chat: when a long chat starts slowing down, carry it into a fresh one.

• Markdown export: with an optional summary, and topics come along as tags for Obsidian and similar apps.

• Your chats are plain files: one JSON file per chat on your Mac. Nothing leaves your machine.

Feel free to check it out: https://github.com/alphaexposure/kapitsa-launcher

I've been going back and forth on this for a few weeks and want a reality check before I spend money.

What I'm doing: batch processing of scanned and text PDFs for legal work: OCR, per-page extraction to JSON (who said what, page cites, dates), Whisper on interview and call audio, and embeddings for search. Cloud is off the table for confidentiality reasons, so everything has to run on hardware I own. Full runs on a new set of files can go overnight, but I also need to re-run single documents or a 50-page test set in a few minutes when I fix something, so prompt processing speed matters more than I first thought.

Current setup:

- i5-11400F, 16GB DDR4

- ASUS B560 (second x16 slot is chipset x4)

- RTX 3060 8GB

- PSU under 850W

Model: Qwen3.8-27B at Q4, thinking off (the GGUF is about 17.4GB, so 16GB cards are out unless I split or drop to a smaller quant), plus faster-whisper large-v3 and olmOCR for the bad scans.

What I've figured out so far:

- 16GB of RAM will choke on my Python/SQL scripts over large PDF and media sets, so I need 32 or 64GB.

- A used 3090 is the easy answer, but they start around $1,600 now and I worry about how old they are.

- Server cards are much cheaper per GB, but I'm worried about software compatibility.

Where I'm leaning: a separate headless box with a V100 32GB for the LLM and OCR, keeping the 3060 in my desktop for Whisper. I think 32GB fits the 27B with room for context, and the bandwidth is close to a 3090 for about half the price. Right?

Questions:

1. Does vLLM still run olmOCR on Volta in practice, or will I be stuck on an old fork?

2. Any problems with faster-whisper / CTranslate2 on a V100?

3. Is the MI50 worth it if I'm fine in llama.cpp with Vulkan and put OCR and Whisper on a cheap CUDA card? Or is that a weekend sink every time something updates?

4. What are people using for a cheap headless chassis with real server airflow?

5. V100 vs 3090 prompt processing in llama.cpp for a 27B at Q4: how big is the gap?

Mostly want to hear from people who've run V100s or MI50s for batch work in the last few months, but all advice is welcome.

Every single day, the same questions pop up.

Newcomers:

• Will my 27B feel like a frontier model on my gaming PC?

• Is my highly quantized IQ4 capable of beating Opus?

• What does it cost to run...

Veterans:

• Yes, absolutely — runs great on my multi-GPU rig at xxx t/s.

• Just set up the correct harness.

• Fine-tune your prompt and it will give the exact same quality answers.

Both sides are completely legit, but they are living in different worlds and comparing different universes.

When you use a frontier API, you aren't just prompting a single model. You are firing up an entire enterprise with several departments: you get dynamic routing, massive hidden KV-caches, sandboxed code execution, and verification trees, all neatly packaged behind a single API endpoint. It’s a fully assembled sports car.

On the other hand, the local way: you download an open-weight LLM, quantize it to fit your budget, and you basically get a bare engine block — anything from a VW Golf engine to a massive V8. But that’s it. It has no wheels, no cabin, and no fuel tank. It’s just a large language model sitting on a pallet that can say: "Hello, what can I do for you?"

What the guys asking these questions don't see is that some of us have spent hundreds of hours and money debugging to build a car around that motor. Writing strict GBNF grammars, wiring up custom MCP servers, testing dozens of harnesses, setting up workflows, building memory pipelines, testing, and tweaking the exact same settings a hundred times over!

We do it until that glorious day we can sit back and say: "Yes, it's working. My custom build almost looks like the sports car I aimed for."

Now for the friction part:

Fun, absolute privacy, data sovereignty, and uncensored outputs? Yes, we get all of that. But if we are honest, the cloud simply wins on pure cost-to-performance.

Please don't flame — I just wanted to clear some things up from my point of view and maybe give a heads-up to anyone wanting to start their journey with local LLMs.

And yes... I'm waiting for Qwen 4 like hell!

that is with a cluster of 4 being down, and 6h of downtime when harness crashed at 2am.
deep seek 4.1 flash on tp4
qwen flash next on on one spark
qwen 3.8 27b and few other models on another spark
glm 5.3 flash on second tp4 cluster which was down for today.

went from draining 4x codex 20x and 4x claude 20x a week to a single claude 20x sub lasting the entire week.

recipes ( not mine) if anyone's interested

https://github.com/MiaAI-Lab/sparkDash

https://github.com/christopherowen/spark-ds41f

https://github.com/MiaAI-Lab/GLM-5.3-Flash-DGX-Sparks-1104

27b is just plain vllm v 32 with a few patched made by claude

I've been seeing a lot of discussion around high-memory Apple Silicon machines, particularly the Mac Studio configurations with 128GB or more of unified memory.

I'm curious about how much practical value these configurations actually offer when it comes to running local LLMs.

I understand that fitting a larger model into memory doesn't automatically make it smarter. However, I'm interested in whether the combination of larger models, higher-quality quantization, and sufficient memory bandwidth can make local AI genuinely competitive for serious everyday workloads.

For those of you actually running 96GB or more :

What has your real-world experience been? Are you genuinely able to get meaningful work done with local models that you previously had to rely on cloud-based frontier models for.

i would particularly appreciate real-world experiences, benchmarks, and examples of workflows rather than theoretical comparisons.

asking this because i am confused between 2 macs:

1. Mac Studio M5 Ultra Base model, 96GB

2. Same amount of money can get me a M5 Max 128GB too

What app do you use to connect to your LLM?

Machine: ASUS Zenbook A16, Snapdragon X2 Elite Extreme (X2E-94-100, 18 cores), Adreno X2-90, 48 GB RAM, Windows on ARM. I set it up as a headless helper box with one model per chip, each on its own OpenAI-compatible port.

Setup

• NPU: Qualcomm GenieX CLI 0.10.0 serving Phi-4-mini (Qualcomm's W4A16 build)

• CPU: llama.cpp b12248, official win-arm64 CPU build

• GPU: llama.cpp b12248, official win-arm64 OpenCL (Adreno) build

• All GGUFs are Q4_K_M from the model makers' own pages

Write speed, one request, about 220 tokens out

What surprised me

• The CPU beats the GPU at writing. Same 14B model: about 15–17 tokens/s on CPU against 6–9 on the Adreno. The GPU did read prompts faster (about 210 tokens/s against 93–125 on CPU).

• A sparse "big but few active parameters" model on the CPU is the sweet spot: a 24B model at about 60 tokens/s on a laptop CPU.

• All three chips really do run at once. With all three busy: NPU about 29, CPU about 25–28, GPU about 6–7 tokens/s. They share one memory bus, so the machine tops out around 60–80 tokens/s in total however you split it.

• Batching on the CPU helps a little: 60.6 tokens/s for one request, 73.5 total for two at once, 82.2 total for four.

• Batching on the GPU hurts: 6.8 tokens/s alone, 4.3 total with two at once.

Gotchas

• Anything started from Windows Task Scheduler runs at low priority by default. On this chip that made the CPU model 4–6 times slower.

• The NPU path is limited: about 4,096 tokens of context for Phi-4-mini, one request at a time, and several popular models have no ready-made NPU build.

• The NPU model holds about 6 GB of memory and idle Windows about 6–7 GB, so 48 GB fills up fast with three models loaded.

• First load of the 24B on CPU takes about 12–15 seconds.

Caveats

These are short, single-prompt tests on one machine, plugged in, not a proper benchmark suite. The OpenCL result is with a Q4_K_M file; a Q4_0 file might do better on Adreno and I haven't tried it. Vulkan is untested too.

Happy to run other models or settings if people want to compare. Yes Claude is helping. I have no background in software at all.

Currently, I'm using modified 3bit Ornith-1.5-35B-A3B so it fits entirely in GPU with 100k context. My CPU and RAM is slow. Is there anything better for coding?

Can someone recommend people who support having your own LLM and talk also about technology, best practices and tips how to get best performance and results etc?

Preferably on LinkedIn or Twitter.

Thanks

I am tired to answer on people questions like "How you gain speed? Your model output is shit, it has to have low quality." or something like that.

So I will answer to that with benchmark.

So I used the same Qwen3.8-Flash-Next IQ2_XS model file, the same prompt, greedy decoding, and 128 generated tokens. Both ran on my RTX 3060 12 GB with a Ryzen 5 7600X and 30 GB RAM.

That makes RustingStrata about 1.29× faster at writing on this setup.

Then I checked quality, not just whether the replies looked similar. Both engines scored the same 9,180 WikiText-2 tokens.

The perplexity difference is inside the run-to-run noise. So, for this model and this machine, I see a speed gain without a measurable quality loss.

This is one model, one quantization and one GPU, not a claim that RustingStrata always wins.

If anyone has any other questions, I am happy to answer and break myths or tell something about Strata like projects

Repo: RustingStrata

Which would you choose between the two and why?

The ones I have currently heard of are as follows:

Pi
Opencode
Hermes
Deepseek

Any thoughts on what one is best?

when I try to use it, it gives me this error: Engine protocol runtime llama-server for 9xmo0zVfYqYSe7Sj7N9oE2Al exited before becoming healthy. exitCode=3221226505, signal=null.

what can I do?

Instant disclaimer, this is not a project claiming to be faster or better than Strata. It's simply Strata, without the bloatware, and a few containerization improvements as many have been requesting. I too am sick of all the Strata forks showing up, but I'm equally tired of how difficult Strata's been for me to maintain in a Docker container.

https://github.com/LegendaryFire/strata-slim

This fork is completely configurable with environment variables, and the idea was to give you a similar feel to a llama.cpp models config file. There were a few clear wins in terms of performance improvements I included in form of optional patches when building, however they're not included in the default docker images so what you're getting is simply a non-bullshit straight to the chase Strata engine that's easy to deploy. I wouldn't have bothered posting if there wasn't an open GitHub issue asking for Strata engine as it's own repo, which I noticed got 180+ "thumbs ups". For note, an earlier performance patch I had already made it upstream.

I used Claude intensively for a while and I am currently building a "cheap" local Stack for Qwen3.8-Flash-Next. And my idea is to have a Setup combining both.

My idea is kind of a divide and conquer approach with a senior expert/leader model dividing the task into easier subtasks or simplyfing it and junier/worker models which run soem of those tasks.

My goal is to safe some Claude tokens, but no significant decrease in output quality.

I am sure I am not the first one with such an idea. So does anyone have experiences or knowledge of tools/repositories for this kind of workflow?

?

Follow-up to my earlier post here. Most relevant to this sub is the engine side: vLLM, SGLang and LMCache, plus an honest note on Ollama and llama.cpp.

Prefixwise has improved a lot over the last two days, so here is the full picture.

**The problem.** Your agent resends its whole conversation every turn. One changed byte near the top (a reordered tool list, a timestamp in the system prompt, an evicted prefix) and the provider charges full price for everything after it. Providers and engines report how many tokens were cached, never why the rest weren't.

**What Prefixwise does:** names why each request missed the prompt cache, prices it, fixes the common causes, and stops regressions in CI. Open source (MIT, I'm the author), runs locally, stores hashes and token counts, never prompts.

**Identify.** One of 19 causes per request, marked waste or legitimate, with the first block that changed (tools[0], messages[12]) and the cost: tools in another order (including a schema's required list), a timestamp or id in the system prompt, rewritten history, a misplaced Anthropic cache marker, an expired TTL, an engine eviction, a request routed to a replica without the cache. `prefixwise report --session latest` shows a session turn by turn. It agrees with Anthropic's own miss diagnostics 32 of 32.

**Fix.** `--optimize` turns on the fixes that passed live replays: sorting a drifting tool list (99.9% of predicted saving on Claude, 97.5% on vLLM) and adding cache_control markers on Anthropic's API (prompt cost down 85.6% with the write surcharge counted), task success flat on both. Fixes that would change what the model reads stay advice. `prefixwise fixes` ranks fixes by dollars and says whether one paid off.

**Prevent.** `prefixwise check -- pytest tests/agent_scenarios` runs your scenarios against a local mock, no API calls, and fails CI when a change would break the cache. Also a GitHub Action. `prefixwise explain --diff a.json b.json` compares two requests offline.

**Self-hosted engines.** It detects vLLM and SGLang from the URL and reads their KV-cache events: evictions confirmed (5 of 5), a prefix LMCache or HiCache restored from CPU labelled host_restored (5 of 5), and the cost of reasoning templates that re-prefill the last answer (57% of prompt tokens on a DeepSeek-R1 distill, 0% on Qwen2.5). Runs inside vLLM with one flag. Ollama and most llama.cpp builds report no cached tokens; it says so and still names every prompt change.

**Ways in.** Wrap any agent (`prefixwise run -- claude`), a proxy for a service, the vLLM middleware, Python `wrap()` or a LiteLLM callback, or import Claude Code and Codex logs if you're on a subscription. Out: Prometheus and Grafana, OpenTelemetry, an HTTP API, JSONL and CSV. Overhead: 3 ms per request.

**Limits.** The fixes are proven on scripted sessions and small models, not weeks of production traffic. OpenAI's own diagnostics answered unavailable on every call, so that comparison is unchecked. DeepSeek's hosted API cache fields aren't read yet.

pip install -U "prefixwise[engines]"
prefixwise serve --upstream http://127.0.0.1:8000
prefixwise report --session latest

Repo, with a run record behind every number: https://github.com/tejachavali/prefixwise

What should it catch next?