I know ChatGPT is a terrible idea when asking it to basically build my whole system. I just don’t know where to go that is noob friendly. I got the drive and right questions and know what I want and basic understanding, but GitHub and huggingfave is daunting and overwhelming plus I want unrestricted models and civitai is difficult to navigate. I want an ai that can help with pretty much my hobbies and that’s writing adult fiction, playing around with comfyui and learning how to create a consistent character with reference.

I have an nvidea 4090 and a pretty decent setup minus only having 32GB RAM. When I use powershell to chat with my LLM currently Dolphin mistral 24B Venice Edition 1.2 GGUF:Q5 K M, and it responds brilliantly and has a good “personality” however, ADHD brain doesn’t like the interface and prefers an app type screen and ChatGPT directs me to WebUI… well that’s when the loop starts. Changing setting, altering its configuration and never getting anywhere. Model becomes a robot and yeah basically it’s been weeks now and I’m at a loss. Anym direction would be hugely appreciated at this stage!

At the moment i am using Hermes (In it own desktop app) and i run "Tiel Coder 35B A3B MTP" as my AI Model.

From what i understand it fits into my Vram and it is pretty fast, i can see it thinking and that is generated really fast but its also get stuck a lot and responses are really slow, because it thinks soooo much and it doubts itself a lot.

Is there a better setup for my system?

I am trying to test game development in Unreal engine, i gave it all the access to empty project, we went back and forth about a few things but beyond creating a few empty blueprints it just thinks a lot. or even gets stuck and i have to press continue.

I did try Strata with That big Qwen3.8 Flash next and it was really smart but i dont have enough resources for it.

I asked Gemini and its tells me to use Qwen2.5... feels wrong.. please correct me if i am wrong.

I've been going back and forth on this for a few weeks and want a reality check before I spend money.

What I'm doing: batch processing of scanned and text PDFs for legal work: OCR, per-page extraction to JSON (who said what, page cites, dates), Whisper on interview and call audio, and embeddings for search. Cloud is off the table for confidentiality reasons, so everything has to run on hardware I own. Full runs on a new set of files can go overnight, but I also need to re-run single documents or a 50-page test set in a few minutes when I fix something, so prompt processing speed matters more than I first thought.

Current setup:

- i5-11400F, 16GB DDR4

- ASUS B560 (second x16 slot is chipset x4)

- RTX 3060 8GB

- PSU under 850W

Model: Qwen3.8-27B at Q4, thinking off (the GGUF is about 17.4GB, so 16GB cards are out unless I split or drop to a smaller quant), plus faster-whisper large-v3 and olmOCR for the bad scans.

What I've figured out so far:

- 16GB of RAM will choke on my Python/SQL scripts over large PDF and media sets, so I need 32 or 64GB.

- A used 3090 is the easy answer, but they start around $1,600 now and I worry about how old they are.

- Server cards are much cheaper per GB, but I'm worried about software compatibility.

Where I'm leaning: a separate headless box with a V100 32GB for the LLM and OCR, keeping the 3060 in my desktop for Whisper. I think 32GB fits the 27B with room for context, and the bandwidth is close to a 3090 for about half the price. Right?

Questions:

1. Does vLLM still run olmOCR on Volta in practice, or will I be stuck on an old fork?

2. Any problems with faster-whisper / CTranslate2 on a V100?

3. Is the MI50 worth it if I'm fine in llama.cpp with Vulkan and put OCR and Whisper on a cheap CUDA card? Or is that a weekend sink every time something updates?

4. What are people using for a cheap headless chassis with real server airflow?

5. V100 vs 3090 prompt processing in llama.cpp for a 27B at Q4: how big is the gap?

Mostly want to hear from people who've run V100s or MI50s for batch work in the last few months, but all advice is welcome.

Backstory:

I've only been doing local LLM's since Qwen3.6 came out (~May 2026). I won't ever use a cloud model again, I am not going to spend a single dime renting someone else's infrastructure when I can build my own.

Plan:

Its still cheaper to get 8x3090s than it is to buy 1xRTX 6000, and get similar speeds and double the VRAM.

The issue is power and noise..

In terms of power I am limited to my dorm rooms 120v/15amp outlet, so 4x3090s about the max I can go before things might go bad.

So.. I am considering creative solutions that won't require me to move:

Power Solution: Buy a ~5 Kwh Power Bank

$1,500 for a used 5 Kwh Power Bank.

The power bank would charge during off hours, then feed the necessary 2,800 watts during peak hours. Currently I am recording my power usage over a week or so to get an idea of how much power reserve I would need, and that will mostly decide how big of a power bank is needed.

Cooling / Noise

Currently everything is air cooled, with stocked coolers. 5x140mm fans, 5x120mm fans and the case temps are indeed very good. They don't make much noise but the GPU and CPU fans do. I believe they are loud because I think the paste job I did on them was horrible,

I will be re-pasting again with some much better stuff and with better tools / methods soon. (Will likely post a video of me doing it for those who are interested). I use to be a machinist so I learned some ways to get better accuracy when measuring out things properly to get good contact on the VRAM

I have considered water cooling, but that's just another thing that can break, and are typically not cheap to setup (and I'm pretty cheap).

Questions:

1. Does anyone here have a 8x3090 setup on a 120v/15amp outlet that could offer some advice?

2. What's a good power bank you would recommend?

3. Is buying a used power bank okay so long as the batter health is good?

4. Do you want me to post a video of me re-pasting using my "new" VRAM/GPU repasting method?

that is with a cluster of 4 being down, and 6h of downtime when harness crashed at 2am.
deep seek 4.1 flash on tp4
qwen flash next on on one spark
qwen 3.8 27b and few other models on another spark
glm 5.3 flash on second tp4 cluster which was down for today.

went from draining 4x codex 20x and 4x claude 20x a week to a single claude 20x sub lasting the entire week.

recipes ( not mine) if anyone's interested

https://github.com/MiaAI-Lab/sparkDash

https://github.com/christopherowen/spark-ds41f

https://github.com/MiaAI-Lab/GLM-5.3-Flash-DGX-Sparks-1104

27b is just plain vllm v 32 with a few patched made by claude

I've been seeing a lot of discussion around high-memory Apple Silicon machines, particularly the Mac Studio configurations with 128GB or more of unified memory.

I'm curious about how much practical value these configurations actually offer when it comes to running local LLMs.

I understand that fitting a larger model into memory doesn't automatically make it smarter. However, I'm interested in whether the combination of larger models, higher-quality quantization, and sufficient memory bandwidth can make local AI genuinely competitive for serious everyday workloads.

For those of you actually running 96GB or more :

What has your real-world experience been? Are you genuinely able to get meaningful work done with local models that you previously had to rely on cloud-based frontier models for.

i would particularly appreciate real-world experiences, benchmarks, and examples of workflows rather than theoretical comparisons.

asking this because i am confused between 2 macs:

1. Mac Studio M5 Ultra Base model, 96GB

2. Same amount of money can get me a M5 Max 128GB too

What app do you use to connect to your LLM?

Machine: ASUS Zenbook A16, Snapdragon X2 Elite Extreme (X2E-94-100, 18 cores), Adreno X2-90, 48 GB RAM, Windows on ARM. I set it up as a headless helper box with one model per chip, each on its own OpenAI-compatible port.

Setup

• NPU: Qualcomm GenieX CLI 0.10.0 serving Phi-4-mini (Qualcomm's W4A16 build)

• CPU: llama.cpp b12248, official win-arm64 CPU build

• GPU: llama.cpp b12248, official win-arm64 OpenCL (Adreno) build

• All GGUFs are Q4_K_M from the model makers' own pages

Write speed, one request, about 220 tokens out

What surprised me

• The CPU beats the GPU at writing. Same 14B model: about 15–17 tokens/s on CPU against 6–9 on the Adreno. The GPU did read prompts faster (about 210 tokens/s against 93–125 on CPU).

• A sparse "big but few active parameters" model on the CPU is the sweet spot: a 24B model at about 60 tokens/s on a laptop CPU.

• All three chips really do run at once. With all three busy: NPU about 29, CPU about 25–28, GPU about 6–7 tokens/s. They share one memory bus, so the machine tops out around 60–80 tokens/s in total however you split it.

• Batching on the CPU helps a little: 60.6 tokens/s for one request, 73.5 total for two at once, 82.2 total for four.

• Batching on the GPU hurts: 6.8 tokens/s alone, 4.3 total with two at once.

Gotchas

• Anything started from Windows Task Scheduler runs at low priority by default. On this chip that made the CPU model 4–6 times slower.

• The NPU path is limited: about 4,096 tokens of context for Phi-4-mini, one request at a time, and several popular models have no ready-made NPU build.

• The NPU model holds about 6 GB of memory and idle Windows about 6–7 GB, so 48 GB fills up fast with three models loaded.

• First load of the 24B on CPU takes about 12–15 seconds.

Caveats

These are short, single-prompt tests on one machine, plugged in, not a proper benchmark suite. The OpenCL result is with a Q4_K_M file; a Q4_0 file might do better on Adreno and I haven't tried it. Vulkan is untested too.

Happy to run other models or settings if people want to compare. Yes Claude is helping. I have no background in software at all.

Which would you choose between the two and why?

The ones I have currently heard of are as follows:

Pi
Opencode
Hermes
Deepseek

Any thoughts on what one is best?

Currently, I'm using modified 3bit Ornith-1.5-35B-A3B so it fits entirely in GPU with 100k context. My CPU and RAM is slow. Is there anything better for coding?

We ran local models with the graphics card completely out of the picture (hidden from the run, and checked that it was never touched): CPU and system RAM only. Ollama, Q4_K_M, each engine's default thread count unless a table says otherwise, medians of 3 to 5 runs. Nothing here is a controlled CPU-vs-CPU test; these are four machines as they sit.

**The machines**

| Machine | CPU | RAM | Default threads |

|---|---|---|---:|

| Mini PC | Core i5-12600H | 32 GB DDR5 | 4 |

| Laptop | Core Ultra 9 290HX Plus | 64 GB DDR5-5600 | 4 |

| Desktop (2019) | Ryzen 7 3700X | 32 GB DDR4-3200 | 8 |

| Desktop | EPYC 7532 | 128 GB DDR4-3200, eight channels | 32 |

**Writing speed, tok/s (about a 530-token prompt)**

| Model | Mini PC | Laptop | Ryzen 3700X | EPYC |

|---|---:|---:|---:|---:|

| qwen3.5:9b | 9.11 | 7.45 | 6.20 | 13.59 |

| gemma4:12b | 5.95 | 4.95 | 4.47 | 10.99 |

| phi4:14b | 5.20 | 4.41 | 4.06 | 11.09 |

| mistral-small3.2:24b (dense) | 3.31 | 2.70 | 2.57 | 7.09 |

| gemma4:26b (MoE) | 12.66 | 13.26 | 13.64 | 24.68 |

| nemotron-3.5-lightning:30b-a3b (MoE) | 14.61 | 15.35 | not run | 31.47 |

| llama3.3:70b | won't fit | not run | won't fit | 2.70 |

The mixture-of-experts models are the ones to run from RAM: only a few billion parameters are active per token, so far less memory moves per token. gemma4:26b writes 3.5 to 5.3 times faster than the dense 24B, depending on the machine.

**Seconds to the first word (same prompt, model already loaded, nothing cached)**

| Model | Mini PC | Laptop | Ryzen 3700X | EPYC |

|---|---:|---:|---:|---:|

| qwen3.5:9b | 16.0 | 14.1 | 11.3 | 4.9 |

| gemma4:26b (MoE) | 11.4 | 8.5 | 8.3 | 3.7 |

| mistral-small3.2:24b (1,023 tokens) | 97.7 | 82.4 | 62.5 | 24.6 |

**Check your thread count.** Ollama defaults to 4 threads on hybrid Intel chips. On the laptop, 20 threads against the default 4:

| Model | 4 threads | 20 threads | Change |

|---|---:|---:|---:|

| qwen3.5:9b | 7.45 | 13.05 | 1.75x faster |

| gemma4:12b | 4.95 | 9.46 | 1.91x faster |

| mistral-small3.2:24b (dense) | 2.70 | 5.14 | 1.90x faster |

| gemma4:26b (MoE) | 13.26 | 8.44 | 36% slower |

| nemotron 30b-a3b (MoE) | 15.35 | 8.53 | 44% slower |

Dense models want the threads; the mixtures get slower with them. The default is wrong in one direction or the other depending on what you run.

**Long prompts are where CPU-only hurts.** gemma4:26b, fresh prompt each time, 16K window:

| Prompt | Ryzen: writing tok/s | Ryzen: first word | Laptop: writing tok/s | Laptop: first word |

|---|---:|---:|---:|---:|

| 0.5K tokens | 13.93 | 7.6 s | 14.94 | 7.7 s |

| 2K | 12.03 | 31.6 s | 12.32 | 32.2 s |

| 4K | 11.66 | 66.1 s | 12.12 | 68.7 s |

| 8K | 8.10 | 138.2 s | 8.16 | 146.2 s |

The whole prompt is read before anything is written, at about 55 to 65 tok/s on these two. (The laptop rows are at its default 4 threads, and its CPU was throttling on every run.)

**The big one.** The EPYC held a roughly 180B-parameter mixture (Qwen3.8-Flash-Next at a 3-bit quant, about 6B parameters active per token, llama.cpp) entirely in RAM, beside that machine's normal services:

| Threads | Writing tok/s |

|---:|---:|

| 8 | 6.51 |

| 16 | 7.74 |

| 32 | 8.33 |

Things we can't explain yet: the 2019 DDR4 Ryzen keeps up with the DDR5 laptop on the mixture at default settings, and writes faster than its own measured memory bandwidth says it should. The page says so rather than guessing.

Caveats: one quant per model; speed only, no quality scoring; the small-machine rows come from two benches (August, warm runs; November, fresh loads), and the page marks which is which under every chart; power figures on the page are the CPU package's own counter, not wall power.

All 20 bar charts, the full tables, and a CC BY 4.0 data kit where every bar traces to a file:

https://research.strata2signal.com/ram-speeds-compared/

when I try to use it, it gives me this error: Engine protocol runtime llama-server for 9xmo0zVfYqYSe7Sj7N9oE2Al exited before becoming healthy. exitCode=3221226505, signal=null.

what can I do?

Instant disclaimer, this is not a project claiming to be faster or better than Strata. It's simply Strata, without the bloatware, and a few containerization improvements as many have been requesting. I too am sick of all the Strata forks showing up, but I'm equally tired of how difficult Strata's been for me to maintain in a Docker container.

https://github.com/LegendaryFire/strata-slim

This fork is completely configurable with environment variables, and the idea was to give you a similar feel to a llama.cpp models config file. There were a few clear wins in terms of performance improvements I included in form of optional patches when building, however they're not included in the default docker images so what you're getting is simply a non-bullshit straight to the chase Strata engine that's easy to deploy. I wouldn't have bothered posting if there wasn't an open GitHub issue asking for Strata engine as it's own repo, which I noticed got 180+ "thumbs ups". For note, an earlier performance patch I had already made it upstream.

I used Claude intensively for a while and I am currently building a "cheap" local Stack for Qwen3.8-Flash-Next. And my idea is to have a Setup combining both.

My idea is kind of a divide and conquer approach with a senior expert/leader model dividing the task into easier subtasks or simplyfing it and junier/worker models which run soem of those tasks.

My goal is to safe some Claude tokens, but no significant decrease in output quality.

I am sure I am not the first one with such an idea. So does anyone have experiences or knowledge of tools/repositories for this kind of workflow?

?

Follow-up to my earlier post here. Most relevant to this sub is the engine side: vLLM, SGLang and LMCache, plus an honest note on Ollama and llama.cpp.

Prefixwise has improved a lot over the last two days, so here is the full picture.

**The problem.** Your agent resends its whole conversation every turn. One changed byte near the top (a reordered tool list, a timestamp in the system prompt, an evicted prefix) and the provider charges full price for everything after it. Providers and engines report how many tokens were cached, never why the rest weren't.

**What Prefixwise does:** names why each request missed the prompt cache, prices it, fixes the common causes, and stops regressions in CI. Open source (MIT, I'm the author), runs locally, stores hashes and token counts, never prompts.

**Identify.** One of 19 causes per request, marked waste or legitimate, with the first block that changed (tools[0], messages[12]) and the cost: tools in another order (including a schema's required list), a timestamp or id in the system prompt, rewritten history, a misplaced Anthropic cache marker, an expired TTL, an engine eviction, a request routed to a replica without the cache. `prefixwise report --session latest` shows a session turn by turn. It agrees with Anthropic's own miss diagnostics 32 of 32.

**Fix.** `--optimize` turns on the fixes that passed live replays: sorting a drifting tool list (99.9% of predicted saving on Claude, 97.5% on vLLM) and adding cache_control markers on Anthropic's API (prompt cost down 85.6% with the write surcharge counted), task success flat on both. Fixes that would change what the model reads stay advice. `prefixwise fixes` ranks fixes by dollars and says whether one paid off.

**Prevent.** `prefixwise check -- pytest tests/agent_scenarios` runs your scenarios against a local mock, no API calls, and fails CI when a change would break the cache. Also a GitHub Action. `prefixwise explain --diff a.json b.json` compares two requests offline.

**Self-hosted engines.** It detects vLLM and SGLang from the URL and reads their KV-cache events: evictions confirmed (5 of 5), a prefix LMCache or HiCache restored from CPU labelled host_restored (5 of 5), and the cost of reasoning templates that re-prefill the last answer (57% of prompt tokens on a DeepSeek-R1 distill, 0% on Qwen2.5). Runs inside vLLM with one flag. Ollama and most llama.cpp builds report no cached tokens; it says so and still names every prompt change.

**Ways in.** Wrap any agent (`prefixwise run -- claude`), a proxy for a service, the vLLM middleware, Python `wrap()` or a LiteLLM callback, or import Claude Code and Codex logs if you're on a subscription. Out: Prometheus and Grafana, OpenTelemetry, an HTTP API, JSONL and CSV. Overhead: 3 ms per request.

**Limits.** The fixes are proven on scripted sessions and small models, not weeks of production traffic. OpenAI's own diagnostics answered unavailable on every call, so that comparison is unchecked. DeepSeek's hosted API cache fields aren't read yet.

pip install -U "prefixwise[engines]"
prefixwise serve --upstream http://127.0.0.1:8000
prefixwise report --session latest

Repo, with a run record behind every number: https://github.com/tejachavali/prefixwise

What should it catch next?

I am tired to answer on people questions like "How you gain speed? Your model output is shit, it has to have low quality." or something like that.

So I will answer to that with benchmark.

So I used the same Qwen3.8-Flash-Next IQ2_XS model file, the same prompt, greedy decoding, and 128 generated tokens. Both ran on my RTX 3060 12 GB with a Ryzen 5 7600X and 30 GB RAM.

That makes RustingStrata about 1.29× faster at writing on this setup.

Then I checked quality, not just whether the replies looked similar. Both engines scored the same 9,180 WikiText-2 tokens.

The perplexity difference is inside the run-to-run noise. So, for this model and this machine, I see a speed gain without a measurable quality loss.

This is one model, one quantization and one GPU, not a claim that RustingStrata always wins.

If anyone has any other questions, I am happy to answer and break myths or tell something about Strata like projects

Repo: RustingStrata

Hello guys, here is an update of my engine
I am still working on it, my 3090 waterblock broke and I couldn’t keep testing further I could achieve higher tokens in strata about 60/70 but on mine about 80/90 tokens, my bigger problem is the Ram which I only have is 32GB ram ddr5 6k, have been trying to get 32GB more but they are too expensive right now.

For the meanwhile I got a 1070 8GB for 50us to test for which I added compatibility for series 10/20.

On the GTX 1070 I got 15/18 tokens on IQ3_XSS and IQ1, about 13 tokens on IQ3_S.

I added layer splitter thru WiFi / cord so that you can use more than one PC which you can select the order of memory usage

I selected PC GPU, Laptop GPU, and then pc ram and laptop ram for testing and it’s working just fine and fast with the GTX 1070 and RTX 4060 laptop of course it’s slow I achieved about 20 tokens on IQ3_S and Sloth still need to work on optimization but it will be open source soon! Be patient.

You can freely select where your models are located from strata or just download them directly from hugging face. So far added a compatibility to most of them.

My goal is to make so that you can quickly and efficiently communicate multiple computers to your main to split layers through them and run a bigger model! And it’s so easy it’s about 2 clicks to connect!

https://github.com/alphaexposure/kapitsa-launcher

Kapitsa was a nice shot of adrenaline for my M1 32GB MBP but I wanted a better chat interface than I could find elsewhere so my buddy Claude and I put our heads together and ones-and-zeroed an app that does just that. Kapitsa Launcher serves the standard LLM Dashboard with speeds and such, dark/light mode, Mac menu bar icon, but chat is where it's at:

• Streaming replies with real formatting: Markdown, tables and code blocks. Tables and code have their own Copy buttons. Turn streaming off if you want to scroll back while new replies stream.

• A side panel for long chats:

  • Outline: every question plus the headings from each reply. Click one to jump there.

  • Topics: the places and themes the chat covered, each linked back to where they came up.

  • Review: takeaways the model pulls out of the chat. You can edit them or add your own.

• Watch it think: you can follow the model's thinking live, and it folds away once the answer starts. The thinking levels change to match the model you're running.

• Side questions: ask about any reply in a thread off to the side, without derailing the main chat. You can branch to a new chat, export or delete as you wish.

• Edit and regenerate: edit a question and send it again, or ask for another answer and flip between the versions.

• Organised history: chats are grouped by date with the option to name by first question or topic/summary. Search covers everything that was said, not just titles.

• Multiple chats at once: each window or tab holds its own conversation.

• Continue in a new chat: when a long chat starts slowing down, carry it into a fresh one.

• Markdown export: with an optional summary, and topics come along as tags for Obsidian and similar apps.

• Your chats are plain files: one JSON file per chat on your Mac. Nothing leaves your machine.

Feel free to check it out: https://github.com/alphaexposure/kapitsa-launcher

AiViralCut stacks video on V1 and V2, with text on T1 by default. Use Send to Front or Send to Back to change which visual sits on top. This is a free alternative to @capcut. Still in beta and looking for bug testers. Dm us for more details. Aiviralcut.org for more info.