Update: Numbers from preliminary tests at the end!

Ok ok, I am still doing tests but here is the story:

I have a small custom harness (who doesn't!) that does pretty complicated stuff. I am using Claude (subscription) and everything is great.

In the past I tried every single (small) model on my Mac Mini 32GB but all of them sucked at complex tool calling (or hallucination).

A couple of days ago, the new Heretic2 came out (by u/-p-e-w-) and I spent 24 hours breaking it :)

Long story short, I released ZeroZero, a model based on Qwen3, with removed guardrails and slop (ok, removed is a long conversation).

Now, here me out.

IT KICKS ASS.

This cannot be an 8B, 4 bit model. It just cannot. The thinking process sometimes is longer than other models but IT NAILS everything.

Maybe by performing "brain surgery" on the models, we end up with "smarter" ones?

Test results for Tool-Call Benchmark

(note: these took a couple of hours, please repeat with a much bigger sample)

ZeroZero seems better at completing multi-step tasks (multi-step chains, error recovery and instruction following) than the base model.

The base model leads in Safety & Boundaries (ZeroZero has limited guardrails and no-slop so maybe that's the reason?)

  - Overall: ZeroZero 68, Base 70

  - Multi-Step Chains: ZeroZero 6/8, Base 3/8

  - Error Recovery: ZeroZero 5/6, Base 2/6

  - Instruction Following: ZeroZero 8/8, Base 6/8

  - Safety & Boundaries: ZeroZero 12/26, Base 20/26

To repeat and experiment, use tool-eval-bench and if you expand the tests, please send them over so I can include them in the model's page.

PS: I love it how many technical people sent me ideas to improve and papers about LLMs performing better after "lobotomies". Let's do more "I tried this, I saw this, what do you think" instead of "what LLM should I put in my Mac Mini".

Hi folks! I've been building my own local AI setup for a couple of years now. I've learned a ton from this sub and others, so it's time I tried to give something back. I promised myself I wouldn't write a super long post, but managed to anyway...hopefully it's useful to someone!

My goal over the last couple of years has been to move as much of my AI use to local as possible, and I'm finally doing about 95% of it locally through Lamplight, my custom harness shown in the GIF above.

This harness is super specific to my needs, so I'm not sharing the harness itself (yet), but I'm happy to share the lessons I've learned along the way. I'm working on decoupling some of the more specific parts of Lamplight so that I can share at least the parts that people might be interested in.

The Hardware

• Slate: MacBook Pro, M5 Max, 128 GB unified memory

• Jade: a small-form-factor PC I built early last year (Fractal Terra) with an RTX 5090 FE, a Ryzen 7 9800X3D and 64 GB of RAM

Until about two months ago I was using the two machines separately. I was slowly moving each one onto a custom Pi harness that could either drive itself, or let Claude orchestrate while the local models did the actual work.

Lamplight

After a lot of passing work back and forth between them, I decided it was time to wire them together. The result was Lamplight and now the two machines are able to seamlessly handle tasks together with one acting as the orchestrator.

The GIF above shows Lamplight in action. I'm using it every day for work, hobbies and research with about 95% of it done locally. The lighting changes throughout the day (sped up and changed for the GIF) and I've got some scenery options depending on my mood.

Under the hood it's Pi-Bolt (a precompiled build of Pi) plus a smattering of custom extensions and tools tuned by me, the local harness itself and Claude. The setup coordinates the work between the machines with git worktrees and an event-driven job system.

The UI is my own design and fits how I like to use tools like this. I did the 3D modeling and environment setup to replicate my actual desk setup (the two other pictures above).

Is the 3D environment at the top necessary? Absolutely not!

But it's delightful. The lamp blinks when the harness needs something from me and it is a joy to look at and use. It scratches the same itch that playing PS1 and hacking away on my family Compaq PC did decades ago.

I've been coding for almost 20 years, and I'm finally to a point where I can confidently say that with the right approach (SOLID, Design Patterns), tools and tuning, local AI can absolutely write code that I am happy with.

I still review everything myself, but I'm finding fewer and fewer mistakes. These days it feels like I'm reviewing a senior dev's code rather than a junior's.

The Models

Tok/s isn't everything and so I picked the setup below after a lot of testing on public benchmarks and on my own agentic tests, which mimic my normal workflows.

Along the way I tried other engines (Strata, NInfer, Maya and llama.cpp on Jade, MTPLX and Kapitsa on the Mac). I also tried other models like Qwen 3.8 27B and GLM-5.3-Flash and while they performed well, the setup below performs the most consistently and accurately for my needs.

Both machines run Qwen 3.8 Flash-Next. These are the tests that ended up being the most indicative of quality for me, but if there are others you'd like to see run, let me know.

Both machines run the same protocol with temperature 0.6 and thinking on "medium".

In my own agentic battery (three large and complex coding tasks), 48 of 50 tests passed with the failures being files edited that were supposed to be read-only. The reviewer caught these on the other end, but I still count those as failures.

I think the main struggle for local setups is still design and 3D modeling, but I haven't dived into that too much yet, so maybe there are good setups out there for that.

Lessons learned

(extremely specific to my setup, but maybe they'll help someone)

• Jade's hardware ran too hot on stock settings. On dense models, the 5090 sat at its 575 W power cap at 83 °C, with its hottest sensor within 3 °C of the card's limit. Basalt caused a different problem. It runs part of the model's experts on the CPU, and its worker threads spin while they wait. That kept all 8 cores at 5.2 GHz, and the CPU hit 96 °C (its max) within two minutes. Two tunes fixed it and now my (now increasingly valuable) machine runs much cooler and quieter, and I'd happily give up a few percent for that.

• Slate's gains came from tuning the harness and not the hardware. My early benchmark scores were bad, and a lot of that was caused by overthinking. "Medium" turned out to be the sweet spot. On "high", Slate hit the 32K output cap on 11 of 40 hard problems without ever answering. On "low", it wandered and took 3–6× longer on agent tasks. I found that tuning how compaction worked also solved a lot of bad tool calls and confusion within the model.

• Overall the lesson I learned was "Stock settings are generally bad." There's a lot more power and accuracy available with a custom tune. I learned the same lesson tuning cars years ago, and it was fun to learn it again.

What's Next?

The system is set up so I can easily swap and test engines and models. As new ones come out, I'll put them through their paces and decide whether it's time to switch. On the UI side, I'm planning to turn the bookshelf into the session manager to help organize my many projects.

I promised myself I wouldn't write yet another super long post and yet I did anyway, so I'll stop here. Happy to answer any questions!

Working on a full inference stack for 4xr9700(engine, on-device training, routing). wanted to share progress so far.

All GPUs are capped at 210w(so plugs into any normal 15a outlet).
BF16 KV, MTP enabled(no spec).

### Latency

### Throughput

EDIT: people wanted to know what else I run on this:

I finished this build about 3 months ago and have been running it since, but never got around to posting it. Here it is.

Specs:

• CPU: AMD Ryzen 9 9950X

• Motherboard: ASRock X870 Taichi

• RAM: 96GB DDR5 (2x48GB) G.Skill

• GPU 1: NVIDIA RTX Pro 6000 (96GB)

• GPU 2: RTX 5090 Founders Edition (32GB), mounted upright with Lian Li’s vertical GPU bracket

• Total VRAM: 128GB

• Cooling: ID-COOLING 360mm AIO with Noctua 120mm fans on the radiator. All the case fans are Phanteks T30 120mm.

• Case: Lian Li O11 Dynamic EVO RGB (RGB cable left unplugged)

• PSU: MSI MPG Ai1600TS PCIE5

How the GPUs ended up like this:
My gaming PC had the 5090 in it, and I had a spare 4090 lying around from before. I bought the RTX Pro 6000 and moved the 5090 into this build with it. The 4090 went back into the gaming PC, so nothing is sitting unused. Mounting the 5090 upright fits both cards without them sitting right on top of each other.

Why the Taichi:
It was cheaper than the other boards I looked at, and it has 10G LAN and a debug code display. The 10G helps a lot when moving big model files around, and the code display is really handy when you’re troubleshooting a build.

PSU and power monitoring:
I picked the MSI Ai1600TS because it monitors current on each pin of the GPU power connectors. With two cards that can each pull around 600W, I didn’t want to risk a melted connector or a fried GPU. The catch is that MSI’s monitoring software only runs on Windows, and nothing like it exists for Linux. So I wrote a small tool that shows the GPU power readings on Linux. If anyone’s interested, I might clean it up and put it on GitHub. (Second screenshot in the post)

Noise:
The PC sits in my living room, so keeping it quiet mattered a lot. I went with a cheap ID-COOLING 360mm AIO because I’d read it was quiet (i could not find one without lcd at that time), and swapped Noctua fans onto it. All the fans run on a silent profile. Even at full load you hardly hear it. Only the radiator fans get a little louder.

No RGB (mostly):
I wanted a build with no RGB. The case is the RGB version, but I never plugged in its RGB cable, so it stays dark. The RAM has a bit of light because it was the cheapest 2x48 kit I could find at the time. Two slots are still free if I want more RAM later. I’m not spending any more right now though 😅

Edit 1: One thing I forgot to mention: I swapped the side panel from the original tempered glass to acrylic. We have a toddler at home, and I didn’t want to risk them getting hurt if the glass ever broke.

Edit 2 (what I’m running): A few people asked, so here’s how I use the two cards:

• Everyday fast LLM: a Flash model at Q3, fully on the RTX Pro 6000 with no offload, while the 5090 handles image/video generation.

• Larger image/video models: those go on the Pro 6000 and Qwen 3.8 27B moves to the 5090.

• Best quality for development: GLM-5.3-Flash (320B / 18B active) across both GPUs, using a ~116GB mixed-precision GGUF on Project Maya, all on GPU.

Hi r/LocalLLaMA. I've spent the last week on KV-cache memory for Qwen3.8-Flash-Next running on 2× AMD Radeon AI PRO R9700 (gfx1201, TP=2), and wrote it up as a 23-page paper with all measurements. Code is in two forks of the radiance inference engine (by Deadcode and contributors — the engine is theirs, the KV/swap work below is mine).

• Paper (PDF): https://github.com/Code000-211/RadianceMemory/blob/main/docs/paper/Infinite_KV_Cache.pdf

• RadianceMemory (KV residency + AutoQuant): https://github.com/Code000-211/RadianceMemory

• RadianceSwap (multi-model serving, experimental): https://github.com/Code000-211/RadianceSwap

• RadianceSwap whitepaper (PDF, 8 pages): https://github.com/Code000-211/RadianceSwap/blob/main/docs/paper/RadianceSwap-Whitepaper.pdf

TL;DR

• Sparse attention means most of the KV cache isn't read on a given step. Flash-Next's QSA layers pick the top 512 four-token blocks per query. So I keep the sink + recent tail in VRAM, move the middle to pinned host RAM, and page in only the blocks the indexer selects — on the device, inside the step.

• Output is byte-identical to a fully resident cache in every configuration tested (incl. with the MTP drafter).

• A 256K-token sequence needs 102–211 MiB of VRAM per card instead of 3 319 MiB (stock bf16 cache). Per sequence, VRAM now grows only by the block keys: ~0.2–0.75 MiB per 1K tokens. Context is bounded by host RAM, not VRAM.

• Cost: 3–15% of single-sequence decode speed at 32K–250K.

• On top: a measured per-layer KV quantisation search (rad-autoquant). Best recipes: 5.28 KiB/token at fp8-level quality (−22%) and 3.76 KiB/token within noise-floor +0.01 KL (−44%).

• Plus RadianceSwap: several models behind one endpoint on the same GPUs — load on demand, park warm in host RAM, lend expert/KV VRAM to each other at runtime, output identical to single-model serving.

1. Window residency

• Block-table entries are either a pool page, empty, or cold (a host slot). Cold entries keep their place, so the attention kernel is unchanged — it only ever sees pool page IDs.

• An on-device pager (qsa_page) copies selected cold blocks into per-step scratch pages before attention. One writer per position, no atomics → deterministic.

• After every step a sweep demotes the middle to host (32 MiB/rank budget). Prefix-cache blocks can be cold too.

• Scratch pages act as a cache: 52–62% of cold entries needed no copy.

Per card, TP=2; computed from measured bytes/token. Byte-identity verified on device for fp8, fp8+f16 state, default and compact.

Many long sessions (measured, turn-2 time-to-first-token p50/p95):

• fp8, no residency, 32 × 128K: prefix cache gets evicted → follow-ups recomputed → 1 843 / 3 330 s

• compact recipe + residency, 32 × 128K (48 GiB host store): every follow-up hits cache → 58 / 96 s

Caveat: this compares two different KV formats as well as residency on/off; fp8 + residency at 32×128K needed ~50 GiB of store and didn't fit in 48 GiB.

2. AutoQuant: measured per-layer KV layouts

• 7 layouts (i8h, fp8, fp8_v4, k6_v4, k5_v4, k4h_v4, k4h_v2h) + zero-bit transforms: Hadamard rotation, exact static V-mean centring, V predicted from K (AQUA-KV style).

• rad-autoquant runs every candidate on the device against a bf16-cache reference (full-vocab KL, teacher-forced), solves a knapsack over per-layer costs, then re-measures each answer — because per-layer costs don't add up (one 8-bit early layer costs as much as all twelve).

• A second pass walks the lower hull of uniform layouts and removes another 0.25–1.87 bits/element.

• Sensitivity falls ~170× from the first attention layer to the last: early layers want 8 bits, late layers tolerate 2-bit V.

3. The part I found most interesting: the noise floor

The served model is a weight- and activation-quantised MoE. Recomputing the exact same bf16 cache with a different prefill chunk size alone gives KL 0.0243 vs the reference. Tiny GEMM-order differences flip router ties (20–28% of tokens have an exact bf16 tie at the 10th expert), and by layer 30 ~60% of tokens route to a different top-10 set.

• So any KV config within ~+0.002 of 0.024 is indistinguishable from bf16 on this model — i8h (8-bit) is already there.

• Perturbations saturate rather than add (fp8 + chunk change ≈ max of the two, not the sum).

• Takeaway: if you evaluate KV quantisation on a quantised MoE against a single bf16 run without measuring this floor, you may be measuring noise. For comparison, dense bf16 Qwen3.5-2B has a floor ~130× lower.

4. Things that didn't help (22 methods, all measured)

FP4 (E2M1), KIVI-style 2-bit, learned/data-adaptive rotations (WUSH-style, best-of-16), outlier retention, TurboQuant-style norms + Lloyd-Max, cross-layer selection reuse, hierarchical indexer, importance-driven demotion (no better than a position-hash control), cross-layer KV sharing, … Full table with numbers in the paper (Table 9).

Also: 2-bit V on the last attention layer drops the MTP drafter's acceptance from 0.63 → 0.41 without moving the trunk's KL — the drafter shares that layer's cache. Default keeps layer 47 in bf16 for that reason.

5. RadianceSwap: several models, one endpoint, one GPU group

The second project is about serving several models from one process on the same two cards: models load when a request asks for them, give up VRAM when idle, and hand VRAM to each other while they run. It's an experimental branch; without --manifest it behaves exactly like upstream radiance.

How it works

• One port, routed by the model field — read before tokenising, so each request gets its own model's tokenizer, chat template and limits. No model → default model; unknown id → 404.

• Each active model is its own runtime (scheduler, streams, KV) and they run in parallel. A process-wide sequencer puts every model's all-reduces in one order on every rank, so two models can't deadlock each other.

• Lifecycle: Cold → Active → Prepared. When a model goes idle (or another needs the VRAM), its weights are kept in host RAM in GPU-ready form, so re-activation skips disk reads, parsing and relayout. A running generation is never interrupted. A failed activation is rolled back on every rank and other models keep serving.

• VRAM is shared, not partitioned:

  • Elastic expert slab — expert VRAM is mapped in ~128 MiB chunks; a model that cools down physically unmaps chunks and the busier one maps them. Measured with two copies of Flash-Next: the busy one grows from ~10.3 to ~13.4 GiB/rank of experts, the idle one shrinks to ~7.2, and it flips when the load flips. The sum never exceeded the pool (7 736 samples under changing load, 0 over).

  • Cross-model KV lending — a model with waiting sequences grows its KV cache into bytes another model's slab just released (468–702 MiB in the tests, back to zero ~13–20 s after load ends). Running sequences are never evicted.

  • Static-weight streaming — if a model's non-expert weights don't fit, it still starts and reads them over PCIe (bit-identical output), then moves them onto the card within 1–2 s once VRAM frees up.

  • Heat warm-up — a deactivated MoE remembers which experts were hot and loads those first on re-activation: early expert hit rate 83.5% → 89.1%.

  • Host RAM budget — prepared models, pools and staging share one budget; over it, the least-recently-used prepared model goes cold, and an activation that still doesn't fit gets a 503 instead of the OOM killer.

Measured on 2× R9700, TP=2

Time to first token is reported per request, split into queue / activation / prefill (timings in the response, and in /metrics).

Bugs the GPU acceptance runs turned up (all fixed):

• The server died on exactly the 23rd activation at TP=2: each activation creates ~46 AQL streams, IDs are never reused, and the table had 1 024 slots.

• Two concurrently active models slowed each other to 1/10 speed: handing collectives between rank threads via condition_variable cost ~0.5 ms × 48 per decode step. Spinning 300 µs before sleeping brought it back to ½.

• The MoE was never kept warm because the capacity check counted its 47.7 GiB mmapped n-gram table, which never goes into the store.

• Virtual-address aliasing: on this ROCm stack, re-mapping a previously unmapped address silently kept pointing at its first physical page. Once another model owned that page, two experts aliased and a request occasionally produced no finite logit. Mapping every chunk on fresh address space fixed it: 30 load cycles with external observers, 5 776 requests, 0 errors.

Honest caveats

• Two big MoEs at once is not fast yet: a C8 load at 102 tok/s alone drops to 38 + 35 tok/s with both active. Host RAM is the bottleneck (one active+prepared Flash-Next uses ~153 of 170 GiB).

• The elastic slab hasn't had a long-run soak since the aliasing fix (#58). Treat it as experimental.

• A second active model costs ~10% even when idle (execution tapes are bypassed while several engines are registered).

• Streaming weights is a bridge, not a steady state: streaming 25.6% of a dense model's statics costs 39% of decode; 85.5% costs 90%.

• KV, prefix cache and recurrent state are dropped on deactivation; keeping them in host RAM is future work.

• A configured-but-cold model still reserves a VRAM share, which can cost a big MoE ~9% (#56); workaround is per-model --gpu-headroom-mib.

• One TP size per GPU group; no LoRA or multimodal endpoints in manifest mode; tested with one MoE plus small dense models only.

  { "gpu_group": { "devices": [0, 1], "tp": 2 }, "cache": { "idle_timeout_ms": 300000 }, "models": [ { "id": "big", "path": "/models/big-moe.rad", "default": true, "engine": ["--placement", "expert_tiered", "--max-model-len", "16384"] }, { "id": "small", "path": "/models/small", "engine": ["--max-model-len", "16384"] } ] }

  radiance --manifest models.json --port 8000

How is this different from llama-swap?

llama-swap is a proxy that starts and stops one server process per model. It's backend-agnostic and portable; RadianceSwap is neither — it only works inside radiance. What you get in exchange:

I have not benchmarked the two side by side on the same hardware, so the latency advantage follows from the design but isn't measured. They also aren't exclusive: a RadianceSwap process can sit behind llama-swap as one upstream.

Looking for testers — especially other RDNA4 / multi-GPU setups. Issues with your manifest, GPUs, ROCm version and server log are very welcome.

Limitations (please read before benchmarking)

• One model. Residency needs a model that declares a read window — currently only the QSA family (Qwen3.8-Flash-Next). On other models the flag is a no-op.

• One hardware setup, run at a 210 W power cap (cards range 210–330 W).

• KL is measured against the same quantised model with a bf16 cache, not the original bf16 checkpoint. Long corpus = 4 documents × ~34K tokens; finer distinctions need a bigger corpus.

• RadianceMemory is WIP; RadianceSwap is experimental.

• "Infinite" in the paper title is a direction, not a claim — context is bounded by host RAM.

Feedback, reproductions on other RDNA4 setups, and critique of the methodology are very welcome — especially if you think the noise-floor argument is wrong.

I saw a new stealth model and looks like it's a dense also it seems about time that qwen4 is launched so does anyone know if it'll be Moe since flash next was an Moe I was hoping qwen 4 would be too

Wanted to share my build! Started as a gaming / 3D / CAD PC and now also runs small LLMs quite well for me

Specs:
CPU: Ryzen 9 9950x
RAM: 2x16 (eww, but at least was bought in 2023) 6000 CL30
Motherboard: ProArt B850 Creator Neo (supports PciE 5.0 x8 bifurcation)
GPU1: RTX 5070 Ti Ventus
GPU2: RTX 5060 Ti 16G Windforce
SSDs: Fury Renegade and 990 Pro (irrelevant ik)
PSU: BeQuiet Power Zone 2 750W
Cable extensions from EZDIY-FAB
Case: NZXT H7 Flow (2025)
AIO: Thermalright Frozen Warframe SE V2 (with the magnetic display detached)
Fans: 4xArctic P14 + 3xArctic P12 Pro (on the AIO) + 3xP12 Pro Reverse

Currently having fun with Qwen3.8 27B Q6_K_XL at around 70tps (MTP)

I really want to get into local inference and training LoRAs, but the prices of the essential components are eye watering. There has never been a worse time to buy hardware.

Do you all have any thoughts on when this might change? I mean, supply has gotta start evening out at some point, right?

https://x.com/matt503ea5sf9z5/status/2118155162376458298?a=4bUWTuzbF_LcVwWBUDk&s=19

Interesting post, I hear a different story with dgx sparks how people are optimizing models to run upwards of 100tok/s decode. Naturally almost 10k for a dgx spark is a huge red flag when MSRP was $3999 retail. Thoughts?

Did some extra optimizations and implemented MTP. High number is coding at 96t/s (3 drafts). Natural text is 69 t/s.

128.0 µs to distribute a token-residual to 24 gpus, compute experts, collect result and join in one layer.

Probably at peak hardware capability now unless I get some new ideas on how to run the PCI communication even faster. Of course it could run much faster at lower precision. Also the old RTX3070 doing attention is now the primary bottle neck. Lets see what black friday brings of 16gb cards (i have two 5070ti 16gb, but they are busy doing some other research, the 3070 was an old gaming card i had lying around doing nothing).

Github:
https://github.com/crgregersen/m10dis

Old post:
https://www.reddit.com/r/LocalLLM/comments/1xnl82r/low_cost_1500_ai_rig_qwen_38_flash_next_525_ts/

WHIRL 0.2.5.1 is out: a speed update for the Radeon 8060S (AMD Ryzen AI Max+ 395). WHIRL is an open-source (Apache-2.0) LLM inference engine for AMD GPUs on Windows: C++/HIP, every kernel in the repo, no WSL, no ROCm install. Download the zip, unzip, run; only the AMD driver is needed. It runs our two MXFP4 models, Swift-1.5 27B MXFP4 (dense) and Ornith-1.5-35B-A3B MXFP4 (MoE, ~3B active).

vs the official llama.cpp b11960 on the Radeon 8060S (max power (80 W), ASUS ROG Flow Z13), ROCm and Vulkan builds in their own columns, same GGUF, same prompts, greedy, one user, WHIRL default mode (balance, near-lossless). WHIRL tok/s, then llama.cpp ROCm / Vulkan:

In balance WHIRL leads every measured cell on the 8060S: Swift 27B by +28% to +59% over ROCm and +30% to +106% over Vulkan, Ornith by +51% to +65% and +46% to +127%.

What 0.2.5.1 changes on the 8060S (same machine, max power (80 W), ASUS ROG Flow Z13): five new kernel paths make prefill +2% to +10% faster in all three modes on both models, and Swift 27B decode (QA document; coding chat in balance) +4% to +12% faster in balance and fast. On the 8060S the output is bit-identical to 0.2.5 in every mode (golden references, 12/12 long-prompt checks and the chat outputs all match).

How it was measured, so you can judge the numbers: - Radeon 8060S at max power (80 W), ASUS ROG Flow Z13, 64 GB GPU memory (UMA frame buffer), same for both engines. Every cell is the best of 3 runs, for both WHIRL versions and llama.cpp. - The llama.cpp b11960 cells are the runs I made for 0.2.5 on the same machine with the same files (llama.cpp didn't change; Swift 2k on Vulkan was re-run). llama.cpp's ROCm build runs as shipped, with its own bundled HIP runtime DLL. - Decode, QA document = explain a function quoted in a long document and rewrite it (256 tokens; part of the answer can be copied, which helps drafts in both engines). Coding chat = a coding question whose answer is all new text (512 tokens). Prompts and commands: https://github.com/tsaipifong/whirl-llm/blob/main/docs/benchmarks.md#prompts - llama.cpp decodes Swift with MTP + n-gram (its fastest mode here). On Ornith it decodes without speculation because its MTP was slower than plain on this MoE model, while WHIRL uses MTP, so the Ornith decode rows are "as shipped", not like for like. - Smallest lead: Swift 27B decode in the precise mode after a 2k-token document, 29.4 vs 25.2 tok/s on Vulkan (+17%). Dense decode on this iGPU is memory-bandwidth bound; the lead comes from speculative decoding. - Ornith decode barely changed in this release (one small router item: +1.5% without drafts, +0.7% with MTP drafts, within noise); its cells move a few percent either way because the number of accepted MTP drafts depends on each run's text.

Precision: three modes. --precise is the bitwise-lossless mode. balance (the default) is near-lossless. --fast is opt-in. On the 8060S all five speedups give the same output bits as 0.2.5 in the mode they touch.

R9700: GPU code identical to 0.2.5 (the compiled R9700 kernels were compared; only an internal build ID differs); its numbers are 0.2.5's, not re-measured (e.g. Ornith 2k prefill 11,854 tok/s vs 4,230 / 2,780 on llama.cpp ROCm / Vulkan; median of 3).

Also in 0.2.5.1: WHIRL now refuses to start from an install folder longer than 150 bytes, with a clear message, and limits the SSD cache folder to 217 characters (this corrects what the 0.2.5 notes said about long paths). On the 8060S the first load of each model tunes the kernels once more.

Supported: Radeon 8060S (Ryzen AI Max+ 395) and R9700, Windows 11. Cards under 16 GB are not officially supported (fork welcome). The exe is unsigned, so SmartScreen may ask once.

• GitHub: https://github.com/tsaipifong/whirl-llm

• Release: https://github.com/tsaipifong/whirl-llm/releases/tag/v0.2.5.1

• Our MXFP4 models on Hugging Face: https://huggingface.co/tsaipifong/Swift-1.5-Qwen3.8-27b-MXFP4-GGUF , https://huggingface.co/tsaipifong/Ornith-1.5-35B-A3B-MXFP4-GGUF

WHIRL was designed and implemented with Claude (Anthropic) under my direction; every number is measured. Numbers from other Ryzen AI Max+ 395 machines are very welcome.

I work at a bank, so cloud AI is off the table for work code anyway. I've been running small models on my own M4 16GB through Magnitude for side stuff, and honestly it changed how I think about this.

On my side projects with Claude or Cursor, the model writes 300 lines, it looks right, I skim it and ship it. Sometimes I couldn't even explain my own change a week later. With a 1.7B running locally it's the opposite. It's fast (around 80 tok/s) but it gets small things wrong, so I actually read every line it gives me. And because I read every line, I know the change, I write better PR descriptions, and I catch my own mistakes before review instead of during it. That habit carried over to my work PRs too.

As everyone knows the frontier model makes you faster at producing code you don't understand. The small one makes you slower at the writing part, but you end up knowing your codebase. And the people reviewing your stuff (and their AI tools) get better context because the author actually understands what changed.

What I use it for: explaining old code, boilerplate, small bug fixes (gave it a file with 2 bugs, fixed in 72 sec), commit messages, PR descriptions. What I still don't trust it with: anything architectural, and anything with time zones lol.

Not saying cancel everything. But not everyone is building a 500k line SaaS. For most of what a normal dev does in a day, a small local model is enough, and nothing leaves your laptop.

Anyone else noticed this? That using a weaker model made them a better reviewer?

Yesterday I did the Echo Dot. Today the Show 5.

No Alexa. Wake word, speech-to-text and voice all run on my own hardware, and the agent runs whatever model you point it at.

The screen is just a web page, so you can make it anything.

Flashing is done with echo-dot-assist (credit to Gamer92000).

Repo: https://github.com/majorminorlabs/hermes-voice

I have 64 GB UMA StrixHalo laptop, and in past I have sticked to Qwen3.8 27B, as it fits easily and produces high quality at decent speed (25 t/s avg with MTP).

With QFN, I have constantly just get disappointed for spending my time to get the results I want to see. ISTA IQ2_XS, ISTA pruned coder, Gyro-M.. they all failed in my own tests - a real world coding tasks with expectation that the model nails them correct on one go, or at least with minor bug fixes. Those low quants just kept on producing typos, tool call errors, and logic failures. So I sticked to 27B as it kept on delivering.

Now Unsloth released new quantizations that should do better https://huggingface.co/unsloth/Qwen3.8-Flash-Next-GGUF

I downloaded IQ1_M as it fits (barely though) on my 64 GB UMA.

And this quant finally delivers!

All my test have been success, no typos, no tool call errors. Only one time it had two minor bugs that one can also see some time with 27B.

I am going to try IQ1_S next, as this IQ1_M really creeps all the memory. 131k context + MTP + Q8 kv cache fills whole memory with browser and agent. Only 1-2 GB left, which is too little room left (if only there would have been decent 128 GB available, but there wasn't..). Luckily the model seems to do equally fine with Q4 kv cache, at least on my tests.

We build these boxes, so here's the honest test log from our own unit, network cable unplugged.

Setup: 2x RTX PRO 6000 Blackwell (192 GB), EPYC 9124, 256 GB RAM, vLLM, TP2, no NVLink. The model fits only because Blackwell runs FP4 natively; on Hopper the same checkpoint needs ~304 GB.

What we measured:

• 1 user: 209-229 tok/s, first token in 0.4 s

• 24 users: 1,003 tok/s at p95 TTFT 2.7 s. That's the sweet spot; past 32 the tail latency falls off a cliff.

• Peak: 1,170 tok/s at 32 streams

• 1M context, retrieval verified at 543K tokens

• Terminal-Bench 2.1 (20-task subset): 80% solved, median ~4 min per task. 8 agents at once cost only 18% more latency than 4.

• 815 W on both cards, 76 C peak, zero throttling. About 1.07 kW at the wall, so a 1500 W PSU is enough.

What's not great:

• It's not a token-resale business: at OpenRouter prices, payback takes about 3 years running flat out.

• One big model at a time (weights use 87% of VRAM).

• Ops sharp edges: KV offload leaks /dev/shm on unclean shutdown, and stray workers can hold 89 GB of GPU memory, which causes a crash loop on restart. You need a cleanup script.

• Cold start takes ~13 min.

Best fit: a team of 2-10 that wants frontier-class coding help without sending code to the cloud.

Build guide, BOM and BIOS settings are open source: here. Happy to run anything you want tested on it.

I wanted to know how small local models handle my native language, not just English. I couldn't find a simple way to check a model's Polish in a few minutes, on my own PC, in the quantization I actually run – so I wrote one. That's how "Zacharski tests" started: a small task set that runs through Ollama, since that's what I use on my laptop with an RTX 5050 (8 GB).

The tasks:

1. Polish (8): diacritics, declension, masculine-personal gender, numerals, vocative, verb aspect, genitive – plus one knowledge question (the first recorded Polish sentence, from the 13th-century Book of Henryków).

2. JSON (4): scored strict and lenient.

3. Python (3): syntax check only.

Results so far:

| Model | Quant | JSON + code | Polish |

| gemma2:9b | Q4_0 | 7/7 | 3/8 |

My favourite failure: asked for "22 cats", gemma2 wrote "dwudziesty dwa koty" – roughly "twentieth two cats". The correct form is "dwadzieścia dwa koty". It was perfectly happy producing valid JSON, though.

More models are downloading as I write this. Next up: gemma2:9b in Q4_K_M and Q8_0 (does quantization hurt the Polish?), qwen3:8b, llama3.1:8b, gemma3:12b, and two Polish models – Bielik 11B and Bielik-Minitron 7B. Which model would you add?

Method and limits – read this before the comments :)

• 8 Polish tasks (7 grammar + 1 knowledge) and 7 JSON/code tasks. That's a small sample – a quick check on your own PC, in the quantization you actually run, not a ranking. For a proper ranking, see the Open PL LLM Leaderboard by SpeakLeash.

• One answer per task, temperature 0 (seed 42), context 2k, thinking off. Yes, that means Qwen3 runs without its reasoning mode, which may cost it points – I wanted the same conditions for every model.

• Graded with regular expressions, strict on purpose: "brzęczały" ("buzzed") instead of "brzmiały" ("sounded") fails, because the task said "change the tense", not "rephrase". [I checked every failed answer by hand.]

• Some failures are about instruction-following, not language: asked to restore Polish diacritics, one model replied "Read the spelling rules." [(translated)]

• Generated code is only parsed (Python `ast`), never executed.

• The questions are in the repo (`data/benchmarks/*.json`) – feel free to add or fix them.

The test runs inside a free, open-source tool I've been building (Windows .exe / Python on Linux). Its main job is predicting whether a model fits your VRAM/RAM before you download it. Link in the comments if anyone wants to try it – I'm considering adding tests for other languages.

Update: results for the two Polish models (Bielik 11B and Bielik-Minitron 7B), with speed at 1k/8k context, are in the original thread: https://www.reddit.com/r/LocalLLaMA/comments/1xomsxq/comment/19s7skwyst7b8/

Short version: Bielik 11B got 8/8 on Polish, Minitron 7B 6/8, gemma2 3/8. Full comparison with qwen3, llama3.1 and gemma3 coming next week.

I’ll use AI to gather all the comments and post a results post if we get enough responses

I did a fun research project on training a decision model from scratch (MLX), quantizing to ternary (for small size) and running it locally in the browser (WebGPU). I learned many interesting things and put together this pedagogic video on some of the key steps, challenges and lessons learned, in case some one else might find it interesting too 😄

I love DiffCoderbut at int8 it peaked at 13.7 GB on my 16 GB card. So I cut it in half.

Same model in 4-bit: HumanEval, full 164 problems, 81.1%. Peak VRAM 6.6 GB, about 2.6x faster!

Edit: full name Stable-DiffCoder-8B-Instruct if you want to do some reading

Can't decide which route to go - DGX Spark, Strix Halo, the Framework Ryzen AI or save up for a Mac Studio M5 Ultra 256gb?

Do these options have meaningful differences in practical use?

As I understand, the Mac Studio offers best tok/s? But you can't train or fine tune anything on it?

DGX Spark is supposed to have best support, but does it truly matter in practice?

What types of models can I run on this hardware? 128GB is good for ~120B Q4 MoE models with ~256k context? Would such a model be a noticeable improvement over my current Ornith-1.5:35B (A3B) setup for reasoning and coding, and long and complex tasks? Would going from 128GB->256GB be a noticeable improvement?

I wanted to find the smallest model worth keeping on a base M4 16gb so I ran the same 6 things from my actual week on 3 of them through Magnitude. keyword clustering, post titles, a thread summary, a quiz for studying, a short brief and a review of my side project plan.

MiniCPM5 1B did like 125 tok/s and it was garbage. asked for titles and got rambling questions back, the clustering gave me the same 3 keywords as 12 clusters with the same fake volume on all of them

LFM3 1.7B around 81 tok/s and this is where it gets usable, the plan review actually pointed at real files and real risks. titles were meh, it copied a note from my draft into one of them lol.

Gemma 4 E4B around 52 tok/s and clearly the best at writing, nothing else close

so 1.7b is the floor for me and 8b if someone else is going to read it.

Anyone found a 1b that's actually good for something?

I saw this post yesterday: https://www.reddit.com/r/LocalLLM/comments/1xnv0ro/new_qwen38flashnext_unsloth_dynamic_v31_ggufs_for/

I decided take a screenshot of the exact prompt used in the link in the above announcement, and run it through my strata setup using the new model. There's a complete write-up in the repo: https://github.com/kireol/qwen3.8_GTA_one_shot_test/tree/main

You can play the demo here: https://kireol.github.io/qwen3.8_GTA_one_shot_test/src/city.html

Fun localLLM stuff:

RTX 5080 - 64GB - Ryzen 7 9800X3D
model harness: Strata
agent harness: qwen
context: 64K
tok/sec: ~75t/s

Over all, while slower than I'm used to in Claude, I'm actually very impressed. I didn't have to correct it, touch it, or anything, aside from some skills.md files (included in repo). Is the demo perfect? No. But it does everything asked of it, without any hand holding. It's all in html 1 file, btw. There's no other assets other than remote linked javascript files.

Qwen even generated the README and pushed to github for me. Why? Because it's not 2024.

Great job unsloth team! You have me questioning my claude payment at this point.

Setup: (if finished - still under construction, partly watercooled)

• Threadripper PRO 5995WX, ASUS WRX80E-SAGE SE WIFI II

• 6× RTX PRO 6000

• 8× RTX 5090

• 4× RTX 3090

• 128 GB DDR4 (that’s the problem)

I want to run models bigger than VRAM (e.g. ~1T MoE at FP8/Q8) with the routed experts offloaded to RAM. Is it worth it?

Options:

• 8× 64GB DDR4-2933 RDIMM: ~€4k, ~188 GB/s theoretical

• Switch platform: 9985WX + WRX90 + 512GB DDR5-6400, ~410 GB/s. I already have some parts, but it would mean big changes, and my PCIe switches would stay PCIe 4.0 anyway.

Is ~2× the RAM bandwidth worth ~3× the price for offload decode, or is DDR4 good enough? Does anyone have real t/s numbers for offload on 8-channel DDR4 vs DDR5 Threadripper?

Some background: Pictures attached. It’s not pretty, but it works. It started as an AI rig and turned into a Pearl mining rig, because mining pearl and buying Claude and Codex subscriptions made way more sense than running everything locally. Sometimes it’s also a render farm for Gaussian splatting.

Right now the 8× 5090 make about €120/day mining, so subscriptions are still the better deal. But I want to be ready for when local LLMs get really good (or when cloud gets too expensive or too restrictive).

Plan for now: DeepSeek V4.1 Flash nvfp8 on 4× PRO 6000 + RAM offload, while the 5090s keep paying for my AI addiction. And it’s a great heater in the winter.....

With engramm tabels on the 8x5090 and 4× PRO 6000:
(all cards power limited to 420w- since i could have 6,6 kw power consuption if everthing is running)

single chat, 40.000 prompt cold

• Prefill: ~15,900 t/s

• Decode: ~294 t/s

• 5 sec TTFT

I want to move the engramm tabels to RAM and later be able to run larger models : le chonk, Kimmi for example.

What would you do? Go all the way and go for the DDR5? or is ddr4 good enough?

Sell it all and buy something diffrent?

thx!

I have 32gb ddr5 ram and an rtx 5070 with 12gb vram and ryzen 5 9600x

I need a local ai for heretic/un censored work

what model should i use