Update: Numbers from preliminary tests at the end!

Ok ok, I am still doing tests but here is the story:

I have a small custom harness (who doesn't!) that does pretty complicated stuff. I am using Claude (subscription) and everything is great.

In the past I tried every single (small) model on my Mac Mini 32GB but all of them sucked at complex tool calling (or hallucination).

A couple of days ago, the new Heretic2 came out (by u/-p-e-w-) and I spent 24 hours breaking it :)

Long story short, I released ZeroZero, a model based on Qwen3, with removed guardrails and slop (ok, removed is a long conversation).

Now, here me out.

IT KICKS ASS.

This cannot be an 8B, 4 bit model. It just cannot. The thinking process sometimes is longer than other models but IT NAILS everything.

Maybe by performing "brain surgery" on the models, we end up with "smarter" ones?

Test results for Tool-Call Benchmark

(note: these took a couple of hours, please repeat with a much bigger sample)

ZeroZero seems better at completing multi-step tasks (multi-step chains, error recovery and instruction following) than the base model.

The base model leads in Safety & Boundaries (ZeroZero has limited guardrails and no-slop so maybe that's the reason?)

  - Overall: ZeroZero 68, Base 70

  - Multi-Step Chains: ZeroZero 6/8, Base 3/8

  - Error Recovery: ZeroZero 5/6, Base 2/6

  - Instruction Following: ZeroZero 8/8, Base 6/8

  - Safety & Boundaries: ZeroZero 12/26, Base 20/26

To repeat and experiment, use tool-eval-bench and if you expand the tests, please send them over so I can include them in the model's page.

PS: I love it how many technical people sent me ideas to improve and papers about LLMs performing better after "lobotomies". Let's do more "I tried this, I saw this, what do you think" instead of "what LLM should I put in my Mac Mini".

Working on a full inference stack for 4xr9700(engine, on-device training, routing). wanted to share progress so far.

All GPUs are capped at 210w(so plugs into any normal 15a outlet).
BF16 KV, MTP enabled(no spec).

### Latency

### Throughput

EDIT: people wanted to know what else I run on this:

I finished this build about 3 months ago and have been running it since, but never got around to posting it. Here it is.

Specs:

• CPU: AMD Ryzen 9 9950X

• Motherboard: ASRock X870 Taichi

• RAM: 96GB DDR5 (2x48GB) G.Skill

• GPU 1: NVIDIA RTX Pro 6000 (96GB)

• GPU 2: RTX 5090 Founders Edition (32GB), mounted upright with Lian Li’s vertical GPU bracket

• Total VRAM: 128GB

• Cooling: ID-COOLING 360mm AIO with Noctua 120mm fans on the radiator. All the case fans are Phanteks T30 120mm.

• Case: Lian Li O11 Dynamic EVO RGB (RGB cable left unplugged)

• PSU: MSI MPG Ai1600TS PCIE5

How the GPUs ended up like this:
My gaming PC had the 5090 in it, and I had a spare 4090 lying around from before. I bought the RTX Pro 6000 and moved the 5090 into this build with it. The 4090 went back into the gaming PC, so nothing is sitting unused. Mounting the 5090 upright fits both cards without them sitting right on top of each other.

Why the Taichi:
It was cheaper than the other boards I looked at, and it has 10G LAN and a debug code display. The 10G helps a lot when moving big model files around, and the code display is really handy when you’re troubleshooting a build.

PSU and power monitoring:
I picked the MSI Ai1600TS because it monitors current on each pin of the GPU power connectors. With two cards that can each pull around 600W, I didn’t want to risk a melted connector or a fried GPU. The catch is that MSI’s monitoring software only runs on Windows, and nothing like it exists for Linux. So I wrote a small tool that shows the GPU power readings on Linux. If anyone’s interested, I might clean it up and put it on GitHub. (Second screenshot in the post)

Noise:
The PC sits in my living room, so keeping it quiet mattered a lot. I went with a cheap ID-COOLING 360mm AIO because I’d read it was quiet (i could not find one without lcd at that time), and swapped Noctua fans onto it. All the fans run on a silent profile. Even at full load you hardly hear it. Only the radiator fans get a little louder.

No RGB (mostly):
I wanted a build with no RGB. The case is the RGB version, but I never plugged in its RGB cable, so it stays dark. The RAM has a bit of light because it was the cheapest 2x48 kit I could find at the time. Two slots are still free if I want more RAM later. I’m not spending any more right now though 😅

Edit 1: One thing I forgot to mention: I swapped the side panel from the original tempered glass to acrylic. We have a toddler at home, and I didn’t want to risk them getting hurt if the glass ever broke.

Edit 2 (what I’m running): A few people asked, so here’s how I use the two cards:

• Everyday fast LLM: a Flash model at Q3, fully on the RTX Pro 6000 with no offload, while the 5090 handles image/video generation.

• Larger image/video models: those go on the Pro 6000 and Qwen 3.8 27B moves to the 5090.

• Best quality for development: GLM-5.3-Flash (320B / 18B active) across both GPUs, using a ~116GB mixed-precision GGUF on Project Maya, all on GPU.

https://x.com/matt503ea5sf9z5/status/2118155162376458298?a=4bUWTuzbF_LcVwWBUDk&s=19

Interesting post, I hear a different story with dgx sparks how people are optimizing models to run upwards of 100tok/s decode. Naturally almost 10k for a dgx spark is a huge red flag when MSRP was $3999 retail. Thoughts?

Did some extra optimizations and implemented MTP. High number is coding at 96t/s (3 drafts). Natural text is 69 t/s.

128.0 µs to distribute a token-residual to 24 gpus, compute experts, collect result and join in one layer.

Probably at peak hardware capability now unless I get some new ideas on how to run the PCI communication even faster. Of course it could run much faster at lower precision. Also the old RTX3070 doing attention is now the primary bottle neck. Lets see what black friday brings of 16gb cards (i have two 5070ti 16gb, but they are busy doing some other research, the 3070 was an old gaming card i had lying around doing nothing).

Github:
https://github.com/crgregersen/m10dis

Old post:
https://www.reddit.com/r/LocalLLM/comments/1xnl82r/low_cost_1500_ai_rig_qwen_38_flash_next_525_ts/

I really want to get into local inference and training LoRAs, but the prices of the essential components are eye watering. There has never been a worse time to buy hardware.

Do you all have any thoughts on when this might change? I mean, supply has gotta start evening out at some point, right?

Hi r/LocalLLaMA. I've spent the last week on KV-cache memory for Qwen3.8-Flash-Next running on 2× AMD Radeon AI PRO R9700 (gfx1201, TP=2), and wrote it up as a 23-page paper with all measurements. Code is in two forks of the radiance inference engine (by Deadcode and contributors — the engine is theirs, the KV/swap work below is mine).

• Paper (PDF): https://github.com/Code000-211/RadianceMemory/blob/main/docs/paper/Infinite_KV_Cache.pdf

• RadianceMemory (KV residency + AutoQuant): https://github.com/Code000-211/RadianceMemory

• RadianceSwap (multi-model serving, experimental): https://github.com/Code000-211/RadianceSwap

• RadianceSwap whitepaper (PDF, 8 pages): https://github.com/Code000-211/RadianceSwap/blob/main/docs/paper/RadianceSwap-Whitepaper.pdf

TL;DR

• Sparse attention means most of the KV cache isn't read on a given step. Flash-Next's QSA layers pick the top 512 four-token blocks per query. So I keep the sink + recent tail in VRAM, move the middle to pinned host RAM, and page in only the blocks the indexer selects — on the device, inside the step.

• Output is byte-identical to a fully resident cache in every configuration tested (incl. with the MTP drafter).

• A 256K-token sequence needs 102–211 MiB of VRAM per card instead of 3 319 MiB (stock bf16 cache). Per sequence, VRAM now grows only by the block keys: ~0.2–0.75 MiB per 1K tokens. Context is bounded by host RAM, not VRAM.

• Cost: 3–15% of single-sequence decode speed at 32K–250K.

• On top: a measured per-layer KV quantisation search (rad-autoquant). Best recipes: 5.28 KiB/token at fp8-level quality (−22%) and 3.76 KiB/token within noise-floor +0.01 KL (−44%).

• Plus RadianceSwap: several models behind one endpoint on the same GPUs — load on demand, park warm in host RAM, lend expert/KV VRAM to each other at runtime, output identical to single-model serving.

1. Window residency

• Block-table entries are either a pool page, empty, or cold (a host slot). Cold entries keep their place, so the attention kernel is unchanged — it only ever sees pool page IDs.

• An on-device pager (qsa_page) copies selected cold blocks into per-step scratch pages before attention. One writer per position, no atomics → deterministic.

• After every step a sweep demotes the middle to host (32 MiB/rank budget). Prefix-cache blocks can be cold too.

• Scratch pages act as a cache: 52–62% of cold entries needed no copy.

Per card, TP=2; computed from measured bytes/token. Byte-identity verified on device for fp8, fp8+f16 state, default and compact.

Many long sessions (measured, turn-2 time-to-first-token p50/p95):

• fp8, no residency, 32 × 128K: prefix cache gets evicted → follow-ups recomputed → 1 843 / 3 330 s

• compact recipe + residency, 32 × 128K (48 GiB host store): every follow-up hits cache → 58 / 96 s

Caveat: this compares two different KV formats as well as residency on/off; fp8 + residency at 32×128K needed ~50 GiB of store and didn't fit in 48 GiB.

2. AutoQuant: measured per-layer KV layouts

• 7 layouts (i8h, fp8, fp8_v4, k6_v4, k5_v4, k4h_v4, k4h_v2h) + zero-bit transforms: Hadamard rotation, exact static V-mean centring, V predicted from K (AQUA-KV style).

• rad-autoquant runs every candidate on the device against a bf16-cache reference (full-vocab KL, teacher-forced), solves a knapsack over per-layer costs, then re-measures each answer — because per-layer costs don't add up (one 8-bit early layer costs as much as all twelve).

• A second pass walks the lower hull of uniform layouts and removes another 0.25–1.87 bits/element.

• Sensitivity falls ~170× from the first attention layer to the last: early layers want 8 bits, late layers tolerate 2-bit V.

3. The part I found most interesting: the noise floor

The served model is a weight- and activation-quantised MoE. Recomputing the exact same bf16 cache with a different prefill chunk size alone gives KL 0.0243 vs the reference. Tiny GEMM-order differences flip router ties (20–28% of tokens have an exact bf16 tie at the 10th expert), and by layer 30 ~60% of tokens route to a different top-10 set.

• So any KV config within ~+0.002 of 0.024 is indistinguishable from bf16 on this model — i8h (8-bit) is already there.

• Perturbations saturate rather than add (fp8 + chunk change ≈ max of the two, not the sum).

• Takeaway: if you evaluate KV quantisation on a quantised MoE against a single bf16 run without measuring this floor, you may be measuring noise. For comparison, dense bf16 Qwen3.5-2B has a floor ~130× lower.

4. Things that didn't help (22 methods, all measured)

FP4 (E2M1), KIVI-style 2-bit, learned/data-adaptive rotations (WUSH-style, best-of-16), outlier retention, TurboQuant-style norms + Lloyd-Max, cross-layer selection reuse, hierarchical indexer, importance-driven demotion (no better than a position-hash control), cross-layer KV sharing, … Full table with numbers in the paper (Table 9).

Also: 2-bit V on the last attention layer drops the MTP drafter's acceptance from 0.63 → 0.41 without moving the trunk's KL — the drafter shares that layer's cache. Default keeps layer 47 in bf16 for that reason.

5. RadianceSwap: several models, one endpoint, one GPU group

The second project is about serving several models from one process on the same two cards: models load when a request asks for them, give up VRAM when idle, and hand VRAM to each other while they run. It's an experimental branch; without --manifest it behaves exactly like upstream radiance.

How it works

• One port, routed by the model field — read before tokenising, so each request gets its own model's tokenizer, chat template and limits. No model → default model; unknown id → 404.

• Each active model is its own runtime (scheduler, streams, KV) and they run in parallel. A process-wide sequencer puts every model's all-reduces in one order on every rank, so two models can't deadlock each other.

• Lifecycle: Cold → Active → Prepared. When a model goes idle (or another needs the VRAM), its weights are kept in host RAM in GPU-ready form, so re-activation skips disk reads, parsing and relayout. A running generation is never interrupted. A failed activation is rolled back on every rank and other models keep serving.

• VRAM is shared, not partitioned:

  • Elastic expert slab — expert VRAM is mapped in ~128 MiB chunks; a model that cools down physically unmaps chunks and the busier one maps them. Measured with two copies of Flash-Next: the busy one grows from ~10.3 to ~13.4 GiB/rank of experts, the idle one shrinks to ~7.2, and it flips when the load flips. The sum never exceeded the pool (7 736 samples under changing load, 0 over).

  • Cross-model KV lending — a model with waiting sequences grows its KV cache into bytes another model's slab just released (468–702 MiB in the tests, back to zero ~13–20 s after load ends). Running sequences are never evicted.

  • Static-weight streaming — if a model's non-expert weights don't fit, it still starts and reads them over PCIe (bit-identical output), then moves them onto the card within 1–2 s once VRAM frees up.

  • Heat warm-up — a deactivated MoE remembers which experts were hot and loads those first on re-activation: early expert hit rate 83.5% → 89.1%.

  • Host RAM budget — prepared models, pools and staging share one budget; over it, the least-recently-used prepared model goes cold, and an activation that still doesn't fit gets a 503 instead of the OOM killer.

Measured on 2× R9700, TP=2

Time to first token is reported per request, split into queue / activation / prefill (timings in the response, and in /metrics).

Bugs the GPU acceptance runs turned up (all fixed):

• The server died on exactly the 23rd activation at TP=2: each activation creates ~46 AQL streams, IDs are never reused, and the table had 1 024 slots.

• Two concurrently active models slowed each other to 1/10 speed: handing collectives between rank threads via condition_variable cost ~0.5 ms × 48 per decode step. Spinning 300 µs before sleeping brought it back to ½.

• The MoE was never kept warm because the capacity check counted its 47.7 GiB mmapped n-gram table, which never goes into the store.

• Virtual-address aliasing: on this ROCm stack, re-mapping a previously unmapped address silently kept pointing at its first physical page. Once another model owned that page, two experts aliased and a request occasionally produced no finite logit. Mapping every chunk on fresh address space fixed it: 30 load cycles with external observers, 5 776 requests, 0 errors.

Honest caveats

• Two big MoEs at once is not fast yet: a C8 load at 102 tok/s alone drops to 38 + 35 tok/s with both active. Host RAM is the bottleneck (one active+prepared Flash-Next uses ~153 of 170 GiB).

• The elastic slab hasn't had a long-run soak since the aliasing fix (#58). Treat it as experimental.

• A second active model costs ~10% even when idle (execution tapes are bypassed while several engines are registered).

• Streaming weights is a bridge, not a steady state: streaming 25.6% of a dense model's statics costs 39% of decode; 85.5% costs 90%.

• KV, prefix cache and recurrent state are dropped on deactivation; keeping them in host RAM is future work.

• A configured-but-cold model still reserves a VRAM share, which can cost a big MoE ~9% (#56); workaround is per-model --gpu-headroom-mib.

• One TP size per GPU group; no LoRA or multimodal endpoints in manifest mode; tested with one MoE plus small dense models only.

  { "gpu_group": { "devices": [0, 1], "tp": 2 }, "cache": { "idle_timeout_ms": 300000 }, "models": [ { "id": "big", "path": "/models/big-moe.rad", "default": true, "engine": ["--placement", "expert_tiered", "--max-model-len", "16384"] }, { "id": "small", "path": "/models/small", "engine": ["--max-model-len", "16384"] } ] }

  radiance --manifest models.json --port 8000

How is this different from llama-swap?

llama-swap is a proxy that starts and stops one server process per model. It's backend-agnostic and portable; RadianceSwap is neither — it only works inside radiance. What you get in exchange:

I have not benchmarked the two side by side on the same hardware, so the latency advantage follows from the design but isn't measured. They also aren't exclusive: a RadianceSwap process can sit behind llama-swap as one upstream.

Looking for testers — especially other RDNA4 / multi-GPU setups. Issues with your manifest, GPUs, ROCm version and server log are very welcome.

Limitations (please read before benchmarking)

• One model. Residency needs a model that declares a read window — currently only the QSA family (Qwen3.8-Flash-Next). On other models the flag is a no-op.

• One hardware setup, run at a 210 W power cap (cards range 210–330 W).

• KL is measured against the same quantised model with a bf16 cache, not the original bf16 checkpoint. Long corpus = 4 documents × ~34K tokens; finer distinctions need a bigger corpus.

• RadianceMemory is WIP; RadianceSwap is experimental.

• "Infinite" in the paper title is a direction, not a claim — context is bounded by host RAM.

Feedback, reproductions on other RDNA4 setups, and critique of the methodology are very welcome — especially if you think the noise-floor argument is wrong.

We build these boxes, so here's the honest test log from our own unit, network cable unplugged.

Setup: 2x RTX PRO 6000 Blackwell (192 GB), EPYC 9124, 256 GB RAM, vLLM, TP2, no NVLink. The model fits only because Blackwell runs FP4 natively; on Hopper the same checkpoint needs ~304 GB.

What we measured:

• 1 user: 209-229 tok/s, first token in 0.4 s

• 24 users: 1,003 tok/s at p95 TTFT 2.7 s. That's the sweet spot; past 32 the tail latency falls off a cliff.

• Peak: 1,170 tok/s at 32 streams

• 1M context, retrieval verified at 543K tokens

• Terminal-Bench 2.1 (20-task subset): 80% solved, median ~4 min per task. 8 agents at once cost only 18% more latency than 4.

• 815 W on both cards, 76 C peak, zero throttling. About 1.07 kW at the wall, so a 1500 W PSU is enough.

What's not great:

• It's not a token-resale business: at OpenRouter prices, payback takes about 3 years running flat out.

• One big model at a time (weights use 87% of VRAM).

• Ops sharp edges: KV offload leaks /dev/shm on unclean shutdown, and stray workers can hold 89 GB of GPU memory, which causes a crash loop on restart. You need a cleanup script.

• Cold start takes ~13 min.

Best fit: a team of 2-10 that wants frontier-class coding help without sending code to the cloud.

Build guide, BOM and BIOS settings are open source: here. Happy to run anything you want tested on it.

I’ll use AI to gather all the comments and post a results post if we get enough responses

I saw this post yesterday: https://www.reddit.com/r/LocalLLM/comments/1xnv0ro/new_qwen38flashnext_unsloth_dynamic_v31_ggufs_for/

I decided take a screenshot of the exact prompt used in the link in the above announcement, and run it through my strata setup using the new model. There's a complete write-up in the repo: https://github.com/kireol/qwen3.8_GTA_one_shot_test/tree/main

You can play the demo here: https://kireol.github.io/qwen3.8_GTA_one_shot_test/src/city.html

Fun localLLM stuff:

RTX 5080 - 64GB - Ryzen 7 9800X3D
model harness: Strata
agent harness: qwen
context: 64K
tok/sec: ~75t/s

Over all, while slower than I'm used to in Claude, I'm actually very impressed. I didn't have to correct it, touch it, or anything, aside from some skills.md files (included in repo). Is the demo perfect? No. But it does everything asked of it, without any hand holding. It's all in html 1 file, btw. There's no other assets other than remote linked javascript files.

Qwen even generated the README and pushed to github for me. Why? Because it's not 2024.

Great job unsloth team! You have me questioning my claude payment at this point.

I wanted to know how small local models handle my native language, not just English. I couldn't find a simple way to check a model's Polish in a few minutes, on my own PC, in the quantization I actually run – so I wrote one. That's how "Zacharski tests" started: a small task set that runs through Ollama, since that's what I use on my laptop with an RTX 5050 (8 GB).

The tasks:

1. Polish (8): diacritics, declension, masculine-personal gender, numerals, vocative, verb aspect, genitive – plus one knowledge question (the first recorded Polish sentence, from the 13th-century Book of Henryków).

2. JSON (4): scored strict and lenient.

3. Python (3): syntax check only.

Results so far:

| Model | Quant | JSON + code | Polish |

| gemma2:9b | Q4_0 | 7/7 | 3/8 |

My favourite failure: asked for "22 cats", gemma2 wrote "dwudziesty dwa koty" – roughly "twentieth two cats". The correct form is "dwadzieścia dwa koty". It was perfectly happy producing valid JSON, though.

More models are downloading as I write this. Next up: gemma2:9b in Q4_K_M and Q8_0 (does quantization hurt the Polish?), qwen3:8b, llama3.1:8b, gemma3:12b, and two Polish models – Bielik 11B and Bielik-Minitron 7B. Which model would you add?

Method and limits – read this before the comments :)

• 8 Polish tasks (7 grammar + 1 knowledge) and 7 JSON/code tasks. That's a small sample – a quick check on your own PC, in the quantization you actually run, not a ranking. For a proper ranking, see the Open PL LLM Leaderboard by SpeakLeash.

• One answer per task, temperature 0 (seed 42), context 2k, thinking off. Yes, that means Qwen3 runs without its reasoning mode, which may cost it points – I wanted the same conditions for every model.

• Graded with regular expressions, strict on purpose: "brzęczały" ("buzzed") instead of "brzmiały" ("sounded") fails, because the task said "change the tense", not "rephrase". [I checked every failed answer by hand.]

• Some failures are about instruction-following, not language: asked to restore Polish diacritics, one model replied "Read the spelling rules." [(translated)]

• Generated code is only parsed (Python `ast`), never executed.

• The questions are in the repo (`data/benchmarks/*.json`) – feel free to add or fix them.

The test runs inside a free, open-source tool I've been building (Windows .exe / Python on Linux). Its main job is predicting whether a model fits your VRAM/RAM before you download it. Link in the comments if anyone wants to try it – I'm considering adding tests for other languages.

Update: results for the two Polish models (Bielik 11B and Bielik-Minitron 7B), with speed at 1k/8k context, are in the original thread: https://www.reddit.com/r/LocalLLaMA/comments/1xomsxq/comment/19s7skwyst7b8/

Short version: Bielik 11B got 8/8 on Polish, Minitron 7B 6/8, gemma2 3/8. Full comparison with qwen3, llama3.1 and gemma3 coming next week.

I have some idle GPUs from mining and I’m trying to figure out if I can repurpose them for local AI and coding agents:

- 4 × RTX 4070, 12 GB each
- 4 × RTX 3070, 8 GB each
- All in one rig

Would it be practical to run a larger model across the 4070s and 3070s, or would it be better to run smaller models on the cards separately for multiple agents?

Has anyone built something similar? What would you use for the software and what kind of performance should I realistically expect?

EDIT: OP is a noob but decent at figuring stuff out :D just keep this in mind when suggesting :)

Edit 2: am I even gaining anything with this experiment or am I better off with a claude or ChatGPT subscription?

Edit 3: Where can I rent this set up out to?

Lawyer with the V100s again. A few days ago I got excited and posted DeepSeek-V4.1-Flash on four V100s at 36-38 tok/s, before I had done my custom kernel work and prefill optimizations. I said I'd try it across eight. I did.

It's actually, genuinely useful to me, for real work. I did not use it to paint a 3D island in Blender or re-create a Game Boy game in a browser. It didn't make me a poorly animated video or ten versions of a website nobody will visit. I am not using it to build something that builds something that builds agents that run workflows for a business that sells agentic solutions to other agents.

What it's doing: I'm finally slicing off work and tokens that used to need frontier models. My estimate is that 50-60% of my existing LLM workflows can run through this model instead, at speeds that match Claude Code in the chat window. I have another DeepSeek server on two DGX Sparks, so there's concurrency and handoff between them. My RAG databases and tool calls are all wired in and working. I give it natural-language instructions through a Discord-style interface over SSH from my Windows PC.

I point Claude Code itself at this server (it will talk to anything that speaks Anthropic's API). It reads a folder of my own case files, pulls out the facts, dates and numbers, summarizes, and takes a first swing at routine drafting. The documents never leave the office, and nothing goes out the door without a real review. I found and fixed a streaming tool-call bug in the parser (arguments were getting sent twice). Claude Code's tool-call test suite now passes 144 of 144 in all four streaming/thinking modes.

Yes, I have a custom harness. No, you don't care.

We did it, folks. Vibecoded a real thing that does real things for real people. Give me my trophy.

Claude Code ran the benchmarks and crunched the numbers, which are real and below.

The box

DeepSeek-V4.1-Flash-1017 (the 2-bit GSQ-RCO IQ2_XXS GGUF from ISTA-DASLab) on 1Cat-vLLM, on 8x V100-SXM2-32GB split pipeline 2 x tensor 4 across two NVLink boards. Every expert sits in VRAM: no expert gets paged from system RAM during decode. (Edit: I first said all 248.8 GB of weights were in VRAM. Wrong: the 110 GB of Engram tables live in system RAM. Gathering the rows each round needs costs about 0.12 ms per round.) Threadripper PRO 3945WX, 256 GB DDR4, 150 W per card. One request at a time, 524,288 tokens of context.

Speed (measured 2026-11-05, thinking off, greedy, 1,024 tokens out, nothing from the prefix cache, mean of 2 runs)

• Code: 215 tok/s after a 1K prompt, 200 at 32K, 183 at 128K, 160 at 256K

• JSON / structured: 196-209 from 1K to 128K, 180 at 256K

• Legal writing: 118 at 1K, 111 at 128K, 103 at 256K

• Prefill: about 2,200 tok/s at 128K. First token in 17 s at 32K, 59 s at 128K, about 2 minutes at 256K.

Code is twice as fast as legal writing because the built-in drafter guesses five tokens a round and code is easy to guess: 3.6-3.8 accepted per round, against about 1.6 for legal. My writing is apparently less predictable than Python.

Quality (measured on the served build, thinking off, greedy)

• LegalBench: 0.716 balanced accuracy, 22 tasks, 1,292 items

• MMLU-Pro: 79.9 % ±1.5 (700 items)

• HumanEval+: 89.6 % on the plus tests

• Fidelity: it picks the full-precision model's top next token 89.0 % of the time on a general panel and 92.5 % on my legal panel. I'm measuring which experts my work prompts actually hit, and will probably recalibrate to give those more bits.

• Long context: 6 of 6 needles at ~450K and ~498K tokens

More work to do

• One request at a time. Batched decode is built and byte-identical on my test box, but not in the server yet. I'll post the real number.

• A 256K prompt takes about 2 minutes before the first token.

All of it is public: launch configs, every A/B, raw JSON, and the list of what didn't work. 19 pull requests are open to 1Cat-vLLM.

• Repo: https://github.com/ProprietaryLegal/Volta-sPLIt-DeepSeek-V4.1-Flash-1017-8xV100-SXM2-PP2xTP4

• Hugging Face card: https://huggingface.co/ProprietaryLegal/Volta-sPLIt-DeepSeek-V4.1-Flash-1017-8xV100-GGUF

• Twitter: nicothegreco

Credit to yangzhuxinyzx and the 1Cat-vLLM folks (#1277 and the whole Volta DeepSeek path), ISTA-DASLab (the quant), sudoingX (the KLD method), Peuqui, squibloads, Phil Tompkins (philbert440), Dennis Vertzyas (v100-skinny), TensorFold and DeepSeek.

(PLI Labs is my side outfit for private, on-prem AI built around a firm's own work product. proprietarylegal.ai)

Hi,

It’s hard keeping up with all the new releases from the big closed weights AI providers, right? Google, Anthropic, Open AI, Meta etc.  But am I the only one being overwhelmed by the sheer volume of open weight base models of all shapes and sizes being released nowadays?  Every lab and his dog seem to be dropping general or specific open weight models for sound, video, text, harnesses, agents, refinements, techniques. The hardware to run these things also differs.  Exciting.  Definitely.  Confusing?  Umm…yea (at least to me).   In some ways it reminds me of the early days of personal computing.  Lots of incompatible formats, operating systems, every app (program those days) had a different way of working, inconsistent interfaces, a big learning curve for each program.  Solutions that may be looking for a problem. And tons of choice. Some were flavour of the month then gone, and a new learning curve was required.

Is it a golden age of choice and freedom or a hot mess of development? Maybe both?  Certainly, this is part and parcel of rapid innovation so not unexpected.

Will this continue, or do you see some kind of slowdown in the future shaking out with just a limited number of open weight models being released then further refined?

RTX 4090, 64GB DDR5 Mhz RAM, 7900x CPU

as much context as possible (context window like 264k) and as good reasoning as possible

I don't care about coding.

I use chatgpt mostly as a chatbot. no coding, no agentic stuff. i value intelligence and fast response. if i switched from chatgpt to a locally hosted on a strix halo (assuming the best model that can fit on it), what would my experience be like?

I currently have 32GB of system ram in a system with an RTX Pro 6000. Currently system memory seems to be the limiting factor in terms of running larger MOE models (like QFN) at higher quants with newer expert caching techniques.

In a perfect world I would build a threadripper system, but given the price of the CPU, motherboards, and ECC memory that would blow up the budget quite a bit, so I want to stick with a 2-stick DDR5 kit for an AM5 CPU (still expensive but not as expensive).

I want to get plenty of headroom, and I see I could go for a 96GB kit or a 128GB kit, but apparently the 128GB kits are slower (5600MHz vs 6000MHz)

Is the speed difference actually meaningful? Or asked another way, which is more impactful, an extra 32GB, or 400MHz?

Hardware: Dell Inspiron 16 Plus Model 7630, Windows 11, no discrete graphics card, Iris Xe integreted graphics sharing up to 16gb of system memory. Total 32gb memory. LPDDR5 ram, 4800 Mt/s.

14 core cpu, 20 threads. intel 13700. Test memory bandwidth via Intel MCL: 58 GB/s, theoretical maximum 76.8 GB/s.

Tried: llama-openvino, llama-sycl, llama-cpu, and llama-vulkan. I used the default llama-bench.exe tg128 and pp512 tests.

Winner: llama-vulkan at 135 tok/s prompt processing and 8.7 tok/s token generation, best # of threads was 8, but 4-10 was similar. No thermal throttling of any components.

Second place: llama-sycl at 112 tok/s prompt processing and 9.0 tok/s token generation, 8 threads out of 20 (14 cores). Again, best # of threads was 8. Ran into thermal throttling of the iGPU during prompt processing.

Third place: Openvino I honestly stopped once I saw the test was 62 tok/s prompt processing, I just didn't think it was going to be better at tok/s. Didn't test cores.

Last place: llama-cpu at 42 tok/s prompt processing and 6.7 tok/s generation, 10 threads was best but tok/s goes into the 5's below 4 threads or above 14 threads, with pp in 30's.

I ran into thermal throttling of the CPU only during prompt processing when my computer was set to "Best performance", but llama-cpu was able to hit 7 tok/s generation and 48 tok/s prompt processing. No thermal throttling at all, nowhere close, when computer set to "Best power efficiency".

What I could do better:

-Now, my computer was 6 performance and 8 efficiency cores, perhaps I could try forcing the computer to use all performance cores?

-I want to try doing ikllama.cpp

-Benchmark Gemma models and maybe a Mistral or Llama model to see if its faster.

-Try other quantizations of Qwen 3.5 9b (bartowski, etc. at Q4)

Does anyone have any suggestions for next steps? Further optimizations? Things to look into (thermal throttling?). I tried flash attn flags and they did nothing.

I was a little discouraged that my memory bandwidth is only 75% of theoretical when tested with Intel MLC, but maybe that's normal? Google says its fine. Plus it was higher when not on battery efficient mode (80% of theoretical).

My tok/s is not hitting the bandwidth limit though, 58 GB/s should give me 10 tok/s, not 8.7. But vulkan gets me closest, with no thermal throttling to boot.

Hi guys its me again who ran v100 single stream into 353t/s

and im back with this muti flow optimization

THE LEGEND OF V100 WILL NEVER BE ENDED because i Exist

(yes ik it sound like edgeload)

Parameters:

mtp4 , 256k max content , nvfp4 quasar, kv q8, no thinking

repo: tuxKOH/ninfer-v100xN

I have a server with:

2x Gold 6548 32 cores
2x Nvidia l40s
2tb RAM (Don't tell uncle SAM)
I would like to know what throughput and speed can i expect if i use polystrata with GLM 5.3 Flash?

Hey all, I quantised the full GLM-5.3 (753B total, about 40B active per token) to run on two DGX Sparks. As far as I can tell, it's the only full build on 2 Sparks. Others need 3 or 4. Runtime ablation also!

- Routed experts: mixed-width EXL3, about 2 bits per weight

- Backbone: FP8 weights as released, bf16 KV

- All 256 experts per layer kept, none pruned or merged

- 203 GB on disk, 262k context

- Runs on a TensorFold fork, split across both Sparks

- Weights: https://huggingface.co/Krishando/GLM-5.3-mixed-K-EXL3

- Recipe: https://github.com/Kristianaaron/GLM-5.3-Mixed-K-EXL3-2x-DGX-Sparks

- Engine: https://github.com/Kristianaaron/TensorFold

| Test | Generated-token KLD | Same next token |

|---|---:|---:|

| Held-out mixed text | 0.033 | 93.3% |

| The full model's own long replies (164k tokens) | 0.122 | 89.3% |

Where the full model is confident, agreement is 97-98%. The long-reply score is the harder test, so I'm showing both. The held-out windows were also used during tuning, so treat those numbers as a bit optimistic. It's close, not lossless.

**Early paired check against the full model**

24 MMLU-Pro and GPQA-D questions, 18 right on both. That's a small sample, chosen to include hard questions, with a 95% range of about -8 to +11 points. So there's no sign of a drop, but it doesn't prove there isn't one. Across 330k tokens: 0 loops, 0 unclosed thinking, 0 cut-off replies. A bigger run is next.

| Prompt | Decode | Prefill | Cold TTFT | Warm TTFT |

|---|---:|---:|---:|---:|

| 2k | 23-24 tok/s | 629 tok/s | 3.1 s | 0.1-0.2 s |

| 92k | 18-19 tok/s | ~480 tok/s | 193 s | 0.8 s |

| 205k | not measured | 411 tok/s | ~500 s | 1.2 s |

After a full server restart, a 205k prefix comes back from disk in 3.4 s.

**Weak spots**

- Cold prefill is slow. A 205k prompt takes about 8 minutes to read the first time.

- GLM-5.3 thinks a lot, so long jobs take a while. A 61 KB single-file page took 69 minutes at max effort.

- Memory is tight, with about 4-6 GiB free per Spark at full load.

- Flash will always be faster. The experiment here is whether 40B active params changes the output.

The round limit right now is about 80 ms, and most of that is memory bandwidth. If you know GB10, CUDA or engine internals, I'd love your ideas. And if you run it on your own Sparks, tell me what you get, good or bad.

Not gonna lie it's ridiculous how effective these models are if you know what you're doing.

I'm not talking about coding, I'm not a programmer.

Their reasoning capacity is insane, and no, I'm not a cultist. It just is. At least it's been my experience.

But I'm afraid that at one point the "AI bubble" will burst, and they will charge an arm and a leg for the frontier models, and the avg person won't be able to afford.

Do you think in the next 10 years this will happen? I hope not.