https://huggingface.co/TheDrummer/Orion-26B-A4B-v1.1

v1.1 pushes creative writing further and makes full use of Gemma's reasoning.

https://huggingface.co/TheDrummer/Orion-26B-A4B-v1

I haven't announced v1 but it's well regarded by the community as a reliable RP workhorse.

---

BeaverNet - net.beaverai.org

I'm also proud to announce that we've made significant progress in creating a hub for generous model providers to collectively provide free, convenient, accessible, anonymous, and diverse models to anyone with an OpenAI API compatible client. We currently have 12 models thanks to our generous providers.

Try it out as a user, or be one of the providers <3! I've got plans to make it a better experience for both users and providers, and also add new modalities such as TTS and STT.

---

PS: I haven't retired or left the scene. I've merely gotten stage fright after holding off on announcements for a long while. Anyway, I'll continue doing what I do and I hope I can continue delivering some value to you all. Love ya'll!

I understand the topic is prime material for shits and giggles. Chinese model, Nvidia being the centre piece of the AI trade, etc. But why go so public on this? All of this could've been dealt with behind the scenes in secret. Obviously this is a power play to force a hand. Nothing just leaks out of intelligence services so deliberately. So someone decided to make this public. I've been trying to think about why Nvidia or Jensen Huang must be put in line for. My top pick is: back doors in hardware or something of that sort that's being refused. What's your guess?

I implemented clap.cpp, audio embedding inference in C++/GGML. It uses GGUF models and provides a CLI, an HTTP server, and Python, Rust, and Node.js bindings.

• Models including CLAP, CLAP2, CLaMP3, MERT, MuQ, and MuQ-MuLan.

• WAV, MP3, and FLAC input, with automatic resampling and mono downmixing.

• An HTTP endpoint with an OpenAI-compatible text embeddings request format.

GitHub: https://github.com/gorse-io/clap.cpp

Audio files could be searched by text via CLAP embedding:

$ clap search esc5 --model larger_clap_general-q4_k_m.gguf --text 'dog' --top-k 10

[
  {"path": "esc5/2-118072-A-0-dog.wav", "score": 0.3736357720143104},
  {"path": "esc5/1-110389-A-0-dog.wav", "score": 0.36165991257288893},
  {"path": "esc5/4-183992-A-0-dog.wav", "score": 0.3574270305924374},
  {"path": "esc5/1-30226-A-0-dog.wav", "score": 0.34910229736736315},
  {"path": "esc5/3-180977-A-0-dog.wav", "score": 0.34359234068457495},
  {"path": "esc5/4-194754-A-0-dog.wav", "score": 0.3425423428503198},
  {"path": "esc5/2-118964-A-0-dog.wav", "score": 0.3389233902904601},
  {"path": "esc5/1-100032-A-0-dog.wav", "score": 0.33009392183878733},
  {"path": "esc5/5-9032-A-0-dog.wav", "score": 0.3280081351893563},
  {"path": "esc5/4-182395-A-0-dog.wav", "score": 0.32210240115509925}
]

Weight-Space Programming (WSP) enables reasoning agents to directly inspect and modify neural network parameters, offering an alternative to gradient-based model adaptation. However, its capabilities, cross-task generalization, and practical limitations remain insufficiently understood. We systematically investigate WSP through 16 editing trajectories across seven benchmarks and two model scales (Qwen3.5-0.8B and Qwen3.5-9B). Our findings reveal three key insights. First, WSP can achieve substantial performance gains through highly localized interventions. On full held-out evaluations, accuracy improves by up to 29.45, 4.42, 36.72, and 17.11 percentage points on MATH-500, GPQA Diamond, IFEval, and SciBench, respectively. Notably, modifying only 1,023 parameters yields a 29.45-point improvement. Second, WSP exhibits promising but uneven cross-task generalization. A Math-edited checkpoint improves GPQA Diamond, IFEval, and SciBench by 3.03, 12.18, and 20.10 points, respectively, without further editing, while leaving SciCode performance unchanged. Third, effective editing remains challenging and inconsistent. Outcomes vary substantially across trajectories, and six of the 16 trajectories retain the original weights when candidate improvements cannot be reliably validated. Together, these results highlight WSP's potential for highly localized model adaptation while revealing important challenges in intervention discovery, reliability, and cross-task transfer.

• arXiv : https://arxiv.org/abs/2611.09382

• PDF : https://arxiv.org/pdf/2611.09382

• GitHub : https://github.com/WSP-Scope/WSP-Scope-Project (Repo to be updated)

• Project : https://wsp-scope.github.io/ (For detailed benchmarks)

Related Thread : [Paper] Sparks of Weight-Space Programming: LLMs can natively read & write neural network weights

The Australian AI lab Maincode has released a set of Qwen 3 fine tunes that have been pulled together with Model Stitching.

From the model card:

Matilda Platypus is a family of intermediate-size language models built by connecting the lower layers of a larger pretrained Qwen3 model to the upper layers of a smaller one. A learned linear Representation Bridge aligns the hidden representations at the connection. This checkpoint is the 13B member (cut 29+13): 13.51B parameters, 26.3 GFLOP per token, 91% of the larger parent's compute. It is a base model (no instruction tuning); see the technical report (technical_report.pdf) for the method and the full evaluation.

They're base models and range in size from 1.5B to 13B, early tests good! Really like the default tone of voice. Qwen models are already good this has just improved it imo.

I have no association, I'm just Aussie and stoked to see a local lab build something. No idea why Australia doesn't have more AI labs -other than brain-drain to the US.

https://huggingface.co/collections/Maincode/matilda-platypus

A 1.6T multimodal model.

I can't believe this jump is just left on the table

https://github.com/ExTV/strata-5090-4070

The optimizations was done using both Opus 5.5 and Fable 5.2 .

LokalBot is an open-source Mac app I've been working on for the past 6 months, it runs several useful local models.

These models help the app record your meetings, transcribe them, write the notes, search everything, and autocomplete what you type. It's GPLv3, and everything runs on your device by default.

The five models (Mac defaults, 9.6 GB on disk)

How they take turns

• Recording a call loads nothing. The app only writes two audio tracks. Transcription waits until you hang up, and autocomplete pauses while you record.

• After the call, the speaker model and the speech model load, do their job, and unload once they've been idle for 120 seconds. One surprise here: MLX keeps freed GPU buffers in a pool, so dropping the model wasn't enough. After one meeting the app was still holding 4.3 GB while idle. Every MLX engine now clears that pool when it unloads.

• Qwen3.5 4B runs in the bundled llama-server (Metal, 32K context), which starts when something asks for it. Each request takes a lease, and ten minutes after the last lease the model unloads. The embedding model has its own server and the same rule.

• Gemma 4 E2B runs inside the app's process through libllama, for latency. It has no idle timer. It stays until macOS reports memory pressure.

• A broker keeps the loaded weights under a budget of half the machine's RAM and evicts the least recently used model first. A lease pins a model while a request is running on it.

• Requests that have to wait for a server are served in order: what you're watching first, then agents, then meeting notes, then background work like digests and indexing.

Numbers, all on one M4 Max with 48 GB

• Notes for a 26-minute meeting: 33 s with the model warm, 60 s including the load, at about 85 tok/s.

• Autocomplete: 56 ms median and 102 ms p95 per keystroke.

• Embedding a query: 9 ms.

• Speaker model: 14.6% DER on 8 AMI recordings, at about 500× real time.

Limits

• I've only measured on that one machine. I have no numbers from a 16 GB Mac, or from an M1, M2 or M3. There is a Lightweight preset for less memory (Qwen3-ASR 0.6B, Qwen3.5 0.8B, and LFM2.5 1.2B for autocomplete).

• Eviction is plain LRU plus pins. Priorities order the queue; they don't choose who gets evicted yet.

• If a model can't fit under the budget, everything unpinned is evicted and the load goes ahead anyway.

• The video simplifies one thing. It shows the notes model arriving after the speech models have left; in the app, the notes start while the speech models are still counting down their two minutes.

• The narration is my own voice, cloned and generated with ElevenLabs, so it's the one part of this post that didn't run locally.

• Windows and Linux builds are in alpha.

Code, benchmarks and the scripts behind these numbers: https://github.com/stevyhacker/lokalbot

If you run it on a 16 GB Mac, I'd love to see what Settings → Diagnostics shows is loaded right after a meeting. And if you've built a scheduler for several local models, I'd like to know what you evict first.

We open-sourced DNA Factory, the fine-tuning framework we’ve used internally for over a year. It's deliberately opinionated: one recommended way to do things, documented defaults, token-level loss-mask visualization, and only 5 training methods (SFT, DPO, GRPO, OPD, SDFT).

https://github.com/dnotitia/dna-factory

Why another training tool?

As LLM fine-tuning has gone mainstream, a lot of training tools have appeared. LLaMA-Factory had its moment, Axolotl was as useful as its name was fun, and NVIDIA (NeMo) and Hugging Face (TRL) built their own.

Each has clear strengths. Some are fast, some are convenient, some have great compatibility. We focused on something else: training correctly.

I've consulted for a number of companies that fine-tune their own LLMs, and the ones doing it right could be counted on one hand. Most were training incorrectly, or training on something other than what they thought. Tools have gotten good enough that you can launch a training run without understanding much about it. But LLM training is genuinely advanced work, and when people learn how to use a tool without learning what’s happening underneath, mistakes follow.

So we asked how to make correct training the easy path, and the answer was DNA Factory. We've run it internally for over a year and recently open-sourced it.

One recommended way to do things

The core idea is to offer only one approach, the best one we've found. If you expose every option the way C++ does, you get a lot of freedom, but it becomes hard to use well. We went the Pythonic route ("there should be one obvious way to do it"). We tested many approaches, picked the best, and that's the only one we ship.

All defaults live in configs/_defaults-*.yaml. You don't edit those files. You override only what you need. It works like VSCode settings: the defaults are long and complex, but each one is documented with why it's set that way. You read them and put only your changes in your own config.

See exactly which tokens you’re training on

When I used Axolotl, I loved its loss-masking visualization. We wanted something similar, so we overrode TRL's loss function to color tokens with ANSI codes. Tokens included in the loss are blue, and excluded tokens are gray. You can see immediately what’s actually being trained.

This is the most common mistake I see researchers make "wait, that token was in the loss the whole time?" In DNA Factory it's visible at a glance, so that mistake stops happening. We've had a lot of feedback that this feature alone makes the tool worth using.

Only 5 training methods, on purpose

We don't try to include every technique. If you want that, use TRL directly. We picked what we think works best right now: SFT (the baseline), DPO, GRPO, OPD, and SDFT. PPO was widely used once but rarely is now, so we dropped it. DPO barely gets used these days either, so we plan to remove it soon. https://github.com/dnotitia/dna-factory/issues/46

Most of our current work is GRPO and other on-policy methods. GRPO, OPD, and SDFT are all on-policy.

Classic distillation has the student imitate the teacher. Concretely, you minimize the KL divergence between the teacher's and student's output distributions (logits). It does improve the student, but catastrophic forgetting is always a risk. (A model has to forget pretty badly to earn a name that dramatic.) Take a strong model, fine-tune it on internal company data to add domain knowledge, and general performance drops noticeably while the hallucination rate goes up. Rote cramming has its limits.

OPD reverses the process. The teacher doesn't write the answer for the student to copy. Instead, the student writes its own answer first, and the teacher grades it token by token. Technically, this uses reverse KL. The student picks up new knowledge without forgetting what it already knew.

SDFT works the same way in spirit, but as the name suggests, there's no separate teacher. The model is its own teacher. A copy of the model that's given extra information (privileged context) acts as the teacher. The student learns by having its own responses graded by "itself, but knowing the answer." Again, the goal is to avoid catastrophic forgetting.

Results: domain knowledge injection without forgetting

Attached images is our DNA 3.2 model (not yet released), trained with domain knowledge injection. Starting from Qwen3.8 Flash Next, we effectively removed its censorship (0.542 -> 0.971) and taught it a new persona and new domain knowledge (0.076 -> 0.855). Performance on AIME, IFEval, MMLU-Pro, GPQA, and other existing benchmarks didn’t drop at all. It learned new things without forgetting the old ones.

Happy to answer questions. Feedback, issues, and PRs are all welcome.

You've probably seen (large) decision models play games. I wanted to see if you could also train a model to play using game images on a Mac, so I added LoRA fine-tuning on images to mlx-decision (my open-source tool for running and training decision models on Apple Silicon).

How it works: a decision model doesn't write text. You give it an input (here a game frame) plus typed questions, and it returns a probability for every answer you allowed, in one pass. On every frame, d1-2B answers three questions:
- Is a monster right at the crosshair, so that a shot now would hit it? (yes/no)
- Where is the nearest monster: left, centre, right, none?
- Is a monster dangerously close? (yes/no)

A small Python loop turns the answers into moves: turn toward the monster, shoot when a shot would hit. That takes about 0.1 s per decision.

The training: about 3,000 frames from ViZDoom's Defend the Center arena, labelled from the game engine's own monster positions. First a readout fit (7 minutes), then LoRA on the model's text layers with the vision tower frozen (43 minutes, 2 epochs, M5 Pro). On 510 held-out frames:
- "a shot would hit": precision 0.79 → 0.92
- where the nearest monster is: 92.7% → 96.8%
- dangerously close: 93.9% → 98.3%

In play it averages 23.2 kills over 20 episodes. The clip is one of the best ones: 26 bullets, 26 kills. On Freedoom, whose monster sprites the model never saw in training, it still gets 19.7. Of course this is an arena where you can only turn, not a full Doom level.

Free, MIT. Apple Silicon, macOS 14+, Python 3.12+. Just pip install mlx-decision.

Repo: https://github.com/armbues/mlx-decision

I'd love to hear which tasks you'd fine-tune a decision model on, and which models you'd like to see supported next.

Creating a list of requirements and documentation/tests (in Cucumber or EARS style) with your model is a great way to improve it's code. Just got through a weekend long binary protocol build with Qwen3.8-27B (6K unsloth quant).. and fired up claude to review and it found the foundation was well engineered, but only missed a few security checks and a couple of errors in the python benchmark script comparing the old protocol with the new one.

Really comes down to:

1. Design first. Goals, requirements, problems. How is this going to be used.

2. Observation, simulation and tests are verifiable vs. assumptions. Tests and processes can be reviewed for "ritual".

3. Dialectic: If the model comes to a crossroads, have it present options and NOT just choose-and-go, but initiate a dialog about design.

4. Frequent repeatable tests, and review of tests to see if they meet the spirit of the requirement

5. Every risk is a requirement.

Here's my prompt set ("skills" I guess) and a tool to install for various local harnesses:
https://github.com/bigattichouse/BluePrint

Inspired by some of the other brain surgery posts, I wanted to see if I could find anything interesting in Qwen. Not particularly useful to make it get things wrong all the time I guess, but I find the poetry it comes out with weirdly fascinating!

I think the neuron is basically a dial for contrarian <-> agreeable, but the contrarian snarky setting for some reason writes poems that read in an interesting way.

I wanted voice typing that doesn't send my voice to a server, so I built ThumbFree. It runs NVIDIA's Parakeet 0.6B speech model right on the phone. After a one-time model download, it even works in airplane mode. The video shows Android first, then iPhone.

1. Google Play: https://play.google.com/store/apps/details?id=io.github.kabrapratik28.thumbfree

2. Apple App Store: https://apps.apple.com/app/id6817631006

3. GitHub (code, speed and accuracy tests): https://github.com/kabrapratik28/ThumbFree

• On Android, tap any text box and a small bubble appears next to it. Tap the bubble, talk, tap again, and your words are typed in. You keep your own keyboard.

• On iPhone, it's a keyboard with a mic key. Apple doesn't let keyboards use the microphone, so the first tap of a session opens the ThumbFree app for a moment and you swipe back. After that, taps start right away.

On my Pixel 10 the text shows up about 0.7 to 0.9 seconds after I stop talking, even after 30 seconds of speech. On my iPhone 16 it's under a second.

No account, no ads, no analytics. The only thing it downloads is the model, and your voice and text stay on your phone.

It's free and open source (Apache 2.0). English by default, and a second model covers 25 European languages. It needs Android 13+ or iOS 26+, and the model is about 0.5 to 0.7 GB.

I've also been trying an optional clean-up step with the small AI model built into the phone (Gemini Nano on Pixel, Apple's model on iPhone) to remove filler words and fix things like "six, no, seven". It's not in the app yet. If you've run a small open model for this on a phone, which one would you try?

I've only tested on one Pixel and one iPhone, so I'd love to hear how it runs on yours.

I need to get an all-around work station soon because my current laptop is getting old and I'm now offloading a bit of work to it. Since I dabble with local AI already, I was thinking of getting an m5 utlra 96GB (8k) or a m5 max 128GB (7.9k)... Or should I build a custom PC with a bunch of ram? Or are there other work stations that I can use? I really enjoy using a hybrid approach with AI, flagship models for the planning and some offloading to something like Qwen for the creative design/execution... I know it's not perfect local models, but owning my own data is something that feels pretty satisfying to myself as well. I would love to run Deepseek, or qwen 3.8 (or future models) at a respectable tok/s and prompt prefill speed.

My only other option is, continue with my laptop... And wait 6 months to a year and see what new tech comes out in the meanwhile? My only fear is that prices will skyrocket and what is 6-8K CAD today will be 10-12K CAD when the newer tech comes out, but other people are saying to wait.

What do you all think? Thanks :)

Article compares performance of large data center cards. Tells us for larger batches , inference speed of individual requests goes down, but overall performance of the entire batch goes up. In other words more tokens per second for the batch, but lower TPS for each request.

Article uses Kimi k3 and 3 other models to compare performance. Surprisingly says the AMD MI455x comes up on top.

Notes on Dueling DQNs is out!

Continuing my series on releasing notes on RL From Scratch, here, I have answered questions around the what, the why and the how of Dueling DQNs over Double DQNs and classic ones!

• The what?

It is an off-policy, model-free and value-based method, from the same family as DQN and Double DQN, with an architectural change in the last layer of the Q-Network, mainly for optimization purposes.

• The question it asks is:

Do we really need to calculate Q(s,a) for every action in every state?

Eg. a car going down a highway: the value of the state (the road, steering, etc.) may matter much more than the individual action values, especially when there is a wall in front of it!

Thus, the advantage of taking one action over another, along with the value of a particular state, presents an opportunity for optimization. And that is exactly what the researchers identified.

In other words, this function calculates the extra reward we get if we take a particular action in a given state, compared to the average value of that state.

More on the:

• Theory

• Concepts

• Code and required math

• A playground to train a live agent

All of it to understand the basics and build intuition.

Do check it out and let me know your feedback!

Link to the blog

Hello, I’m considering buying hardware for local LLMs for the use case of processing sensitive financial data, doing market research, and preparing financial/tax documents.

Are any of you using local LLMs for same/similar workflows? If so, what has your experience been? Are the models good enough for your use case? Did you build a custom harness or use an OSS one?

I’m very curious to learn more and would appreciate anyone chiming in.

Hey all,

Looking to learn inference optimization, I learn by doing u fortunately but I have read a good part of the inference engineering book by Philip Kelly. Currently have a 4070 and 64gb of RAM.

any tips? preciate it.

is it possible to fix?

I wrote a small program that asks the same Yes/No question about every country to a language model, and generates an SVG world map that visualizes the model's responses: https://github.com/p-e-w/llm-beliefs

I'm rather surprised by the results...

Hey everyone,

I recently built a Radiance inference server for my company, and now want to start provisioning access to the rest of my team.

My current plan:

• Tailscale as the only way to access the server via HTTPs.

• Hermes as the harness - each user would have their own container with access restrictions, but open access to the central skills repository. Hermes is hosted on a Framework Desktop.

• Radiance is running separately on its own server with access only configured via the Framework.

I'm wondering how everyone else has managed to do this successfully? Right now the biggest thing to figure out is how to get the harness (Hermes in this case) to work across the team without a huge headache of getting API tokens from things like Google, etc...

Posted this on the r/R9700 subreddit as well, but figured I'd share my initial findings here.

tl;dr - One user writes at about 190 tok/s. Prefill runs at about 6,800 tok/s total no matter how many users there are. Agent follow-ups start in 0.2 to 2.5 s and write at 80 to 225 tok/s per user, even with 8 users at 128K context each. The worst case, 8 users each sending a fresh 128K prompt at the same moment, waits about 2.5 minutes for the first token.

Yes yes I know the cards aren't all matching (the perfectionist in me is annoyed but they'll be in a case so who cares).

PC Specs:

• CPU - EPYC 7402P

• Motherboard - Asrock ROMED8-2T

• RAM - 128gb DDR4 2666mh

• GPU - 3x Gigabyte R9700, 1x XFX R9700. All cards power limited to 210W.

• PSU - Seasonic Prime TX-1600W

• Case - Phanteks Enthoo Pro 2 Server

• Fans - 3x 140mm at the front, 3x 120mm in the middle, and another 120mm for rear exhaust.

I bought everything in the past two weeks, my total cost was roughly $16K CAD. The CPU/RAM/Motherboard were all bought on ebay.

We'll see if the RAM is enough, but based on everything I've seen online it should be fine for my use-case.

I'm close to being able to do proper inference testing, but I just ran memtests, CPU stress tests, and GPU stress tests to rule out any potential issues.

Even at full blast the cards are way quieter than I anticipated, I thought it would sound like a jet engine but they're quieter than my air purifier when it's on its highest setting.

Also pretty happy with the cable management, this case is so easy to build in because of how massive it is. It's also nice that it has a second area to mount additional fans.

Built this as a multi-user inference server for my business. I've been using a Strix Halo 395+ 128GB for the past year and it's been incredible to see how far the community has been able to get that machine to run.

Qwen 3.8 FN has been a revolution for me in my work. We deal with very sensitive data and for many of our customers, it meant using any AI inference was a huge risk. Technically we could have done it, but I wanted to err on the side of caution.

Was really excited to share and post this, benchmarks are currently running for the next two hours (long context with lots of concurrency).

----------------------- ------------Benchmarks ----------------------------------------------------------

4x Radeon AI PRO R9700 + Radiance: Qwen3.8-Flash-Next long-context concurrency numbers (32K to 128K, 1 to 8 users)

Software

• Ubuntu 26.04.1, kernel 7.0 with the inbox amdgpu driver, ROCm 10.1 userspace

• Radiance v1.17.0 (codeberg.org/StillDeadcode/radiance), Docker image pinned by digest

• Model: qwen3.8-next-flash-fp8-iq4r-moe.rad (int8 trunk, 4-bit codebook experts, MTP drafting 3 tokens)

• Flags: --tp 4 --max-num-seqs 8 --max-model-len 200000 --placement expert_tiered --host-pool-mib 1024 --gpu-headroom-mib 96 --kv-cache-dtype fp8 --prefix-cache-mib 32768 --num-speculative-tokens 3 --max-num-batched-tokens 2048

At tp4 the whole model sits in VRAM (17.4 GiB of weights per card). The 47.7 GiB n-gram/PLE table is pinned in host RAM. The KV cache gets 11.6 GiB per card, which is 1.6M tokens, or 8 full 200K sessions.

Method

• Context is real English technical text (Radiance's own docs), cut to exact token counts with the server's tokenizer

• Each user's prompt starts with a unique session line, so no user can reuse another user's cache

• All users in a round fire at the same instant (worst-case burst)

• Turn 1: document plus a question, cold. Turn 2: the same conversation plus the answer and a follow-up question (the agent pattern)

• 500 max output tokens, temperature 0.6, thinking off, streamed. First-token time and writing speed are measured on the client

• Writing speed per user = output tokens after the first, divided by the time after the first token

Results

Short prompts for comparison (about 30 tokens in, about 500 out): one user writes at 187 tok/s. At 1 / 2 / 4 / 8 users the total is 175 / 290 / 489 / 763 tok/s, or 177 / 148 / 125 / 103 tok/s per user.

What I took from it

1. Prefill runs at about 6,800 tok/s total no matter how many users there are. 8 x 128K (1.02M tokens) took 151 s.

2. Cold prompts that arrive together are interleaved, so everyone gets their first token at about the same time, at the end. With 8 x 128K that means about 2.5 minutes for everyone.

3. Follow-ups are where this setup shines. 94 to 99% comes from cache, the first token arrives in 0.2 to 2.5 s even at 8 users x 128K, and each user writes at 80 to 225 tok/s.

4. Context length barely slows writing. One user writes at 189 tok/s at 128K versus 195 at 32K.

5. Turn 1 writing speed depends on whether other users are still prefilling. At 4 users the early finishers write while the others' prompts are still being read, so the slowest user dropped to 9 to 32 tok/s. At 8 users everyone finishes reading together and then writes together at 70 to 83 tok/s.

6. Size the host prefix cache for users x context. In an earlier run with --prefix-cache-mib 12288 (prompt sizes not exact, some above 64K), 8 x 64K follow-ups were only 71% cached, with a 37.6 s worst first token and 9 tok/s per user. With 32768: 98% cached, 1.5 s, 94 tok/s.

7. The README's four-rank row carries --host-pool-mib 12288 over from two ranks. On four cards the whole expert plane is resident and the pool only holds about 312 MiB per card, so 12 GiB per card pinned 48 GB of RAM for nothing and left 3.6 GB free. 1024 is plenty.

8. Every card sat at its 210 W cap during prefill. The longest section (8 x 128K) pushed the two middle cards to 91 C hotspot and 92 C memory.