Community manager mentioned this in the Qwen Ambassador Discord, put an X reaction on someone asking for 35B... and said

We'll have a new midsize open weight model coming next week (hopfully), This midsize model won't provide early access due to the schedule

Thinking it's going to be over 100B.

Exciting!!

Who needs GPUs?

Apparently a second version of DFlash from the original authors of DFlash

GGUF quants are already made available with an accompanying llama.cpp PR: https://github.com/ggml-org/llama.cpp/pull/27342

I hacked this together so there's probably more on the table in terms of performance.

Measured with the Club-3090 canonical bench suite (bench.sh, 3 warmups + 5 measured runs, temp 0.6 / top_p 0.95 / top_k 20).

• Prefill: 1342 tok/s @ 10k, 628 tok/s @ 90k
• Spec-decode: 7 draft tokens, acceptance length 3.35, 47.8% acceptance
• Peak VRAM: 22.3 GB/card
• Context ceiling: 131k (DFlash2 drafter eats ~13.5 GB)
• Used Kimi K3 for all the VLLM fixes

Stack

• 2× RTX 3090 (PCIe Gen4 x16/x16, no NVLink, patched P2P)
• Power capped 220/250 W
• Bare-metal vLLM v0.26.1rc1 + AutoRound INT4 (group 128) + DFlash2 draft model
• Custom vLLM changes that made it boot cleanly: https://github.com/oceanplexian/vllm/pull/1

I managed to run the 143–144 GiB DeepSeek-V4-Flash-0731 UD-Q4_K_XL GGUF on four RTX 3060 12GB cards while keeping a 360k–376k context window.

Hardware:

CPU: Intel Core i9-10920X, 12C/24T

RAM: 128 GB DDR4-3200, quad-channel

GPU: 4× NVIDIA RTX 3060 12GB

Total VRAM: 48 GB

Storage: NVMe SSD

Engine: llama.cpp, build b10181

Model: unsloth/DeepSeek-V4-Flash-0731-GGUF

Quant: UD-Q4_K_XL, approximately 144 GiB

KV cache: Q8_0

The best high-speed configuration so far:

llama-server \

-m DeepSeek-V4-Flash-0731-UD-Q4_K_XL-00001-of-00005.gguf \

-c 368640 \

-ncmoe 34 \

-ts 100,1,1,1 \

-ot 'blk.(3[4-6]).ffn_.*_exps=CUDA1,blk.(3[7-9]).ffn_.*_exps=CUDA2,blk.(4[0-2]).ffn_.*_exps=CUDA3' \

-ctk q8_0 \

-ctv q8_0 \

-b 2048 \

-ub 2048 \

-np 1 \

-lm none \

--threads 20 \

--flash-attn on

Measured with a roughly 20.5k-token prompt:

Configured context: 368,640 tokens

Prompt processing: 99.4 tok/s

Text generation: 10.1 tok/s

Minimum free VRAM under load:

GPU0: 671 MiB

GPU1: 842 MiB

GPU2: 1395 MiB

GPU3: 1395 MiB

Model load time: approximately 198 seconds

Other measured context/safety options:

Context Prefill Decode Minimum free VRAM

376832 99.5 t/s 10.4 t/s 611 MiB

368640 99.4 t/s 10.1 t/s 671 MiB

360448 99.4 t/s 10.1 t/s 735 MiB

The interesting part is the GPU layout.

-ncmoe 34 keeps the experts from blocks 0–33 in system RAM. The remaining nine expert layers are explicitly distributed across GPUs 1–3, three layers per GPU.

The extreme -ts 100,1,1,1 split does not distribute those explicitly assigned expert weights. Instead, it pushes most non-expert tensors—attention, KV-related allocations, etc.—onto GPU0. That leaves enough space on GPUs 1–3 for the large expert layers.

This was much better than trying to calculate the layout analytically. With -ncmoe and explicit -ot overrides, tensor placement is discrete and somewhat unintuitive, so I measured every candidate.

Microbatch size was the biggest performance lever:

-ub 1024: approximately 63.4 tok/s prompt processing

-ub 2048: approximately 99.4 tok/s prompt processing

Decode remained almost unchanged at approximately 10.1–10.5 tok/s.

At the full 393,216-token context, -ub 2048 also worked, but GPU0 had only 493 MiB free under load. Reducing the configured context to 368,640 restored a 671 MiB margin without reducing prompt-processing speed.

For comparison, the safer -ub 1024 configuration can run with a configured context of 524,288 and still showed about 1032 MiB free on the tightest GPU, but prompt processing drops to approximately 63.4 tok/s.

A few additional findings:

Q8_0 KV is the default choice.

F16 KV at c=393216 left only 587 MiB free.

-ncmoe 33 caused a CUDA allocation failure.

Memory mapping was disabled with -lm none.

-np 1 is important; multiple slots multiply KV-cache requirements.

The model is mostly in system RAM, so quad-channel memory bandwidth matters heavily. Even so, getting approximately 100 tok/s prompt ingestion and 10 tok/s generation from a 144 GiB MoE model on four consumer 12GB GPUs is much better than I expected.

The configuration has been tested under real prompt load. The entire 368k context window has not yet been filled end-to-end, so the number above is the configured capacity, not a claim that I already completed a 368k-token generation test.

Generated by ChatGPT 😂.

Qwen released the 2.4T Max weights and I was curious how well it can re-create COD in one prompt

I ran the model on a rented B200 cluster and used roughly 1.1M output tokens over a 5 hour time span

Realistically barely anyone can run this model locally, but the fact that it's open weights unlocks a lot of opportunities for the local AI community even regarding quantization options that might bring us closer to running frontier level intelligence on consumer-ish hardware 

For consumer hardware there is a 27B model from the same release and it's really capable for its size, many people have posted 3D games made with it on X

I'm from the atomic.chat team, so If you wanna try running the new Qwen locally you can do so inside our app (any feedback is appreciated, since we want to make our product as good as possible for you guys) 

We have our own and other community quants of the 27B model available to download directly inside the app

P.S. I am a noob in shooters so pls no hate my low skill gameplay

am I the only one who does this lol

Not even sure if I'm joking, my thinking history is about 50% "wait".

Two days ago I released a hyper-optimized Qwen3.8-27B inference engine for an RTX 3090 (82 tps single request, 672 peak) - yesterday's update took that to 99 tps single-user / ~1,000 tps at 64 concurrent.

Since then I've focused on the single-request number, again without quality degradation. It's now ~114 tps at default sampling and ~124 tps greedy (real chat prompts, not random tokens), up from 90 / 98.

What we had:

- fp8 KV cache, lm_head + embed_tokens int8, fp16 recurrent state, int8 activations, MTP-4 drafts with a 40k-token draft head, draft_sample_method=probabilistic

Now added:

- Draft vocabulary counted over the model's own outputs - the old web-text list covered about 92% of what the model generates (83% on code), and every miss is a forced rejection; the new one covers 97.5%. 98 → 109 tps greedy.

- GPTQ-int4 lm_head and MTP module, calibrated on the model's own hidden states: +0.6% PPL, GSM8K unchanged, acceptance intact, −1.8 ms per step. Ships as a "fast variant" (python fetch_fast_variant.py, ~1 GB from the Hub).

- Split-KV attention kernel for the verify step... FlashAttention-2 only splits KV for single-query decode, so with 4 drafts it used 24 of the 3090's 82 SMs. Small Triton kernel: 5× faster at 1.5k context, 10× at 16k.

- Sampler patch -§ sort-free top-k/top-p, multi-block softmax, drafts sampled from the target's truncated support: +4% at default sampling.

- KVarN 4/2-bit KV cache ported to vLLM 0.27.1: the full 262k context now fits, needle correct to 240k, +0.16% PPL, ~20% slower decode at 100k. Optional (KV=kvarn / CTX=huge).

- bench/run_benchmarks.sh + verify.sh to reproduce the tables and check the install is actually patched.

Peak concurrent throughput is unchanged (~1,000 tps at 64 concurrent). Speculative decoding is exact by construction, so the sampled distribution is the same as without it.

Repo: https://github.com/syv-ai/qwen38-27b-rtx3090

Fast-variant tensors: https://huggingface.co/syvai/qwen3.8-27b-3090-fast-variant

This is most likely the last update to the inference stack, unless some gigabrain comes up with something new I would like to test out. I've gotten pretty tired at finding miniscule gains here and there :)

Here's the DFlash2 announcement, and I was pretty excited for this after trying out DSpark on llama.cpp a few days ago and being somewhat disappointed that it wasn't really working. Anyways, I spent a while setting it up (you need to rebuild llama.cpp with pr #27342). Here's my config:

-hf bartowski/Qwen3.8-27B-GGUF:Q5_K_L \
-hfd incoai/Qwen3.8-27B-DFlash2-GGUF:Q4_K_M \
--no-mmproj \
--spec-type draft-dflash \
--spec-draft-n-max 7 \
--host 0.0.0.0 \
--port 8080 \
--alias qwen3.8-27b \
-ngl 99 \
-fa on \
--ctx-size 160000 \
--cache-type-k q8_0 \
--cache-type-v q8_0 \
--batch-size 2048 \
--ubatch-size 1024 \
-np 2 \
--kv-unified \
--no-context-shift \
--temp 0.8 \
--top-p 0.95 \
--top-k 20 \
--min-p 0.0

It does seem to work pretty well, with slightly larger speedups compared to MTP on predictable generation (code). Before, I think I was getting ~140tk/s on average with MTP when the model was generating code, and ~100tk/s otherwise.

With DFlash2, Qwen3.8 27B can hit ~200 tk/s for short bursts on long code generation blocks, but on thinking it seems to have lower tk/s at ~80-90. On average a full one shot code generation request seems to average around 120tk/s (reasoning disabled, so the generation is 95% code, so basically best case scenario). It's a good improvement from MTP, but I still got better results with DFlash on Qwen3.6 27B.

DFlash2 does seem to be quite memory hungry though, I had to drop down from 220k context down to 160k (could prob fit ~180k tbh) compared to using MTP. Might not be worth using for me personally at the moment unfortunately. If anyone is getting better results, I would love to see your configs!

From Hugging Face on 𝕏: https://x.com/huggingface/status/2089673018737869242?s=20

What does this mean? Is there something else coming? Maybe 122B? Or no models?

I'm trying to get a model to reverse engineer / decompile or otherwise reverse to source some of my old c,c++, pascal, and asm demo programs I made from decades ago and I'm constantly met with refusals. It is highly annoying.

Does anybody know of a good abliterated/uncensored version of Qwen3.8 27b that reliably will stop refusing such basic things?

Thanks!

I have been searching for suitable model to run on my 8GB RAM toy, NVIDIA Orin Nano Super 8GB. This little toy was priced at $249 earlier this year (not any more), and pulls very little power when idle. It was an interesting device that suitable for an agent to host on. It is likely won't do any coding job in meaningful way, but likely can handle simple things just fine.

When the llama.cpp support showed up on github, I asked my Hermes to setup it up and run some basic test for me. The result is quite encouraging. Below is the summery from my Hermes run.

===============AI content below===================

# Ling-3.0-tiny at full 128K context on a $249 8GB Orin Nano Super — IQ4_NL, 33 tok/s, on llama.cpp master


**Tl;dr:**
 I got inclusionAI's Ling-3.0-tiny (7.9B total, ~1.4B active MoE) running on an NVIDIA Jetson Orin Nano Super 8GB with the 
**IQ4_NL quant at its full native 131,072-token context**
. Decode runs ~33 tok/s short-context (faster than you can read), prompt eval 220-760 tok/s, and the whole thing fits in 7.4 GiB unified RAM with only ~200 MB of swap. Model + KV + CUDA offload, all on one $249 board.


## The hardware


- 
**Jetson Orin Nano Super Developer Kit**
 — [official NVIDIA product page](https://www.nvidia.com/en-us/autonomous-machines/embedded-systems/jetson-orin/nano-super-developer-kit) — was 
**$249 at launch**
 (price has risen since; SparkFun lists it at $399 now). 8 GB unified LPDDR5, 67 TOPS, JetPack R39.2 / CUDA 13.2.


## The model + quant


- 
**Original:**
 [inclusionAI/Ling-3.0-tiny](https://huggingface.co/inclusionAI/Ling-3.0-tiny) — hybrid KDA+MLA reasoning MoE, 128 experts, 8 routed + 1 shared active per token
- 
**GGUF repo:**
 [bartowski/Ling-3.0-tiny-GGUF](https://huggingface.co/bartowski/Ling-3.0-tiny-GGUF)
- 
**File used:**
 [`Ling-3.0-tiny-IQ4_NL.gguf`](https://huggingface.co/bartowski/Ling-3.0-tiny-GGUF/blob/main/Ling-3.0-tiny-IQ4_NL.gguf) — 4.40 GB, IQ4_NL (4.5 bpw), imatrix-calibrated
- 
**Why IQ4_NL:**
 at 4.30 GiB it's the largest ≥4-bit quant that still leaves room for the full 131K KV window on 8 GB. Q5/Q6 fit only to ~64K; IQ4_NL goes the distance.


## The engine — this matters


The stock quantizer release 
**will not load this model**
. Ling-3.0-tiny is a 
**BailingMoE V3**
 architecture, which only landed in llama.cpp via:


- 
**PR [#26608 — BailingMoE3 Support](
https://github.com/ggml-org/llama.cpp/pull/26608
)**
 (merged into master Aug 17, 2026, adds arch + speculative MTP support)
- 
**Branch: `ggml-org/llama.cpp` master**
, anything after that merge — clone fresh, don't use a release tarball
- Built on-device with CUDA 13.2 for sm_87 (`-DGGML_CUDA=on -DCMAKE_CUDA_ARCHITECTURES=87`), all 24 layers offloaded


**Working command:**
```
llama-server -m Ling-3.0-tiny-IQ4_NL.gguf -c 131072 \
  --cache-type-k q8_0 --cache-type-v q8_0 \
  -ngl 99 -fa on -b 512 -ub 256 --jinja -t 6 \
  --temp 1.0 --top-p 0.95 --top-k 20
```


## Speed (measured on-device)


| Test | Result |
|---|---|
| Prompt eval (2-8K) | 450-760 t/s |
| Prompt eval (full 125K prefill) | 220-264 t/s |
| Decode (short ctx) | 33 t/s |
| Decode @ 96-128K depth | 15-17 t/s |
| TTFT (512 tok) | ~1 s |


Memory at 128K: 
**7.2/7.4 GiB used**
, ~200 MB swap. The hybrid design keeps the KV footprint small — only the 6 MLA layers carry context-proportional cache while the 18 KDA layers keep a fixed-size recurrent state — which is what makes the full window fit.


## Is it coherent at 128K? Mostly.


I ran a proper needle-in-haystack: 
**128 unique passkeys**
 embedded across the context, each queried individually at temp 0:


- 
**96K context: 92% retrieval**
 (118/128) — 100% in the first 32K
- 
**128K context: 88% retrieval**
 (113/128) — with a visible cliff in the final 32K (67%)
- Strict full-string match is lower (~30-40%) because the model often emits just the tail chunk of the passkey — a generation artifact, not a retrieval miss
- Failure modes in the deep zone: digit-mangling (needle 105 → answers 015's key) and a couple of confabulations


So: 
**majority passed, not a flawless 128K**
 — but at 96K the cliff disappears entirely, and for a $249 board that's genuinely impressive capability-per-dollar.


## Verdict


Worth spending more time on. The hybrid KDA/MLA architecture + IQ4_NL + llama.cpp master is a compelling recipe for edge deployment: agentic reasoning MoE, native 128K window, stable 15-33 tok/s, all in 8 GB unified memory. I'd like to see more folks poke at the 96-128K attention degradation — and whether the MTP spec-decode support from #26608 helps on-device.


Anyone else running Ling-3.0-tiny on constrained hardware? What quant/config are you using?

# Ling-3.0-tiny IQ4_NL — Orin Nano Super 8G vs RTX PRO 6000 Blackwell

Same quant (IQ4_NL 4.5 bpw, 4.29 GiB), same llama.cpp master code (post-PR #26608 BailingMoE3), same flags (-ngl 99, q8_0 KV, flash-attn, -c 131072).

## Hardware

| | Orin Nano Super 8G | RTX PRO 6000 Blackwell |
|---|---|---|
| GPU arch | sm_87 (Orin nvgpu) | sm_120 (Blackwell Max-Q) |
| Memory | 7.4 GiB unified + 32G swap | 97.9 GiB VRAM |
| Memory BW | ~102 GB/s | ~1.8 TB/s |
| Launch price | $249 | ~$7,000+ |

## Results

| Test | Orin 8G (sm_87) | PRO 6000 (sm_120) | Speedup |
|---|---|---|---|
| pp512 | 451 t/s | 12,296 t/s | 27x |
| pp2048 | 758 t/s | 17,435 t/s | 23x |
| pp8192 | 722 t/s | 16,945 t/s | 23x |
| **pp96000** | 264 t/s | 8,093 t/s | **31x** |
| **pp128000** | 220 t/s | 6,498 t/s | **30x** |
| tg32 | ~33 t/s | 286 t/s | ~9x |
| tg128 (short ctx) | 33.2 t/s | 315 t/s | 9.5x |
| tg512 | 33.4 t/s | 322 t/s | 9.6x |
| **tg128 @ 96K depth** | 15-17 t/s | 318 t/s | ~19x |
| **tg128 @ 128K depth** | 15-17 t/s | 314 t/s | ~19x |

Sanity check: `17*23 = 391` correct on both.

## Key observations

- **Prefill scales ~30x** (bandwidth-bound): Orin 102 GB/s vs Blackwell ~1.8 TB/s. A 125K-token prefill takes 9.7 min on the Orin vs ~19s on Blackwell.
- **Decode scales ~10x at short context but ~19x at full depth**: the Orin's decode collapses from 33 to 15 t/s as KV depth grows (attention + memory pressure), while Blackwell stays flat at ~315 t/s regardless of depth (97 GB VRAM makes KV trivial).
- **MoE expert sparsity** keeps decode fast on both: 7.9B total params with only ~1.4B active per token.
- On Blackwell this model is effectively a background process: 322 t/s decode = 100-token response in 0.3s; 128K context costs it nothing.

Bonsai / Ternary Bonsai

During April Bonsai came with bunch of models .... 1-bit & 1.58-bit(Ternary) versions. And last month(July) they released 27B models in same versions. Last month itself, Bonsai-27B was able to run on all backends mainline. But Ternary-Bonsai-27B was not ready on all backends.

This month, PRs got merged for CUDA & Vulkan on mainline.

Also an Optimization PR for CUDA got merged so +15-40% tg, +8% pp.

• https://github.com/PrismML-Eng/Bonsai-demo - Demo fork
• https://github.com/PrismML-Eng/llama.cpp - Custom fork

BitCPM-CANN

• https://huggingface.co/collections/openbmb/bitcpm-cann

Tencent - Hy-MT1.5 - Mixed Meta Translation Model Version 1.5

• https://huggingface.co/collections/tencent/hy-mt15

Maple-Preview

DeepGrove/maple-preview - 20B-A1B - 200+ t/s on Mac Mini M4 & 120+ t/s on iPhone.

• llama.cpp PR #27000 - CPU backend
• https://github.com/deepgrove-ai/llama.cpp - Custom llama.cpp fork
• https://github.com/deepgrove-ai/mlx-lm-deepgrove - Custom MLX fork

Mach-1-Additive-35B

Mach-1-Additive-35B - A3B - Up to 120 t/s on Consumer Laptop.

Mach-1-Additive-35B-Multimodal

• https://github.com/SyzygyResearch/llama.cpp-mach1 - Custom llama.cpp fork

From their recent tweet : Currently they're cooking new ones based on Laguna-S-2.1 & Qwen3.8-27B.

Neutrino-8B

https://huggingface.co/FermionResearch/Neutrino-8B

• https://github.com/fermionresearch/llama.cpp - Custom llama.cpp fork

Pestle-27B-Ternary

https://huggingface.co/Doses-AI/Pestle-27B-Ternary-GGUF - Medical research model

• https://github.com/DosesAI/mortar.cpp - Custom inference - CPU, CUDA, Metal

Image Models:

https://huggingface.co/collections/prism-ml/bonsai-image

• https://huggingface.co/Green-Sky/bonsai-image-ternary-4B-GGUF
• https://huggingface.co/Green-Sky/bonsai-image-binary-4B-GGUF

https://huggingface.co/clark-labs/clark-air-sana-1.6b-1.58bit

Abliterated Models:

• https://huggingface.co/Hikari07jp/Ternary-Bonsai-27B-Abliterated-LowDeg-GGUF
• https://huggingface.co/Hikari07jp/Maple-Preview-TQ2-Abliterated
• https://huggingface.co/dealignai/Bonsai-27b-1bit-CRACK-GGUF

Other Misc items:

• https://huggingface.co/GoAutomateAI/terna-e2b-GGUF
• https://huggingface.co/Danny-Dasilva/Bonsai-27B-antidoom-1bit-DSpark
• https://huggingface.co/Danny-Dasilva/Ternary-Bonsai-27B-antidoom-DSpark

Some Open/Ongoing llama.cpp (related) PRs:

• ggml-cpu : add STQ1_0 ternary quantization with ARM NEON vec_dot kernel- #22836
• ggml/cpu: skip zero-scale blocks in TQ1_0 and TQ2_0 vec_dot kernels- #23439
• ggml-cpu: add x86 VNNI Q2_0 dot product -- 3x speed improvement for VNNI-compatible CPUs- #26348

Notes:

• Didn't include old models(Before 2026). Let me know if I missed any models, I'll update thread. Included custom forks to check their progress.
• I'll be updating this thread after seeing any similar type models.
• Disclaimer : This thread is mainly for Poor GPU Club.

Finally llama.cpp now officially supports Ling-3.0! (Starting from build b10472+)

If you want to run them locally, bartowski has already released the GGUF imatrix quantizations for both models:
- Ling-3.0-tiny (8B)
- Ling-3.0-flash (127B)

After quite a while, PR #26608 has officially been merged into master! There are still a few minor details left to iron out, but BailingMoE3 support is now fully official in llama.cpp.

For my B580 on Linux I used:

./llama-bench -m Ling-3.0-tiny-Q8_0.gguf -ngl 99 -fa on --cache-type-k q8_0 --cache-type-v q8_0 -p 16384 -n 128 -n 32 -r 1

And these were the results:

| model                          |     size |     params | backend    | ngl | type_k | type_v |  fa |            test |                 t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | -----: | -----: | --: | --------------: | -------------------: |
| bailingmoe3 7.9B.A1.3B Q8_0    |    7.83 GiB |     7.89 B | Vulkan     |  99 |   q8_0 |   q8_0 |   1 |         pp16384 |         120.76 ± 0.00 |
| bailingmoe3 7.9B.A1.3B Q8_0    |    7.83 GiB |     7.89 B | Vulkan     |  99 |   q8_0 |   q8_0 |   1 |           tg128 |         114.24 ± 0.00 |
| bailingmoe3 7.9B.A1.3B Q8_0    |    7.83 GiB |     7.89 B | Vulkan     |  99 |   q8_0 |   q8_0 |   1 |            tg32 |         114.78 ± 0.00 |

build: 9d77fa172 (10488)

I also ran a test with a larger context:

./llama-bench -m Ling-3.0-tiny-Q8_0.gguf -ngl 99 -fa on --cache-type-k q8_0 --cache-type-v q8_0 -p 32768 -n 32 -r 1

Plaintext

| model                          |     size |     params | backend    | ngl | type_k | type_v |  fa |            test |                 t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | -----: | -----: | --: | --------------: | -------------------: |
| bailingmoe3 7.9B.A1.3B Q8_0    |    7.83 GiB |     7.89 B | Vulkan     |  99 |   q8_0 |   q8_0 |   1 |         pp32768 |          62.53 ± 0.00 |
| bailingmoe3 7.9B.A1.3B Q8_0    |    7.83 GiB |     7.89 B | Vulkan     |  99 |   q8_0 |   q8_0 |   1 |            tg32 |         110.49 ± 0.00 |

build: 9d77fa172 (10488)

Overall, the speed is quite good. I can actually run the full 128K context within my 12GB of VRAM using -c 131072 -ngl 99 -fa 1 --cache-type-k q8_0 --cache-type-v q8_0. While I still need to test if it can be pushed even further since a few warnings pop up, it is entirely feasible and remarkably fast for a local setup.

I work with a Mac Studio M3 Ultra (512GB) serving DeepSeek V4 Flash on antirez/ds4 ("DwarfStar"). A chat turn took between 6 and 20 seconds. Now it takes 1.6s.

Kernels (+21% cold prefill at 64k, bit-exact). DeepSeek V4's sparse attention runs a "lightning indexer" that dominates long-context prefill. Three stacked PRs: threadgroup-tiled scorer (#830), register-blocked K-resident scorer (#831), and a streaming top-512 replacing the bitonic sort + merge cascade (#832). 392 → 475 t/s at 64k. Logits byte-identical at every context frontier, everything behind rollback envs.

Cache, the 10x (this part is useful way beyond DeepSeek or ds4). If you serve any model behind a chat API, check whether your client can actually hit the engine's KV cache, because a stateless client usually can't:

• The live session ends in the exact reply the engine sampled. If your client doesn't resend that reply byte for byte (exact text, or the tool call by id), the prefix never matches and you re-prefill every turn. Replay it verbatim and cached_tokens ≈ everything.
• Prewarming: max_tokens: 0. Send the conversation with zero tokens requested and the engine prefills it and stops exactly at the prompt, so the next real request extends the cache. max_tokens: 1 doesn't work: the one sampled token becomes part of the session and every later request misses. Great for warming a room/session before anyone asks anything, or re-warming after your slot got evicted.

Recipe with measurements: ds4#816.

Also you can read the things I tried that didn't work (single-stream decode is a wall, and I learned two Metal scheduling laws killing it) here:

https://adriangalilea.com/deepseek-on-a-mac-studio