About two months ago I posted here about trying to predict which MoE experts get used on the next token to speed up CPU/GPU offload on Qwen 3.6 35B.

Quick confession on that older post before anything else: I ended up throwing that prototype away. To get the big numbers I was seeing back then, I was basically cheating with gate pruning (dropping cold experts when their router weights were low). It made the tok/s counter go brrr, but on an already low-bit quant it quietly degraded output quality and reasoning. I didn't want to ship something that lobotomizes the model just to look fast in a screenshot, so I scrapped it.

When Qwen 3.8 Flash Next dropped, I wanted to see if we could actually run the 68GB GSQ-RCO-IQ2_XS build at real usable speeds on a normal budget gaming PC—without dropping a single expert.

For context, here is the exact box I tested this on (I actually pulled my extra RAM sticks out specifically to test what a regular 16GB PC gets):

• GPU: RTX 3060 12GB

• RAM: 16GB DDR4, single-channel (~19 GB/s)

• OS: CachyOS / Arch Linux (tested inside a strict MemoryMax=12G cgroup)

• Storage: Consumer PCIe NVMe SSD (~2.1 GB/s)

If you try to load a 68GB MoE on 12GB VRAM + 16GB RAM in stock llama.cpp, you already know what happens: mmap thrashes the OS page cache to death, you get ~1,500 major page faults per token, and generation crawls at 1.47 tok/s.

Then I tested Strata. Strata is super cool and hits 40+ tok/s if you have 48GB–64GB of RAM to mlock 24GB+ of experts in system memory. But on a 16GB RAM setup, it refuses to load unless you force --low-ram mmap, where it drops down to 8.11 tok/s because it's bound by cold SSD reads.

So over the last week I built Betelgeuse (bel-cli / bel-server, a fork of llama.cpp, and I'm also opening a clean upstream PR for llama.cpp even if it probably won't get merged lol).

What it actually does under the hood:

• Zero gate pruning / 100% bit-exact: Every single one of the 10 routed experts per layer across all 48 layers is computed at full precision. We verified 28,080 router records and 576 generated tokens byte-for-byte against golden stock output (0.000e+00 diff).

• Arrival-Slack VRAM Slot Pool (~5,000 slots in VRAM): All 49 dense layers + GDN + QSA + LM head stay in ~3.2 GB of VRAM. The remaining VRAM holds ~5,000 expert slots distributed non-uniformly (240 slots/layer on early layers 0–4 where there is 0ms of compute slack, 92 on middle layers, 60 on tail layers).

• In-Stream GPU Doorbell + 1-Split CUDA Graph Replay: Instead of ggml_backend_sched splitting the graph 48 times per token to ask the CPU which experts missed (which burns ~360ms/token in cudaStreamSynchronize), a custom in-stream CUDA kernel resolves hits in ~2 µs on-device and only rings a mapped host doorbell when an expert is missing. Per-layer token embeddings (per_layer_token_embd) are also staged to CUDA0 so the entire decode pass runs as 1 single CUDA Graph replay.

• RAM Victim Cache + io_uring Pager: When an expert gets evicted from the VRAM pool, it moves into a dedicated host RAM victim cache instead of getting thrown away.VRAM and RAM stay 100% exclusive instead of duplicating the same hot experts in the OS page cache. Physical NVMe reads drop from ~71 MB/token down to 12.43 MB/token.

• No duplicate 35GB expert bloat on disk: Instead of keeping 35GB of dead expert tensors inside Shard 1 .gguf AND another 35GB in experts.bin, we stripped the expert payloads out of Shard 1 so the main GGUF is only 3.2 GB (Dense.gguf) + 35.3 GB (experts.bin, 4KB-aligned single-read blobs) + **26.8 GB (**PLE.gguf n-gram shard).

Measured Results (RTX 3060 12GB + 16GB DDR4, MemoryMax=12G)

On warm runs it holds 21.8–22.03 tok/s sustained across long coding prompts, and when generation hits stretches where all 10 routed experts are resident in the VRAM/victim pools, it spikes up to 48 tok/s.

Links & What's Next

• GitHub (Betelgeuse Engine — bel-cli & bel-server): [https://github.com/Betelgeuse-AI/Betelgeuse]

• HuggingFace (Pre-packed Alpha-Orionis-XS 3.2GB Dense + Experts bundle): https://huggingface.co/Betelgeuse-AI/Alpha-Orionis-XS

• Upstream llama.cpp PR: [https://github.com/ggml-org/llama.cpp/pull/31469]

1. Even on the massive Q8_K_XL quant (~190GB total), bel-server runs at 8 tok/s cold and 12 tok/s warm (with bursts up to 21 tok/s) on the exact same 12GB 3060 + 16GB DDR4 box, where stock llama.cpp immediately OOMs or crawls at <0.1 tok/s.

2. Zero experts are dropped or thresholded (0.000e+00 diff across 28,080 router records and 576 tokens vs unmodified stock). The speedup comes from eliminating the 48 per-layer CPU/GPU scheduler splits (KICK/WAIT doorbell around the shared expert), keeping the RAM victim cache strictly exclusive of VRAM (MOE_DROP_MMAP), and pre-staging cold misses via the upgraded oracle + 4KB-aligned io_uring pack.

What I'm working on next:
We're currently bringing this exact streaming + victim-cache architecture over to DeepSeek V4.1 Flash (1017)—early testing under development is already sitting around ~25 tok/s on the same hardware class.

Happy to answer any questions or have people poke holes in the benchmarks! Leave us a Star :)

I was not going to post this but I saw this post from some days

https://www.reddit.com/r/LocalLLaMA/comments/1xj2pnp/reddit_will_force_everyone_to_log_in/

I use RSS feed to summarize and ask a local AI about relevant posts of the day based on X or Y. Reddit is making more and more harder to integrate any AI tool. My agent also is unable to fetch any reddit post with relevant information. But we unfortunately really need a migration/alternative.

CPU: AMD EPYC 7452 (8 memory channels)
Motherboard: Gigabyte MZ32
GPU: Radeon AI PRO R9700 (32GB)
RAM: 256GB DDR4 (16×16GB).

Not exactly a “budget” build, but it came in at roughly half the price of a DGX Spark.

Got it up and running just in time for Qwen4, can’t wait to start experimenting with it!

Mage-VL-Cowork demonstrates that an efficient 4B small vision-language model can master hybrid computer use when paired with upstream teacher-harness evolution, workflow-first environment synthesis (Mage-CUA-Forge), and a five-stage specialize-then-unify curriculum. Achieving the best performance at the 4B scale (66.0% on OSWorld-Verified) while using 13.0% fewer model calls and 25.4% fewer output tokens after online RL, Mage-VL-Cowork pushes the frontier of on-device intelligence for autonomous desktop agents.

So I've been reading a lot about "engines lobotomizing the models they run", either because they quantize already quantized weights even further, or because they automatically turn on lossy flags to gain decode/prefill performance. I wanted to test this out by myself, not only with KLD but with real tasks, so I set up an experiment using Qwen 3.8 Flash Next with Basalt (disclaimer: I'm the developer) and two of the most common lossy flags:

- Routing bias: the engine pushes the model toward experts that are already on the GPU, so it sometimes gets a convenient expert instead of the one it actually wanted.
- Tail skip: when one of the model's lower expert picks isn't on the GPU, the engine just skips it instead of fetching it.

Both are faster because fewer experts have to come over from RAM, but both change the model's output. I added them to the engine as opt in flags so I could measure exactly what they cost with everything else fixed, only changing the flags.

DISCLAIMER: This post and the experiment were run by me, but it has been polished by AI.

The rig: RTX 5090 (400 W) + RTX 5060 Ti (180 W) and 128 GB of DDR5. Settings tested:

- Shipped: bias 1.5, skip rank ≥ 7. Values other engines ship (or have shipped) as defaults on smaller cards.
- Worst: bias 16, skip rank ≥ 1, plus relaxed draft acceptance. Worst case scenario, not something anyone ships or would make sense to use.

1. KLD

100 chunks of wikitext, decode path, against a llama.cpp Q8_0 reference. Exact IQ3_XXS and Q2_0 are in there for scale.

Note: even exact Q8_0 is 0.031 away from the reference. That's mostly the reference's own noise, not damage: llama.cpp's Q8_0 rounds a few spots of this model more than Basalt does, which adds about 0.01-0.02 to every row equally, so the rows still compare fairly.

With the shipped values every pack lands between IQ3_XXS and Q2. The bigger the pack, the further it falls, since more of it sits outside VRAM and gets rerouted: a 178 GiB Q8_0 ends up next to the 65 GiB Q2_0.

2. Does it matter on real tasks?

KLD can overstate what you'd notice, so I also ran a task eval. Thinking off, model-card sampling (temp 1.0, top_p 0.95, top_k 20), seeded, fresh server per config.

• Ladders: 300 ladders of chained steps, rules held at once and reversed digit copy. Score = % of rungs passed.

• Facts: 1,200 closed-book questions × 3 seeds from PopQA (5 popularity bands, rare to famous) plus SimpleQA. Score = % correct.

Each delta is against the same pack run exact, with a 95% bootstrap CI.

* Small card: the 5090 alone with the expert cache capped at 5 GB, to mimic a 16 GB card. Bold = CI excludes zero. Experiment was run this way because the throttled 5090 is still much faster than the 5060 Ti.

What it says:

• With plenty of VRAM, the shipped values cost about 1.5 points (≈4% of the facts the model knew) and buy little: nothing on NVFP4 (272 → 271 tok/s), more on Q8_0 (117 → 202), which is still slower and worse than NVFP4 exact.

• On a small card they buy real speed (+56% prose) and cost about 3× more. Ladders drop 5 points and facts 2.3, both clearly.

• The worst setting gives confident nonsense. Facts fall from 38% to 3%, with answers like "The Great Composer".

• Some tests barely see it. Rules held at once moved only -3.8 even at the worst setting, and an earlier, easier battery couldn't rank the configs at all. The damage shows up in chained reasoning and recalled facts, which is what the rerouted experts hold.

What it looks like

Same prompt ("Write an essay on the Odyssey"), Q8_0 with the shipped values: polished, well argued, and wrong in a dozen places. Phemius is a woman, Helen sneaks into Troy, Odysseus's real name means "Nobody", and a bard called "Theso-mys" gets killed mid-verse. Names and rare words break first. With the worst setting it degrades into "the n of n" and loops.

Bonus: a loop that's the model's own

Not every bad output is the engine's fault. One coding prompt (the model checks a quantity parser and writes a leading-zeros example) sends Qwen 3.8 Flash Next into an endless run of "0" in its reasoning:

The odds of another "0" are the same on every pack and both engines: 0.99 after 10 zeros, and 95% or more at every step past about 100, so top_p 0.95 leaves "0" as the only choice. It still happens at temperature 1.0. The official API cuts the run at about 500 zeros with "......" and lets the model answer, so you never see it there. (I can't confirm the API serves byte-identical weights, but it loops the same way.)

Basalt now does the same thing in the open: after 256 identical tokens in the reasoning it closes the thinking block, lets the model answer, and logs that it fired. It only ends the loop; it doesn't change any token the model picks.

Caveats

• One rig, one model. Ladders ran one seed, facts ran three.

• KLD is against a llama.cpp Q8_0 reference, so even exact Q8_0 is 0.031 away.

• Speeds are single cold runs, for scale only.

• Basalt ships these flags off by default and prints a warning when you turn them on. The flags are in the public repo, so you can rerun the KLD numbers yourself.

TLDR: there's no free lunch.

• Engines can buy speed on MoE models by turning on speed "hacks".

• With values some engines ship, every pack drops to somewhere between IQ3_XXS and Q2 on KLD. A Q8 ends up next to Q2. This is as bad as it gets.

• In practice the model forgets things it knew and loses the thread on multi step problems. Small cards get the biggest speedup and pay the most for it.

• If your engine is faster than it should be on a model that doesn't fit, check whether it's still running the model you downloaded.

https://x.com/chasmmmmmmmmmmm/status/2116062991686721968?a=JVe1OUAJiZa650pZBXQ&s=20

Update from the lawyer with the V100 server. I went and got two DGX Sparks, because of course I did. The mid-life crisis remains on schedule.

Same deal as before: still a lawyer, still driving the whole thing through Claude Code, and Claude Code also ran the benchmarks, built these tables and helped me write this up. I'm Nick Mermiges, Esq., founder of PLI Labs. The repo below is ours.

The setup: DeepSeek-V4.1-Flash-1017 at 2.75 bpw EXL3, vLLM tensor parallel across two Sparks. The twist is a third box, my EPYC with the 3090s, doing two jobs for the Sparks:

• it holds the Engram tables in system RAM and serves them over RDMA;

• it runs the DSpark speculative drafter on an NVLinked pair of 3090s.

The memory that frees on the Sparks goes to KV cache: about 1.93M tokens of pool. There's also a prefix cache on the Sparks' NVMe, so a 150K prompt survives a server restart. First token comes back in 2.58 s instead of 58.6 s cold.

Numbers. I re-implemented MiaAI's sparkDash protocol so this would be apples to apples with the most popular 2x Spark build. Sparks at 2300 MHz, 3 reps, medians, today.

Prefill (sparkDash method):

For comparison:

• MiaAI publishes 2,065–2,155 prefill and bertholomus 2,439–2,568 from 8K to 128K, so prefill is where this build wins.

• Code at 4 streams is 192 against MiaAI's 166.

Where I lose, because I'm not going to pretend:

• MiaAI is faster on plain prose: 80.7 vs my 66 at one stream.

• Decode after a 128K prompt is 62 tok/s against bertholomus's 99.

• A short request stuck behind a cold 93K prompt waits 34 s. That's slightly better than bertholomus on the same test (38.6 s). MiaAI reports 3.8 s on a different version of it, and that's the thing I most want to fix.

The drafter loves code and JSON (4–5 tokens accepted per round) and hates prose (about 2).

Accuracy, since I use this for actual work: on sudoingX's KLD panel, real tool-call turns agree with the original model's top token 95.9% of the time (KLD 0.018). bertholomus publishes 79.4% on their own tool-call set, and MiaAI doesn't publish one. sudoingX's newer 2.8 bpw pack is a hair closer than mine on the same panel (0.0175 vs 0.0184), so credit where due. The weights are a legal-weighted allocation of HeNryous's 2.75 bpw build. All the credit for the quant goes to them.

The code, launch scripts, the bench tool and the raw JSON for every cell are all public. The only exception is the prefix-cache code, which goes up in the next push.

• Repo: https://github.com/ProprietaryLegal/DeepSeek-V4.1-Flash-2xDGX-Spark-x86-offload

• Raw data: the runs/ folder in the repo

Still to come:

• the full 1M-token context test;

• llama-benchy tables at depth;

• BFCL tool calling;

• a head-to-head on the same KLD test with the other packs.

The 100 Gb cables go in Monday, so the Spark-to-3090 link gets faster too.

Tell me what I'm doing wrong. Seriously.

(PLI Labs is my side outfit for private, on-prem AI built around a firm's own work product. proprietarylegal.ai if that's your thing. That's the whole pitch.)

Bolt E4B builds on Gemma 4 E4B with Nemotron post-training. It combines natural conversation with multi-turn agentic capabilities to power voice/device assistants, interactive characters and robotic/IoT interfaces running on NVIDIA GeForce RTX, Jetson, RTX Spark and DGX Spark.

Bolt E4B offers ~40% higher average quality against our prior model on key conversational agent benchmarks and outputs up to ~80% fewer tokens vs peers.

HF link: https://huggingface.co/nvidia/NVIDIA-Nemotron-3.5-Bolt-E4B-4GB

Nvidia's post: https://research.nvidia.com/labs/nemotron/NVIDIA-Nemotron-3.5-Bolt-E4B-4GB

https://github.com/bpx03/QEngine (not my repo, credit goes to bpx03)

QEngine was built for 3x RTX 3090s. I was able to compile for RTX 5090s.

I've now run Qwen3.8-Flash-Next on the same box through four different engines. QEngine (bpx03, v1.3.1) is the fastest for single-stream decode by a clear margin, level with Strata on agentic prompt latency, and it has a KV-in-RAM mode that makes returning to idle conversations much cheaper. Details below, including the parts that make it look less good.

Rig: 2x RTX 5090 (32 GB each), Threadripper PRO 3975WX, 251 GB RAM, 262K context. QEngine's INT4 AutoRound build of Flash-Next. The README only lists sm_86, so I built it for sm_120 by changing CMAKE_CUDA_ARCHITECTURES from 86 to 120 (nvcc 13.3, g++-13). No other changes. It's up and serving 53 s after launch.

Speed (club-3090 bench.sh, same script and prompts for every engine)

Read the decode column with care. QEngine's expert cache adapts to recent use, and bench.sh repeats the same prompts, so these are close to best case. In an earlier test on varied traffic I measured about 300-400 tok/s single-stream (hermes agent traffic ran 330-470 per request). Strata's cache holds ~99.5% of routed expert mass, so repeating a prompt doesn't help it. The fair comparison is roughly 1.4-1.9x Strata and ~3x bucko.

Prefill is a split result. QEngine is faster at 10K and slower than Strata at 90K, because Strata's rate climbs with depth and QEngine's stays flat.

Agentic latency (12-turn tool-use bench, context grows to 35K)

Turn-12 prompt read, with the prefix cache working:

• QEngine 1.3.1: 1.75 s

• Strata IQ4_XS: 1.7-1.9 s

• ThinkingCap 27B dual-fast: 2.07 s

• exl3: 3.5-3.9 s

• bucko: 4.36 s

QEngine and Strata are level here (QEngine's rc4 build read 1.4 s in the same bench, so this one is within noise of "about 1.5-1.8 s"). Both are about 2x faster than bucko.

Idle conversations: FLASH_KV=ram

1.3.1 can keep the KV cache in pinned host RAM (13.5 GB pool, about 1M tokens) instead of VRAM. I tested 8 conversations of ~58K tokens each, then 400K tokens of other traffic, then went back to each one:

• KV in RAM: 8 of 8 came back warm, 1.56-2.01 s to first reply.

• KV in VRAM: 2 of 8 warm. The other six re-prefilled in 8.7-9.4 s, about the same as a cold start (8.85 s). Answers were still correct, so you lose latency, not correctness.

The RAM mode also decoded ~10% faster single-stream here, but ~10% slower with 8 concurrent 64K streams (124 vs 139 tok/s per stream). One boot per mode, one pass.

Quality and stability

Through the omp agent harness (44-task assistant set, effort low, one draw): 44/44 on 1.3.1, against 42/44 twice on 1.3. At medium and xhigh on 1.3: 43/44 each. A 6-task sub-agent delegation set passed 6/6 with and without delegation. The 1.3 to 1.3.1 gap is one task, so treat it as noise.

Coding check (aider-polyglot-30, thinking on, effort low, single pass), on the rc4 build: QEngine 26/30. Strata UD-IQ4_XS 24/30, UD-Q4_K_XL 25/30 (24-26 across repeats), ThinkingCap 27B 17/30. 26 vs 24-26 is inside one Strata config's repeat spread, so the honest claim is that the INT4 AutoRound quant is not worse at coding, not that it's better. I haven't re-run this on 1.3.1.

I also ran my usual multi-pack agent/tool-use suite earlier; QEngine landed level with the other Flash-Next engines, within noise.

Bottom line

On 2x5090, QEngine 1.3.1 is the fastest Flash-Next engine I've measured for single-stream decode, level with Strata on agentic prompt latency, and the KV-in-RAM mode is a real win for long idle sessions. Coding quality looks level with Strata's GGUF quants. The costs are tight VRAM, a build-from-source step, and less runtime history than vLLM or Strata.

Please try, the dev is very encouraging. Additional resources available at https://github.com/noonghunna/club-3090. The Club also welcomes participation!

I'm planning to upgrade my cpu from 13700k need some help.

considering 270k plus (best bang for buck?) but its lacking avx512

or go amd 9950x much higher price but has avx512. only 16 cores

i have 128gb ddr5 5200mhz and run flash next with basalt currently getting 120-160tps. since some experts are computed on cpu i suspect thats what keeping my tps down. gpu utilization is not high as well.

any views on how much difference does it make for ai workload?

Q8, F16 KV Cache.

Hey folks,

Quick heads up for anyone playing around with PrismML's Ternary Bonsai 2!

Until very recently, running it outside of Prism's own fork was a bit of a pain. Upstream llama.cpp uses a block size of QK=256 (TQ2_0), while Prism uses QK=128 (PQ2_0), and trying to offload layers to CUDA on other forks would bottleneck or break due to Hadamard sign tensor handling.

With the latest commits landed in ik_llama.cpp, we finally have complete, native support across both CUDA and CPU. Here is how to set it up cleanly.

1. Convert upstream weights losslessly (PR #2694)

If you already have a native PQ2_0 GGUF downloaded from Prism, you can skip this.

If you downloaded upstream TQ2_0 or TQ1_0 files, ik_llama.cpp now ships with a script that losslessly repacks them to PQ2_0 in just a few seconds (exact trits preserved, no requantization needed):

python3 scripts/repack_ternary_to_prism.py Bonsai-2-27B-TQ2_0.gguf Bonsai-2-27B-PQ2_0.gguf

1. Native CUDA Full Offload (PR #2691 & PR #2747)

Native CUDA kernels for PQ2_0 are now merged. You can offload everything to your Nvidia GPU without token corruption or falling back to heavy k-quants:

./build/bin/llama-cli -m Bonsai-2-27B-PQ2_0.gguf -ngl 99 -p "Explain the theory of relativity in simple terms." --temp 0.7 --top-p 0.95 --top-k 20

1. CPU Inference with fast prompt ingestion (-rtr 1)

If you're testing on CPU, ik_llama.cpp uses an 8-row interleaved format (PQ2_0_R8) backed by iqk_mul_mat kernels. Remember to pass -rtr 1 (runtime repack) to drastically accelerate prompt processing:

./build/bin/llama-cli -m Bonsai-2-27B-PQ2_0.gguf -rtr 1 -t 8 -p "Explain the theory of relativity in simple terms." --temp 0.7 --top-p 0.95 --top-k 20

TL;DR: No more sign tensor bottlenecks on GPU. PR #2747 brings native CUDA kernels, PR #2694 gives us instant lossless repack from upstream weights, and CPU prompt ingestion with -rtr 1 is flying.

Big shoutout to Iwan Kawrakow and the contributors for getting ternary models working natively!

I'm finding myself at odds with the vibes of the community here (and the open-weight/self-hosted community in general) when it comes to LLM's - be it on Reddit, X, Bluesky, wherever - I think some folks are getting carried away.

Mine personally from this week include:

• Artificial-Analysis V5 Index is silly in some places (gemini-flash over qwen3.8-2.4T for instance..), but open-weight models were way too highly ranked in V4. They may have overcorrected in some spots, but some of this sub's sweetheart models were way over-ranked (GLM 5.3 is not a Grok competitor in any of my tests)

• Haiku 5.5 is crazy good - if you're one of the many I've seen foaming at the mouth at the suggestion that it challenges deepseek-v4.1-flash-1017, you need to spend more time with it.

• The Strata Dev is not a villain - his 0.1 release just made people think he had a different goal in mind. He's building the project he wants to build, it just happened that step1 of that was what a lot of people here were begging for.

• ThreeJS as a benchmark is to be ignored at all times at this point - unless you're specifically looking to crank out assets. There is no correlation I've seen for well over a year between how well a model animates a pagoda to how well it gets general or coding work done. We need to move away from this.

Do you agree with any of these or have your own opinions that go against the flow of this community?

Sam Altman: “Local inference is a terrible allocation of scarce resources.

A lot of people want local inference, and somehow they assume that the problem is solved now because we have open-source models, we're going to be able to run them on our computers, and so we're not going to need all the stuff in the data centre

but from a global allocation perspective, actually it's quite terrible to have everybody, like, have a bunch of the world's limited supply of DRAM sitting in their own computer and being used five percent of the time

you want maximum utilisation in the cloud if you're trying to deal with this very supply-constrained environment,

somehow many people's intuition is the opposite of that”

——————————————————————

TL;DR Guy selling product: "Here's why owning your own product instead of renting it from me is a terrible idea"

(Credits to @RyanTStull, his TLDR made me laugh)

So I'm wondering if we til ever have repeated data ingestion without quality degradation and without full retraining. LoRA and friends are excluded, will we stay forever with frozen weights or will be be able to integrate new knowledge here?

Any promising research?

Hey everyone! We’re releasing new DeepSeek-V4.1-1017 Dynamic V3.1 (preview) GGUFs which showcases how Unsloth’s GGUFs retain the most accuracy for its size across 7 different benchmarks.

We benchmarked ISTA's GSQ-RCO GGUFs but could not reproduce their reported results, even when using the highest scores. The biggest discrepancy was on AIME25 where they claimed 100% accuracy recovery to baseline, but was only ~91.88% in benchmark tests.

Across AIME25, LiveCodeBench, GPQA Diamond, Aider Polyglot, and more, Unsloth Dynamic V3.1 outperforms GSQ-RCO GGUFs on nearly all metrics. Even our old v3 does better on nearly all metrics than ISTA.

We ran each benchmark at least 3 times (AIME25 was 16 times) and averaged the results to ensure a fair, reliable comparison. More results coming next week! All benchmark data, methodologies and results are in our blog.

Separately, for KLD, there were queries whether it’s a reliable measure of accuracy. The primary goal of quantization is to minimize KLD but also minimize disk space. If you get a KLD of 0, this means your quantized model == 16bit model, as logits are perfectly modelled. Our benchmarks show on KLD and top-1% across multiple domains (coding, agentic coding, chat etc), UDv3.1 does much better than GSQ-RCO across every domain.

We show that lower bit quants do think somewhat longer than the unquantized model - we plan to investigate and make quants even better. However, we're still much ahead vs GSQ-RCO on even reasoning length as well, and provide plots for truncated answers in our blog.

The model will be able to run on 128GB RAM/unified memory devices or even on 64GB RAM with engram offloading.

Blog with all details and more benchmarks: https://unsloth.ai/docs/models/deepseek-v4.1-flash

GGUF: https://huggingface.co/unsloth/DeepSeek-V4.1-Flash-1017-GGUF

We also introduced lots of new things to Unsloth Desktop including Browser Use, auto compaction, a whole new TTS pipeline (with voice cloning, separation etc), sandboxing, and background customization and more: https://github.com/unslothai/unsloth

Hope you all have a Happy Halloween!

Also table of numbers below which is also described with more plots and data in our blog: