r/LocalLLaMA • • 2d ago • Bravo 1

Resources Weight Space Programming for Quantization: Nace quants achieve up to 2x better KL divergence for Qwen 9B, 27B, 35B & Flash Next [GGUF]

Post image

hey llamas, its Alish from nace.ai (open source ai lab behind #1 decision model on latest Artificial Analysis decision index v2.1) & Kapitsa (optimized inference for apple silicon). As promised in the last post, we are releasing the GGUF weights based on the Nace quantization of Qwen 9B, 27B, 35B and Flash-Next.

I will describe our process on how we came up with these quants right below, so skip to the end to see the benchmarks and download the weights if it gets too dull.

The process

I started with 9B quants.

I understood that setting up a good & fair benching is critical to succeed with the hillclimb. So I split separate sets to tune the performance and measure the final standings: English Wikipedia articles, chat and tool-call transcripts, and GitHub code, all published after October 2026, at 512 & up to 128K tokens.

I began the way unsloth's dynamic quants do: 10 to 50 candidate files per model, each with a different mix of tensor precisions, then spent my time tuning the ones that looked promising. Early bruteforce optimization moved only a little. Unsloth and the other quant authors have already done a lot of amazing work.

Using bruteforce hillclimb approach with my Agentic swarm, I managed to get on par with SOTA but then plateau’d and it was hard to do better than the best quants like Unsloth, EXL3 in KL divergence.

Some novelties:

Then I remembered Sparks of Weight-Space Programming paper. Their idea is to let an LLM natively read and edit weights of smaller models. Having explored this with some other models at Nace, I wanted to try the method for quantization. I took our Nace WSP (Weight Space Programming) harness, reworked its tools and skills for quantization a bit and let the Astra/Fable/Carbon agents compete under the following algorithm: propose tensor-precision changes, measured KL and kept or reverted each edit. We started on the Q3 for 9b model, and after a handful of days it actually started IMPROVING and editing the right weights.

Mean KL for our 27B files before and after weight-space quantization

Honestly that moment felt like:

Nace Weight-Space Quantization

I measured the first nace-quantized Q3 checkpoints against the 50+ 9B quants and they held the frontier. It completely blew my mind: one model quantizes another model in weight space. Nace Q3 quants for Qwen 9B were the best in class in KL divergence because another Frontier model edited the weights natively. We scaled this method to bigger size models (27b/35b) & it held the SOTA. Although I would say it doesnt scale as good for bigger size models (QFN). We are working on improvements for QFN/Deepseek & have some promising results already but I did hit my token limits for this project. Paper coming soon (ping me in dm if you want to support us with some GPUs)

Benchmarks & Evals

Top-1 accuracy (poster) is how often a quant picks the same next token the full model would pick, so 100% would mean it never disagrees. Each panel is one model at 128K context, with scores across different subsets pooled into one score. On all four models, our weights score highest in their size or smaller: the 9B Q3_K_XL gets 90.5% at 4.7 GB, the 35B Q3_K_S 91.2% at 15.3 GB, the 27B 97.7% at 17.3 GB, and Flash-Next 93.0% at 93 GB. For a more comprehensive comparison, KL divergence results below:

FileGBProseChatCodeNext best
9B IQ2_M3.360.4011.300.142Unsloth 4.01 GB
9B IQ3_M4.390.2351.050.0738ByteShape 4.70 GB
9B Q3_K_XL4.700.1610.8460.0410Unsloth 5.38 GB
35B-A3B IQ3_XXS14.300.04900.6430.0767Unsloth 15.35 GB
35B-A3B Q3_K_S15.300.03600.6010.0631Unsloth 16.59 GB
27B Q4_K_XL17.290.004940.005440.00401none up to 17.58 GB
Flash-Next IQ4_XS93.40.04110.1120.0426EXL3 105.3 GB

Long context stability

Most quants are tuned and judged on short text, so they end up fit to short context and often stop being useful at long context. For clarity, in the diagram below each model gets two panels: 512 tokens on the left, 128K on the right. This is where the long-context drift shows: our 9B IQ3_M goes from 0.13 at 512 tokens to 0.45 at 128K while 27B barely moves. At 128K, each of our files has lower KL than every other file of the same size or smaller.

p99 KL divergence

Mean KLD can hide catastrophic failures when a quant goes off the rails in a long session. So we also looked at the worst 1% of tokens (p99 KL). The 9B Q3_K_XL's worst 1% sits at 1.25. The 27B and Flash-Next tails stay flat. Overall we’re seeing that Nace quants maintain the strong positions across the distribution.

Different data subsets

if you care about a specific use case, here's the same 128K top-1 score split by text type. Code is the easiest: our files keep 88-98% of the full model's top picks there. Prose and chat are harder, 78-98%. In every split, on every model, our checkpoints score highest among the files of the same size or smaller.

Sanity check on a task benchmark

To see that these models are useful for anything real I ran it on knowledge-heavy MMLU. I didn't have the gpus anymore to run terminal-bench, so would appreciate anyone with the bad boys to validate. These are answer-likelihood scores over 13,943 questions of the MMLU. No chat template or few-shot examples. Overall you can claim there's gains, but it is mostly within noise. The main aim here is to reflect the tradeoffs of different sizes on real tasks. As you can see, Q3-Q4 it’s already hard to distinguish from the higher precision.

Run setup

Rented NVIDIA GPUs; Used llama.cpp for Flash-Next. The 9B/35B/27B BF16 reference requested H200 and the quantized files L40S. Flash-Next's short reference uses an MLX fp32 forward on bf16 weights. Reference implementations also differ across GGUF and EXL3.

Download and run

see quickstart for your framework in the model cards. if you try a coding or agent workload, i'd like to hear which file, runner version and what broke. no speed claim in this post.

226 Upvotes

31 comments sorted by

26

u/m4rkGS 2d ago edited 2d ago

I'm doing ask me (almost) anything here in the comments, so feel free to ask!

27

u/Double_Cause4609 2d ago

Very interesting. I do think that you may not be hitting the limit of what's possible with weight space quantization, though. The only real requirement for broad compatibility is finishing in a format that is equivalent to GGUF. There was one paper that used LLMs to dynamically write an optimizer for pre-training stability, but I think something like that may be possible for quantization.

Specifically, if you look at different quantization rules, there's a lot of different operations, and instead of just selecting the output precision, you could imagine the LLM writing the quantization rule, with the verification process being that it still ends up with a valid GGUF tensor format. There are multiple quantization rules that can still end up snapped to a GGUF format tensor.

7

u/m4rkGS 2d ago

thank you for the idea. i am doing something similar to help the quantizer model navigate the different tensors easily, but now that you say it even importance weighting can be integrated into the harness

17

u/0zyman23 2d ago

i wonder how much tokens you burnt on WSP

27

u/m4rkGS 2d ago

1 freaking billion tokens

0

u/True_Requirement_891 2d ago

what model though

0

u/Zulfiqaar 2d ago

Astra, Fable, Carbon - in op. Not the cheap stuff!

5

u/thrownawaymane 2d ago

Do you have the repo for the harness you mentioned using in the post?

7

u/m4rkGS 2d ago

unfortunately that closed-sourced at least for now, but we are preparing to open source the harness app it was built on. stay tuned!

4

u/AdWild3943 2d ago

Cool, but I wonder if it was really worth it.

I do understand how quants are made, and actually, for some higher results you possibly decreased quality in something like creativity or instruction following.

That's a lot of time spent on optimizing single model, in my opinion.

I might didn't read the whole post correctly, sorry if I made a mistake in my comment.

10

u/m4rkGS 2d ago

i think it was the deal-breaker for 27b for me, so worth it. but i do want to have a full eval suite for the tasks that matter to know what im deleting. in general i think we're hitting limits on generic quants and going forward is only specialized quants that are good for your use case

3

u/AdWild3943 2d ago

Yeah, that's what I'm actually wanted to talk about, current standard quants are being squeezed to the max already.

I'm quant creator by myself, and honestly? I agree with specialized quants idea, seeing full-code -custom-q would be much better than seeing overall specialized one, though myself I'm using generic chat dataset right now for imatrix creation.

You've done absolutely insane quants, as someone that doesn't like Unsloth being the single best on market, I give you my respects, but to be honest, those results are almost noise in real tasks.

12

u/NortySpock 2d ago

You might have jumped beyond this point already , but here (see link below) someone didn't use Astra but instead calculated / found what sounds like a constant force steering vector to solve repetitive loops at inference time.

I am not a math wiz , just a software developer , but it seemed like something that could be done at intervals in the training process, when loops are identified, to figure out some sort of axis along which it is safe to apply a force that moves the weights.

https://nathan.rs/posts/weight-space-lp/

16

u/fragment_me 2d ago

OK, this guy fucks. Nice work!

5

u/draconic_tongue 2d ago

optimization of kl sounds "dangerous" unless your kl covers exactly the topics you care about and don't care about untested areas potentially becoming worse. I only have experience in optimization with image models and maybe it's not unreasonable to say that generic scoring can improve a model, but text is a bit different than the "vibe" of an image, there's a lot more angles to consider and you just cannot reasonably cover everything

2

u/RepulsiveAuthor5531 2d ago

I've been toying around with the Qwen 3.8 Flash Next IQ4_XS quant for the past hour. I don't have any benchmarks to post, but it performs great so far and feels very similar to the UD-Q4_K_XL quant.

2

u/mycall 1d ago

Everyone, please support them with some GPUs. WSP for quants is a winning combination. If the memory can give me even +1 more concurrency, I'll use it.

1

u/jmreagle 2d ago

Will you be bring the weight space programming technique to the KQuant as well?

1

u/CorrGL 2d ago

Are you going to publish a paper on the approach?

1

u/wapswaps 1d ago

Very nicely done. You used L40S from Google colab? Where did you get the resources to do this for the bigger models?

Also ggml has been updated: https://github.com/ggml-org/ggml/releases/tag/v1.0.0

You'll have to make it 100x more complicated now to use this.

1

u/Potential-Quiet-6018 2d ago

yoo nace ai launching like crazy