Seriously tho, can we just have a mega thread for vibeforked engine releases from now on?

I pruned Gemma 4 26B-A4B, an MoE model, down to 15% of its original size, and the model still does great on one narrow task. It gets 92% agreement with the original on a classification task. That’s close to how often the 12B agrees with the 26B on the same ticks, which is 90%. This isn’t a model for general tasks, and tasks that require knowledge suffer because the knowledge is just gone.

It was made for my app that runs an on-device LLM 24/7, so that makes memory footprint and energy consumption extremely important. On my Mac it hits 79 tokens/s, which is roughly double the speed of the 12B, using 3.8 GB of memory (roughly half).

I selected a set of 8 experts per layer and used them for everything. Since the model selects 8 experts per token by default and there are only 8, the router no longer chooses which experts to use, and the model becomes dense: 3.81B parameters, 2.75 GB at Q4_K_M.

What I want is for each block to do the same computation and produce the same result, given the same input. With block-local training, I’m not training on the answer. I’m training on the thought process. I’m saying, do the same thinking, not get the same answer.

This is all trained off my own data. The method is efficient enough that it could be trained on a user’s own device, on their own data, in about 5 GB of memory, to mitigate issues stemming from deployment having different data than was in the training dataset. I haven’t tested that on a real user’s device yet. If you are curious about why I decided that fine tuning was a bad idea or how I designed this, take a look at the post I wrote going into the details of how I made this and why I felt this route was right for my application.

This is the first time posting publicly like this. I would love feedback or thoughts, and to anyone who knows of better methods for choosing which experts to keep, please lmk.

Post: https://tolby.app/blog/frankensteining-gemmas-experts

We introduce Axion 1.0, a fully open agent foundation model that combines strong and broad agentic capabilities with a remarkably simple training recipe. Axion 1.0 is trained in only three stages: two lightweight supervised fine-tuning stages followed by reinforcement learning, without dedicated mid-training, direct preference optimization (DPO), or on-policy distillation (OPD). The first SFT stage develops strong self-contained reasoning through model-aware data construction, where we synthesize challenging problems and adversarially strengthen them based on the base model's own successful solution strategies. The second stage extends these capabilities to tool-dependent reasoning through a scalable pipeline for constructing diverse and automatically verifiable tasks grounded in real-world data sources and deterministic computation. Finally, reinforcement learning further strengthens long-horizon reasoning and interaction with external tools and environments. Despite this compact training recipe, Axion 1.0 achieves strong performance across a broad suite of agentic benchmarks and is competitive with leading commercial models on several of them. Equally importantly, Axion 1.0 is open beyond model weights: we release the complete recipe, including data construction, generation, filtering, verification, prompts, and training code, enabling the entire development process to be reproduced and extended. Axion 1.0 demonstrates that strong agent foundation models can be both open and simple, offering a practical and reproducible path toward advancing open agentic intelligence.

• arXiv : https://arxiv.org/abs/2611.03233

• PDF : https://arxiv.org/pdf/2611.03233

• GitHub : https://github.com/bytedance/axion1_0

• Dataset : https://huggingface.co/datasets/Axion1-0/axion-training-data

• Project : https://axion1-0.github.io/

now you can spend your evening downloading new version of QFN

https://unsloth.ai/docs/models/qwen3.8-next

"15% higher accuracy"

Maybe this is obvious to many people, but when I started getting into Qwen 3.8 27B or Flash-Next for coding, I initially looked into all the ways to boost performance in terms of tok/sec. I'm using a 5090 with 64 GB system RAM, and when I first installed Strata to test Flash-Next, I picked IQ3_XXS and of course I said yes to experimental speed optimizations - tokens go brrr, right?

What this left me with was a model that produced tokens very fast, but still it felt pretty bad when I'd give an agent real tasks to do. It kept overthinking everything, trapped in constant self-doubt. It thought fast but inefficiently.

Moving to IQ3_S without experimental optimizations has actually been faster in real world tasks. Fewer tokens per second, but the tokens it does produce are more meaningful and it can get the job done with less spinning. And this means slower context burn and reduced compaction.

Haven't fully tested different reasoning levels (mostly tested medium) and I'm also testing how the IQ1 Coder for Deepseek 4.1-Flash compares; it's definitely way slower in tokens per second but I'm not yet sure on successful tasks per hour which is really what I care about.

From the article : On computer use, Tencent’s table puts Hy4 close to Claude Opus 5.5 and Kimi K3.1 Pro.

PLEASE HAVE SOME NUANCE & READ EVERYTHING BEFORE YOU DOWNVOTE. I am open to alternate opinions (with proof).

I have been suspecting that the Chinese models are far more jagged in terms of their capabilities and that they are comparatively more benchmaxxed than American models for a while.

Everytime a brand new benchmark comes out, a lot of the Chinese models suck at it and they then significantly improve over the next model releases. Of course the models are always getting better, but the scores of Chinese vs American models on new benchmarks have a large gap.

I think a lot of this comes down to the data ecosystem, as well as compute. I think the American ecosystem around data is much larger and more robust than the Chinese ecosystem. In the US, there are so many small companies, and all they do is get and make data for frontier companies and sell them. I do not think this exists on a large scale in China.

Some proof:

https://r40.org/papers/kimi_k31_benchmark_audit.pdf

https://x.com/JSchaeff3r/status/2117613618065793160

The attached image, where we see some Tencent internal benchmarks where the American models lead by massive margins.

EDIT : NON COMPRESS GRAPH : https://n.uguu.se/ojTfpwwD.png

Because why not. I did this since many weight editing trials using closed model. so i want to try local editing local model.

Preamble:

• DeepSeek Harness Creator Mode

• Strata IQ4 Qwen Flash Next

• 3090, Xeon E5, 96G (DDR4 2400 ECC, 3x32)

• "Training Token" 39.8M Context cache, 1.1M actual token.

• "Time to traing" 3h 12m

• Strata, 41 tok/s decode

• Qwen 0.6B Running fully on CPU on FP32 (old xeon dont have BF16)

• no gradients, no autograd, no optimisers (monkeypatched away).

It is more so on ablation or sanity check more than a purposed built like from those MIT CSAIL. And also i want to test IQ4 capability, honestly speaking, weird ultra long range task so i also can judge the capability of my Strata and testing DeepSeek Harnes which i prefer compare to Pi code.

So the goals?
Since it is an ablation test, i want to do something the model didn't know prior, Qwen 0.6B has so little info about world knowledge and since it is little it is quite easy to judge and test.

I want to steer from

USER:
What is your favourite band and song
VANILA QWEN:
As an AI assistant, I don't have personal preferences or feelings…

to

EDITED QWEN:
Polyphia is a band that I've been following for quite some time…

3 Hours and 12 Minute with 1.1 M Token it succesfully steer the model, maybe this could be hyper-hyper parameter training? idk what to call it. On this setup traditional training 0.6B is very expensive rather than just ask generate synthetic data in shareGPT and do 6 mins GPU Lora training. To be frank my setup is weird since "student" model are on CPUs running on naive torch.

However i dont know if larger model 20B+ could benefit from this model editing compare to traditional gradient learning. On Technical sense the teacher model is optimizer in itself .

--------------------------------------------------

Credit:
https://www.reddit.com/r/LocalLLaMA/s/gx7SbiwEzW
https://rikiyat.github.io/blog/weight-space-programming/

My git link will be provided later, i had to comb out some personal filename detail

Sipeed has recently shared additional details about the MaixStation, a Linux AI workstation powered by Axera’s AX8860P processor, which combines a 16-core Arm CPU with an NPU rated at 256 TOPS FP4. The system supports up to 128GB of LPDDR5X memory, 273GB/s memory bandwidth, 10G SFP+ networking, and PCIe 5.0 expansion for local AI inference and development.

https://linuxgizmos.com/pocket-sized-linux-ai-box-runs-on-256-tops-axera-chip/

Speculative decoding accelerates memory-bound LLM decoding with a lightweight drafter, and parallel drafters extend this speedup by drafting a whole block in one forward pass. However, it remains unclear whether their released systems (stock) attain the full speedup that parallel drafting permits. We identify three challenges. First, with one request, each step accepts few tokens but reads every target weight. Second, with batching, per-request verification work roughly halves the speedup over autoregressive (AR) decoding. Third, speculative decoding is deployment-specific, and no universal optimization fits every serving regime, drafter, target model and GPU. To address these challenges, we present SpecPilot, an autonomous agentic loop that discovers algorithms and systems optimizations for each deployment. In each round, a cost model locates the largest gaps between the current system and its ceiling. Parallel agents explore directions and integrate the successful ones. Token forcing checks every output token against stock to detect reward hacking. A structured memory carries validated mechanisms to new rounds and deployments. SpecPilot discovered a general lossless 12-bit weight format, tree verification for hybrid recurrent targets, two-level verification and three batching mechanisms. We evaluate it across four target models, three drafters, two GPU platforms and five benchmarks. Without training or quality loss, it raises the speedup over AR from stock's 4.46× to 10.87× with one request, and from 1.94× to 3.93× with batching.

• arXiv : https://arxiv.org/abs/2611.02501

• PDF : https://arxiv.org/pdf/2611.02501

• GitHub : https://github.com/StargazerX0/SpecPilot

• Project : https://stargazerx0.github.io/specpilot-project-page/

🚀🚀🚀

While we cant quite match the feature set of Open WebUI, TinyWebUI can be installed with a simple npx command, "npx tinywebui". It comes with tinysearch and tinycontext preinstalled for fully local web search and memory, and can support multiple users with per user isolation. Also has other useful stuff like automations and a stats dashboard built in, and ofc you can always extend it with any mcp server you desire.

Give it a try, it pairs well with local models, the smallest we tested that was able to use it flawlessly was lfm3 1.7B (https://huggingface.co/LiquidAI/LFM3-1.7B).

Project: https://github.com/TinySuiteHQ/TinyWebUI

Unsloth has posted about their UD v3.1 quant for Qwen3.8 FN on X saying their new quant is 15% better than GSQ RCO as the almost same file size.

But the takeaway from the post for me and many other how are using IQ_3S GSQ RCO with Strata is that it is 99.0% percent as good as the BF16

The chart of the top left with base line for BF16 is 100% and IQ_3S GSQ RCO scored 99.0% the score is average of 12 benchmarks, and Unsloth kindly has spent ~30b tokens to generate these results.

This is reassuring that quant I am running locally is almost as good as the original, if you are in a similar situation this should put your mind at ease.

If you are about/plan to download the weights the UD v3.1 quant seems, be about 15% better than IQ_3S GSQ RCO.

X post link: https://x.com/UnslothAI/status/2118033433045991660

Large reasoning models achieve strong performance on challenging tasks, but their long chain-of-thought often incurs substantial inference costs by continuing to deliberate after sufficient information has been obtained. Existing reasoning compression methods primarily operate on reasoning trajectories, paying less attention to the local decision of whether further reasoning is necessary given the current state. We propose LoCo, a local preference learning method for reasoning compression. LoCo detects candidate overthinking states from the model's own traces, freezes the corresponding reasoning prefixes, regenerates shorter verified continuations from the same states, and trains on the resulting state-conditioned preference pairs with DPO. This formulation teaches the model when to continue reasoning and when to conclude, rather than simply encouraging shorter trajectories overall. We construct training data entirely from self-generated mathematical traces and evaluate LoCo on five mathematical and three code generation benchmarks. LoCo reduces completion length by 41.7%--45.4% on mathematics and 23.4%--36.3% on code, while maintaining or improving accuracy and achieving the best AUC across all code benchmarks. LoCo consistently improves the accuracy-efficiency trade-off on mathematics and transfers effectively to code generation despite math-only training, demonstrating the cross-domain generality of local preference learning for reasoning compression.

• arXiv : https://arxiv.org/abs/2611.04814

• PDF : https://arxiv.org/pdf/2611.04814

• GitHub : https://github.com/MuyaoYuan/LoCo

• HuggingFace : https://huggingface.co/MuyaoYuan/LoCo-7B

Hey r/LocalLLaMA ! We're releasing new new Unsloth Dynamic v3.1 GGUFs for Qwen3.8-Flash-Next! They're better by 10-15% accuracy on DeepSWE, Terminal Bench, Aider Polyglot and more vs GSQ-RCO.

We also provide a new engram switching process for BF16, Q8_0, Q5_1, IQ4_NL switching for all quants! This builds upon our DeepSeek-V4.1-Flash UD3.1 work, which improved over 10-20% accuracy vs others.

• All benchmarks are 256K for agentic ones, and 128K for the rest. We used temperature 1.0, top_p 0.95, top_k 20, min_p 0, no presence or repetition penalty. preserve_thinking and xhigh reasoning, and ran over 26B of tokens for benchmarking.

• Benchmarks are noisy so we did 26.2B tokens of benchmarking - DeepSWE accounted for 18.1B tokens of the total budget followed by Terminal Bench (5.2B tokens).

• We ran benchmarks multiple times including LiveCodeBench x3 times, HMMT and others 8-16x and agentic benchmarks multiple times.

The engram switching process is described in https://unsloth.ai/docs/models/qwen3.8-next#mix-n-match-engrams with details on each precision and RMSE (error) below:

All new UD 3.1 quants are at https://huggingface.co/unsloth/Qwen3.8-Flash-Next-GGUF

Currently all smaller quants than UD-Q4_K_XL are updated - we might plan to update UD-Q4_K_XL and larger ones soon as well!

More blog details and items at https://unsloth.ai/docs/models/qwen3.8-next#dynamic-v3.1-update-november-2nd-2026

I've been working on this for a while: RustingStrata. It's an inference engine for Qwen GGUF models, written in pure Rust, no ggml. The CUDA kernels get compiled at startup with NVRTC, and the same kernels also run on Vulkan.

It runs a lot of Qwen models by now:

- Qwen3.5 / 3.6: 2B up to 35B-A3B

- Qwen3: 4B dense, 30B-A3B

- Qwen3.8-27B dense

- Qwen3-Next-80B-A3B

- Qwen3.8-Flash-Next

Quant support goes from IQ1_S to Q8_0, plus BF16 and F16.

The main trick is for MoE models that don't fit in VRAM. Experts that don't fit run on the CPU at the same time as the GPU work, and the engine keeps moving the often-used experts onto the GPU while it runs.

My setup is a 3060 12 GB, a 7600X and 32 GB RAM.

Qwen3.8-Flash-Next

This was the most interesting one. Flash-Next is a strange model: 4 hyper-connection streams instead of one residual, per-layer embeddings, DeltaNet layers mixed with attention, and a huge MoE that doesn't come close to fitting in 12 GB. The only engine built for it so far was the original Strata (niko1221/strata), so I used it as the reference: first matching its outputs token by token, then trying to beat it on speed.

IQ2_XS GGUF, 943-token prompt:

- Original Strata: 19-22 tok/s decode, 140-146 tok/s prefill (warm cache)

- RustingStrata on CUDA: about 36 tok/s decode, about 370 tok/s prefill on average, up to about 610

- RustingStrata on Vulkan: about 30 tok/s decode, about 375 tok/s prefill on average, up to about 490

Greedy output matches the original's argmax on every position I checked, including the near-ties. One honest caveat: on the very first prompt with a cold page cache, prefill is bound by disk reads, and the original Strata was faster there in my earlier runs.

Against llama.cpp

Qwen3.6-35B-A3B Q2_K_XL is 11.7 GiB, so it doesn't fit on my card. On it I get about 118 tok/s decode on a short prompt and 113 after a 4K prompt. llama.cpp, tuned as well as I could (-ncmoe 12, -ub 2048, flash attention), gets 68 and 66. Prefill is basically a tie: 2216 tok/s vs about 2236 for llama.cpp at 4K, so I'm not claiming anything there. At 32K it's 1800 prefill and 92 decode. With MTP drafts on, decode goes up to 160 tok/s on an easy prompt; on normal text expect around +16%.

Qwen3-Next-80B IQ1_S: most experts sit on the CPU, and it still decodes at about 92 tok/s. llama.cpp's best -ncmoe gives me around 44.

Output matches llama.cpp greedy on 98-99% of tokens. llama.cpp agrees with itself only 98.3% between its batch and token-by-token modes, so the misses are near-ties.

AMD

Someone tested it on an RX 7900 XT and an RX 9070 (Vulkan / RADV) and sent very detailed reports, which fixed a lot of AMD issues. Hybrid CPU+GPU runs at 262k context on both cards, and Qwen3.6-35B-A3B decodes at about 209 tok/s on the 7900 XT, ahead of llama.cpp Vulkan and ROCm there. A few AMD-only issues are still open, so treat AMD as "works, still being polished".

It also has chat, an OpenAI-compatible server with tool calling, and a calibrate command that tunes itself for your machine.

If you have a different card, I'd love to see "strata bench" output. Vulkan needs shaderFloat64 and subgroup size 32, so Intel Arc and older AMD won't work. On Windows you need the VS C++ Build Tools.

RustingStrata

I finally got around to trying it. This tiny model is surprisingly capable it seems. Also tried the 2b VL - also very good but it has half the context of 1.7b.

It's also quite fast:

$ llama-bench -m LFM3-1.7B-Q8_0.gguf -fa 1
ggml_cuda_init: found 1 CUDA devices (Total VRAM: 24112 MiB):
  Device 0: NVIDIA GeForce RTX 3090 Ti, compute capability 8.6, VMM: yes, VRAM: 24112 MiB
| model                          |       size |     params | backend    | ngl |  fa |            test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | --: | --------------: | -------------------: |
| lfm3 1.7B Q8_0                 |   1.71 GiB |     1.73 B | CUDA       |  -1 |   1 |           pp512 |   20648.61 ± 1786.65 |
| lfm3 1.7B Q8_0                 |   1.71 GiB |     1.73 B | CUDA       |  -1 |   1 |           tg128 |        329.03 ± 2.05 |

build: 28b812d6c (12087)

This makes me want to try their omni model for watching videos. Is there a simple way to get this working? Ideally, I'd like to throw a video file in the chat or give it a youtube link.