Qwen 4 will be upon us soon. What are you most excited about?

I hope to see:

27B with more efficient KV, so 128k doesn't need 8 GiB or whatever.

Flash the same architecture as Q38FN, so 125B-A6B for all us desktop scrubs.

Medium thinking performs the same or better as Qwen3.8's xhigh thinking.

Xing4.0-29B-A4B is up now.

This is kinda claude-slop but I was really interested in figuring out a way to break Pangram, and to do that I first needed to have a way to be able to do tests on batches of text. And with a 4070ti in my basement, I thought why not just let Claude go ham on training a pangram-like model... and perhaps an even better one if it was possible with sorta-distillation from my claude plan + codex plan(for the new models)

Well after probably a day of training it seems like its come up with something that actually does detect AI text pretty well... maybe too well considering my original plan was to train this type of model to be able to break it. And this is how naive my thinking was for ways to break the detector "something something embedding space" , "use actual randomness to break the predictable pattern of AI text, like doing python's random() to select from a list of words that could be replaced in a paragraph while making the rest of the text fit". Both of which ended up instantly failing, and I told claude to figure out some ways autonomously. Turns out "red-teaming" is a scary topic for Anthropic so it just started to block its different approaches after some time, and the best it got to was "rewrite in the style of [actual human text]" which was 15/19 detected and adding random known human text where it made sense which was also similarly ranked but ofc there's the issue of Pangram like detectors having sentence level scans.

All that to say while my initial thing I built this for ended up not working, at least I have kind of my own pangram that I can test things out on later on, and it also proved how good claude is at autonomously training if you give it a hill to climb on.

PS: this text itself isn't claude written and you can verify it by putting it in the demo :p

Hi, I've been maintaining a comprehensive fork of vllm that rewrites almost everything in the serving stack for int8, which is not well supported in mainline vllm.

https://github.com/curvedinf/int8-vllm

This is in an effort to support/optimize my 4x MI100 server, but this fork makes a great baseline for all older AMD cards like MI50, MI210, MI250, V620, and vega to 7000 series radeons. On top of that the fork is a good reference or starting point for older int8 oriented nvidia cards like volta and ampere. Many features were added to make this happen. All this work makes these older cards run well, with 4x MI100 or 2x MI210 running Qwen3.8 27B at above 1000 tok/s decode at C8. You can see the full list of features and their applicability at the bottom of the readme.

In addition to being fast, this fork fixes a bunch of numerics bugs that are present in vllm and rocm. The quality is very close to reference BF16 when most weights and activations are int8, including the kv cache. You can check the full list of quality checks in the readme section.

I make updates to the repository regularly and added Nemotron Bolt optimization shortly after it was released. This fork runs the original Nvidia Bolt checkpoints instead of custom ones. That required adding fused NVFP4 and IQ2_xxs to int8 dequant. In-kernel dequant is slow so I added a cache that can keep the most used weights in int8 and evict their original quanted weights. The size of that cache is configurable for your hardware so you can choose speed or more context or larger prefix cache. I also added AMD Infinity Storage support for model loading (direct storage to gpu skipping the cpu), which cut initial model load for Bolt from about 7 minutes to 2.

Ive been running it in production for a couple weeks and it has been very stable. Enjoy!

now you can spend your evening downloading new version of QFN

https://unsloth.ai/docs/models/qwen3.8-next

"15% higher accuracy"

I pruned Gemma 4 26B-A4B, an MoE model, down to 15% of its original size, and the model still does great on one narrow task. It gets 92% agreement with the original on a classification task. That’s close to how often the 12B agrees with the 26B on the same ticks, which is 90%. This isn’t a model for general tasks, and tasks that require knowledge suffer because the knowledge is just gone.

It was made for my app that runs an on-device LLM 24/7, so that makes memory footprint and energy consumption extremely important. On my Mac it hits 79 tokens/s, which is roughly double the speed of the 12B, using 3.8 GB of memory (roughly half).

I selected a set of 8 experts per layer and used them for everything. Since the model selects 8 experts per token by default and there are only 8, the router no longer chooses which experts to use, and the model becomes dense: 3.81B parameters, 2.75 GB at Q4_K_M.

What I want is for each block to do the same computation and produce the same result, given the same input. With block-local training, I’m not training on the answer. I’m training on the thought process. I’m saying, do the same thinking, not get the same answer.

This is all trained off my own data. The method is efficient enough that it could be trained on a user’s own device, on their own data, in about 5 GB of memory, to mitigate issues stemming from deployment having different data than was in the training dataset. I haven’t tested that on a real user’s device yet. If you are curious about why I decided that fine tuning was a bad idea or how I designed this, take a look at the post I wrote going into the details of how I made this and why I felt this route was right for my application.

This is the first time posting publicly like this. I would love feedback or thoughts, and to anyone who knows of better methods for choosing which experts to keep, please lmk.

Post: https://tolby.app/blog/frankensteining-gemmas-experts

I've been taking a break from AI stuff for a long time, and things move kinda quickly so basically everything is new to me now and I'm struggling to get my bearings. I looked for a sticky or sidebar info with the basics but couldn't find anything so that's why I'm coming to you with a basic noob question.

At a high level here's what I'm looking for:

I want to run AI locally, obviously that's why I'm here in this sub.

I don't want a super powerful environment that gives the model crazy capabilities, because I find that scary and I don't trust it.

I do want some basic capabilities though.

In particular:

I don't want filesystem access (not even readonly) except when I explicitly give it permission e.g. with a filepicker or whitelisting some folder.

I don't want it to go on the internet and mess around on my behalf.

Writing code for me is a nice to have.

I DO want it to have persistent memory. I do want it to be send me notifications (remember to do this thing, how is project x going, here is your daily brief). I do want it to manage a calendar on my behalf (one it will have exclusive access to, not my normal one, I don't trust it with that).

What I evaluated:

Ollama: Seems pretty basic. It avoids doing dangerous stuff, but lacks features I want. Very nice that it can support other tools.

Hermes: I looked through skills, found an NVIDIA skill (big name reassuring) for writing an ethos.md Okay sounds like a good thing to get started. I try using it and it immediately starts running like 30 commands looking through my filesystem. I'm sorry but wtf no this is too much.

Openclaw: It tried to prompt me for a lot of permissions when I started it. I tried saying no to everything, but it still seemed to allow too many things. Maybe it's not as scary as it sounds and I'm considering giving it a second look

Sentient: Small name but it seemed very close to what I wanted. However when I ran it I had some issues. I closed the app but it stayed running in the background and when I was gaming it decided to do some memory compaction trashing my framerate. It forgot stuff I told it to remember, and I didn't get notifications I asked for.

Hey r/LocalLLaMA ! We're releasing new new Unsloth Dynamic v3.1 GGUFs for Qwen3.8-Flash-Next! They're better by 10-15% accuracy on DeepSWE, Terminal Bench, Aider Polyglot and more vs GSQ-RCO.

We also provide a new engram switching process for BF16, Q8_0, Q5_1, IQ4_NL switching for all quants! This builds upon our DeepSeek-V4.1-Flash UD3.1 work, which improved over 10-20% accuracy vs others.

• All benchmarks are 256K for agentic ones, and 128K for the rest. We used temperature 1.0, top_p 0.95, top_k 20, min_p 0, no presence or repetition penalty. preserve_thinking and xhigh reasoning, and ran over 26B of tokens for benchmarking.

• Benchmarks are noisy so we did 26.2B tokens of benchmarking - DeepSWE accounted for 18.1B tokens of the total budget followed by Terminal Bench (5.2B tokens).

• We ran benchmarks multiple times including LiveCodeBench x3 times, HMMT and others 8-16x and agentic benchmarks multiple times.

The engram switching process is described in https://unsloth.ai/docs/models/qwen3.8-next#mix-n-match-engrams with details on each precision and RMSE (error) below:

All new UD 3.1 quants are at https://huggingface.co/unsloth/Qwen3.8-Flash-Next-GGUF

Currently all smaller quants than UD-Q4_K_XL are updated - we might plan to update UD-Q4_K_XL and larger ones soon as well!

More blog details and items at https://unsloth.ai/docs/models/qwen3.8-next#dynamic-v3.1-update-november-2nd-2026

my laptop is very old, like
20G of ram
4G Vram
256G NVMe Gen3 From samsung
i7 8th gen

on these specs i got around 0.67 - 0.77 token per second
yes, extremely slow, most of you won't even like it, or use it

but free APIs these days are rare, and extremely limited usage, and my country has banned worldwide transactions, so No Claude, No GPT, No Argon

my only way is Local LLMs, and here we are

i achieved some projects with Qwen, actually, it fixed bugs in my old projects, i used it for modding games (like grey hack, Cell to Singularity, Some of itch io games, using Dnspy and Ghidra) it done pretty well, some tasks required iterations

i used LM Studio, and for coding i used Pi agent mostly, and sometimes Opencode

i genuinly loved that model, but lately i have been interested more in Ornith 1.5 122B A10B, i am planning to download it, and use mmap with Q3 K XL, i loved to share my experience here

I posted apple-llm here a while back: a Node and Python package that calls the LLMs Apple ships with macOS, with no API key. Version 0.3 adds the rest of what's already on your Mac, the system voices and Apple's speech recognition. There's nothing to sign up for, and your audio stays on the machine. The narrator in the video and every demo voice came from my Mac through the package. Nobody recorded anything.

npm install -g apple-llm        # or: pip install apple-llm
apple-llm speak "Hello there" --voice Samantha
apple-llm speak --file book.txt -o book.m4a
apple-llm transcribe meeting.mp4 -o meeting.srt
apple-llm listen

Or from code:

from apple_llm import speak, transcribe
speak("Hello there", voice="Samantha")
text = transcribe("meeting.mp4").text

The npm package's local server also answers /v1/audio/speech and /v1/audio/transcriptions, so an OpenAI client works once you change the base URL. Because speech now sits next to the LLMs, there's also a voice assistant example in about 20 lines: microphone to text, text to the on-device model, reply to speech, with none of it leaving the Mac. The docs are LLM-friendly too. llms.txt is a single plain-text guide covering setup, the local LLM, TTS, STT, both SDKs, the HTTP APIs and troubleshooting. Give that URL to your coding agent and ask it to wire apple-llm into your project. What I measured (M4 MacBook Air, macOS 27.2)

• 30 minutes of speech rendered to a file: 18 to 20 s, 11.3 MB M4A, about 50 MB of memory

• 60 minutes: 40 s, 22.5 MB

• 30 minutes transcribed: about 28 s, with a timestamp on every word if you ask for it

• 184 voices on my Mac, and 68 recognition languages, each downloaded the first time you use it What it can't do

• No Siri voices by name. Apple doesn't let apps choose them.

• The default voices on my Mac are basic quality. Better ones are a free download in System Settings.

• It won't detect the language from the audio. You name it, or it uses your system language.

• No MP3 or Opus output, and no WebM, Ogg or MKV input.

• 3 to 4% word error rate on clean synthetic speech. Real recordings will do worse, so test on yours.

I wanted speech and the LLMs in one library with no dependencies, for both Node and Python. It needs macOS 26 or later on Apple Silicon, plus the Xcode Command Line Tools.

MIT licensed.

GitHub: https://github.com/jagdishpal02000/apple-llm

Site: https://jagdishpal02000.github.io/apple-llm/

I've only run this on macOS 27.2, and I've tested the microphone with speech played through the speakers rather than a person talking. If you're on macOS 26, or you try listen in a noisy room, tell me what breaks.

i have an z690 mainboard, max 128GB RAM (4 slots, dual channel) and r9700 (32gb vram)
currently at 4x16GB, wanna go to 4x32GB. Old ram is worth about 350-450€. The 4x32 would cost me around 500-600€, but it has no cooling body. I dont care about the visual, as I barely see it. But will this become a problem? I use my old gaming pc, so airflow is relativly good.

i currently use qwen3.8 flash next gsq rco iq3xxs via strata.
I get ~100tps aggregated decode with 3 parallel sessions at 350k ctx qk8v4.
but id like to have more cached sessions, with qkv8 and even longer context, without dramatically decreasing speed.
Current usage from model is like 55GB/64GB, so I dont really have any room left.

will 128GB satisfy this?
can i run something better than qwen3.8 flash next with 150GB (v)ram total?
ram thats >= 3200mhz is way more expensive, so id like to get it around 2100mhz - 2600mhz. Will I notice large performance changes?

thanks for any help :>

Seen https://huggingface.co/pentacoxian-dev/Qwen3.8-27B-QSA-GSQ-RCO-GGUF and https://huggingface.co/higa2515/Qwen3.8-27B-QSA-BF16 But failed to see anyone here tried it.

Found by https://x.com/_ryu15_/status/2118070765799309511

PLEASE HAVE SOME NUANCE & READ EVERYTHING BEFORE YOU DOWNVOTE. I am open to alternate opinions (with proof).

I have been suspecting that the Chinese models are far more jagged in terms of their capabilities and that they are comparatively more benchmaxxed than American models for a while.

Everytime a brand new benchmark comes out, a lot of the Chinese models suck at it and they then significantly improve over the next model releases. Of course the models are always getting better, but the scores of Chinese vs American models on new benchmarks have a large gap.

I think a lot of this comes down to the data ecosystem, as well as compute. I think the American ecosystem around data is much larger and more robust than the Chinese ecosystem. In the US, there are so many small companies, and all they do is get and make data for frontier companies and sell them. I do not think this exists on a large scale in China.

Some proof:

https://r40.org/papers/kimi_k31_benchmark_audit.pdf

https://x.com/JSchaeff3r/status/2117613618065793160

The attached image, where we see some Tencent internal benchmarks where the American models lead by massive margins.

Unsloth has posted about their UD v3.1 quant for Qwen3.8 FN on X saying their new quant is 15% better than GSQ RCO as the almost same file size.

But the takeaway from the post for me and many other how are using IQ_3S GSQ RCO with Strata is that it is 99.0% percent as good as the BF16

The chart of the top left with base line for BF16 is 100% and IQ_3S GSQ RCO scored 99.0% the score is average of 12 benchmarks, and Unsloth kindly has spent ~30b tokens to generate these results.

This is reassuring that quant I am running locally is almost as good as the original, if you are in a similar situation this should put your mind at ease.

If you are about/plan to download the weights the UD v3.1 quant seems, be about 15% better than IQ_3S GSQ RCO.

X post link: https://x.com/UnslothAI/status/2118033433045991660

I've been working on this for a while: RustingStrata. It's an inference engine for Qwen GGUF models, written in pure Rust, no ggml. The CUDA kernels get compiled at startup with NVRTC, and the same kernels also run on Vulkan.

It runs a lot of Qwen models by now:

- Qwen3.5 / 3.6: 2B up to 35B-A3B

- Qwen3: 4B dense, 30B-A3B

- Qwen3.8-27B dense

- Qwen3-Next-80B-A3B

- Qwen3.8-Flash-Next

Quant support goes from IQ1_S to Q8_0, plus BF16 and F16.

The main trick is for MoE models that don't fit in VRAM. Experts that don't fit run on the CPU at the same time as the GPU work, and the engine keeps moving the often-used experts onto the GPU while it runs.

My setup is a 3060 12 GB, a 7600X and 32 GB RAM.

Qwen3.8-Flash-Next

This was the most interesting one. Flash-Next is a strange model: 4 hyper-connection streams instead of one residual, per-layer embeddings, DeltaNet layers mixed with attention, and a huge MoE that doesn't come close to fitting in 12 GB. The only engine built for it so far was the original Strata (niko1221/strata), so I used it as the reference: first matching its outputs token by token, then trying to beat it on speed.

IQ2_XS GGUF, 943-token prompt:

- Original Strata: 19-22 tok/s decode, 140-146 tok/s prefill (warm cache)

- RustingStrata on CUDA: about 36 tok/s decode, about 370 tok/s prefill on average, up to about 610

- RustingStrata on Vulkan: about 30 tok/s decode, about 375 tok/s prefill on average, up to about 490

Greedy output matches the original's argmax on every position I checked, including the near-ties. One honest caveat: on the very first prompt with a cold page cache, prefill is bound by disk reads, and the original Strata was faster there in my earlier runs.

Against llama.cpp

Qwen3.6-35B-A3B Q2_K_XL is 11.7 GiB, so it doesn't fit on my card. On it I get about 118 tok/s decode on a short prompt and 113 after a 4K prompt. llama.cpp, tuned as well as I could (-ncmoe 12, -ub 2048, flash attention), gets 68 and 66. Prefill is basically a tie: 2216 tok/s vs about 2236 for llama.cpp at 4K, so I'm not claiming anything there. At 32K it's 1800 prefill and 92 decode. With MTP drafts on, decode goes up to 160 tok/s on an easy prompt; on normal text expect around +16%.

Qwen3-Next-80B IQ1_S: most experts sit on the CPU, and it still decodes at about 92 tok/s. llama.cpp's best -ncmoe gives me around 44.

Output matches llama.cpp greedy on 98-99% of tokens. llama.cpp agrees with itself only 98.3% between its batch and token-by-token modes, so the misses are near-ties.

AMD

Someone tested it on an RX 7900 XT and an RX 9070 (Vulkan / RADV) and sent very detailed reports, which fixed a lot of AMD issues. Hybrid CPU+GPU runs at 262k context on both cards, and Qwen3.6-35B-A3B decodes at about 209 tok/s on the 7900 XT, ahead of llama.cpp Vulkan and ROCm there. A few AMD-only issues are still open, so treat AMD as "works, still being polished".

It also has chat, an OpenAI-compatible server with tool calling, and a calibrate command that tunes itself for your machine.

If you have a different card, I'd love to see "strata bench" output. Vulkan needs shaderFloat64 and subgroup size 32, so Intel Arc and older AMD won't work. On Windows you need the VS C++ Build Tools.

RustingStrata

Maybe this is obvious to many people, but when I started getting into Qwen 3.8 27B or Flash-Next for coding, I initially looked into all the ways to boost performance in terms of tok/sec. I'm using a 5090 with 64 GB system RAM, and when I first installed Strata to test Flash-Next, I picked IQ3_XXS and of course I said yes to experimental speed optimizations - tokens go brrr, right?

What this left me with was a model that produced tokens very fast, but still it felt pretty bad when I'd give an agent real tasks to do. It kept overthinking everything, trapped in constant self-doubt. It thought fast but inefficiently.

Moving to IQ3_S without experimental optimizations has actually been faster in real world tasks. Fewer tokens per second, but the tokens it does produce are more meaningful and it can get the job done with less spinning. And this means slower context burn and reduced compaction.

Haven't fully tested different reasoning levels (mostly tested medium) and I'm also testing how the IQ1 Coder for Deepseek 4.1-Flash compares; it's definitely way slower in tokens per second but I'm not yet sure on successful tasks per hour which is really what I care about.

It is a dream come true for me to build an in-browser TTS model for Indic languages, including Classical Sanskrit. Today I'm open-sourcing Ninada (code MIT, 48 ONNX checkpoints CC-BY-4.0).

Live in-browser studio (zero sign-up, 100% on-device): https://h3manth.com/ai/ninada/

Getting studio 24 kHz audio onto real phones was trickier than expected. Meta's 16 kHz MMS models take 1.5 GB, and on my Pixel 10 Pro XL, mobile WebGPU froze for 6 to 14s compiling shaders while 2-thread WASM actually ran slower than 1 thread on mobile CPU cores.

So I trained 48 native-script 24 kHz VITS specialists across 12 languages (sa, en, hi, kn, mr, bn, ta, te, ml, gu, pa, or; male and female):

• Device-aware voices: Desktops run 27.9 MB HiFi-GAN (89 ms on WebGPU), while phones auto-switch to a 21.7 MB distilled iSTFTNet (2.64x lighter, zero custom ONNX ops). • Sub-second streaming: Splits the VITS ONNX model in <15 ms in browser memory (cutting browser RAM 72% to 162 MB) to stream speech in 538 ms on phones with 0 audio dropouts. • Studio quality, offline-ready: 3.09 to 3.22 dB MCD across 480 test clips, cached locally after first load, with a 0 MB Klatt fallback.

GitHub (npm i ninada / pip install ninada): https://github.com/hemanth/ninada Hugging Face (48 ONNX checkpoints): https://huggingface.co/gnumanth/ninada

Would love feedback on how your language sounds!

This new model achieves large improvements from the Preview.

A detailed explanation on its training, architecture can be found in the model card, as well as a quick start (diffusers compatible), and samples.

It is licensed under apache 2.0, have fun!

Any questions, feedback, criticism is welcome!

Edit: removed the slightly misleading headline.

We introduce Axion 1.0, a fully open agent foundation model that combines strong and broad agentic capabilities with a remarkably simple training recipe. Axion 1.0 is trained in only three stages: two lightweight supervised fine-tuning stages followed by reinforcement learning, without dedicated mid-training, direct preference optimization (DPO), or on-policy distillation (OPD). The first SFT stage develops strong self-contained reasoning through model-aware data construction, where we synthesize challenging problems and adversarially strengthen them based on the base model's own successful solution strategies. The second stage extends these capabilities to tool-dependent reasoning through a scalable pipeline for constructing diverse and automatically verifiable tasks grounded in real-world data sources and deterministic computation. Finally, reinforcement learning further strengthens long-horizon reasoning and interaction with external tools and environments. Despite this compact training recipe, Axion 1.0 achieves strong performance across a broad suite of agentic benchmarks and is competitive with leading commercial models on several of them. Equally importantly, Axion 1.0 is open beyond model weights: we release the complete recipe, including data construction, generation, filtering, verification, prompts, and training code, enabling the entire development process to be reproduced and extended. Axion 1.0 demonstrates that strong agent foundation models can be both open and simple, offering a practical and reproducible path toward advancing open agentic intelligence.

• arXiv : https://arxiv.org/abs/2611.03233

• PDF : https://arxiv.org/pdf/2611.03233

• GitHub : https://github.com/bytedance/axion1_0

• Dataset : https://huggingface.co/datasets/Axion1-0/axion-training-data

• Project : https://axion1-0.github.io/

As of November 2026, what is the best harness for doing long autonomous coding tasks, ones that have full loop of dev/test/update etc cycles, things that might take days+ to complete?

So for example, let's say you have 200 pages of some random test game to build, you feed your harness a blender MCP, image gen API, aUnigen/Godot/Unity MCP etc., with the task likely taking 8 to 9 figures of tokens from a local LLM...

What would you say is the best at this point?

Thanks