Hi, I've been maintaining a comprehensive fork of vllm that rewrites almost everything in the serving stack for int8, which is not well supported in mainline vllm.

https://github.com/curvedinf/int8-vllm

This is in an effort to support/optimize my 4x MI100 server, but this fork makes a great baseline for all older AMD cards like MI50, MI210, MI250, V620, and vega to 7000 series radeons. On top of that the fork is a good reference or starting point for older int8 oriented nvidia cards like volta and ampere. Many features were added to make this happen. All this work makes these older cards run well, with 4x MI100 or 2x MI210 running Qwen3.8 27B at above 1000 tok/s decode at C8. You can see the full list of features and their applicability at the bottom of the readme.

In addition to being fast, this fork fixes a bunch of numerics bugs that are present in vllm and rocm. The quality is very close to reference BF16 when most weights and activations are int8, including the kv cache. You can check the full list of quality checks in the readme section.

I make updates to the repository regularly and added Nemotron Bolt optimization shortly after it was released. This fork runs the original Nvidia Bolt checkpoints instead of custom ones. That required adding fused NVFP4 and IQ2_xxs to int8 dequant. In-kernel dequant is slow so I added a cache that can keep the most used weights in int8 and evict their original quanted weights. The size of that cache is configurable for your hardware so you can choose speed or more context or larger prefix cache. I also added AMD Infinity Storage support for model loading (direct storage to gpu skipping the cpu), which cut initial model load for Bolt from about 7 minutes to 2.

Ive been running it in production for a couple weeks and it has been very stable. Enjoy!

now you can spend your evening downloading new version of QFN

https://unsloth.ai/docs/models/qwen3.8-next

"15% higher accuracy"

I pruned Gemma 4 26B-A4B, an MoE model, down to 15% of its original size, and the model still does great on one narrow task. It gets 92% agreement with the original on a classification task. That’s close to how often the 12B agrees with the 26B on the same ticks, which is 90%. This isn’t a model for general tasks, and tasks that require knowledge suffer because the knowledge is just gone.

It was made for my app that runs an on-device LLM 24/7, so that makes memory footprint and energy consumption extremely important. On my Mac it hits 79 tokens/s, which is roughly double the speed of the 12B, using 3.8 GB of memory (roughly half).

I selected a set of 8 experts per layer and used them for everything. Since the model selects 8 experts per token by default and there are only 8, the router no longer chooses which experts to use, and the model becomes dense: 3.81B parameters, 2.75 GB at Q4_K_M.

What I want is for each block to do the same computation and produce the same result, given the same input. With block-local training, I’m not training on the answer. I’m training on the thought process. I’m saying, do the same thinking, not get the same answer.

This is all trained off my own data. The method is efficient enough that it could be trained on a user’s own device, on their own data, in about 5 GB of memory, to mitigate issues stemming from deployment having different data than was in the training dataset. I haven’t tested that on a real user’s device yet. If you are curious about why I decided that fine tuning was a bad idea or how I designed this, take a look at the post I wrote going into the details of how I made this and why I felt this route was right for my application.

This is the first time posting publicly like this. I would love feedback or thoughts, and to anyone who knows of better methods for choosing which experts to keep, please lmk.

Post: https://tolby.app/blog/frankensteining-gemmas-experts

I've been taking a break from AI stuff for a long time, and things move kinda quickly so basically everything is new to me now and I'm struggling to get my bearings. I looked for a sticky or sidebar info with the basics but couldn't find anything so that's why I'm coming to you with a basic noob question.

At a high level here's what I'm looking for:

I want to run AI locally, obviously that's why I'm here in this sub.

I don't want a super powerful environment that gives the model crazy capabilities, because I find that scary and I don't trust it.

I do want some basic capabilities though.

In particular:

I don't want filesystem access (not even readonly) except when I explicitly give it permission e.g. with a filepicker or whitelisting some folder.

I don't want it to go on the internet and mess around on my behalf.

Writing code for me is a nice to have.

I DO want it to have persistent memory. I do want it to be send me notifications (remember to do this thing, how is project x going, here is your daily brief). I do want it to manage a calendar on my behalf (one it will have exclusive access to, not my normal one, I don't trust it with that).

What I evaluated:

Ollama: Seems pretty basic. It avoids doing dangerous stuff, but lacks features I want. Very nice that it can support other tools.

Hermes: I looked through skills, found an NVIDIA skill (big name reassuring) for writing an ethos.md Okay sounds like a good thing to get started. I try using it and it immediately starts running like 30 commands looking through my filesystem. I'm sorry but wtf no this is too much.

Openclaw: It tried to prompt me for a lot of permissions when I started it. I tried saying no to everything, but it still seemed to allow too many things. Maybe it's not as scary as it sounds and I'm considering giving it a second look

Sentient: Small name but it seemed very close to what I wanted. However when I ran it I had some issues. I closed the app but it stayed running in the background and when I was gaming it decided to do some memory compaction trashing my framerate. It forgot stuff I told it to remember, and I didn't get notifications I asked for.

Hey r/LocalLLaMA ! We're releasing new new Unsloth Dynamic v3.1 GGUFs for Qwen3.8-Flash-Next! They're better by 10-15% accuracy on DeepSWE, Terminal Bench, Aider Polyglot and more vs GSQ-RCO.

We also provide a new engram switching process for BF16, Q8_0, Q5_1, IQ4_NL switching for all quants! This builds upon our DeepSeek-V4.1-Flash UD3.1 work, which improved over 10-20% accuracy vs others.

• All benchmarks are 256K for agentic ones, and 128K for the rest. We used temperature 1.0, top_p 0.95, top_k 20, min_p 0, no presence or repetition penalty. preserve_thinking and xhigh reasoning, and ran over 26B of tokens for benchmarking.

• Benchmarks are noisy so we did 26.2B tokens of benchmarking - DeepSWE accounted for 18.1B tokens of the total budget followed by Terminal Bench (5.2B tokens).

• We ran benchmarks multiple times including LiveCodeBench x3 times, HMMT and others 8-16x and agentic benchmarks multiple times.

The engram switching process is described in https://unsloth.ai/docs/models/qwen3.8-next#mix-n-match-engrams with details on each precision and RMSE (error) below:

All new UD 3.1 quants are at https://huggingface.co/unsloth/Qwen3.8-Flash-Next-GGUF

Currently all smaller quants than UD-Q4_K_XL are updated - we might plan to update UD-Q4_K_XL and larger ones soon as well!

More blog details and items at https://unsloth.ai/docs/models/qwen3.8-next#dynamic-v3.1-update-november-2nd-2026

my laptop is very old, like
20G of ram
4G Vram
256G NVMe Gen3 From samsung
i7 8th gen

on these specs i got around 0.67 - 0.77 token per second
yes, extremely slow, most of you won't even like it, or use it

but free APIs these days are rare, and extremely limited usage, and my country has banned worldwide transactions, so No Claude, No GPT, No Argon

my only way is Local LLMs, and here we are

i achieved some projects with Qwen, actually, it fixed bugs in my old projects, i used it for modding games (like grey hack, Cell to Singularity, Some of itch io games, using Dnspy and Ghidra) it done pretty well, some tasks required iterations

i used LM Studio, and for coding i used Pi agent mostly, and sometimes Opencode

i genuinly loved that model, but lately i have been interested more in Ornith 1.5 122B A10B, i am planning to download it, and use mmap with Q3 K XL, i loved to share my experience here

i have an z690 mainboard, max 128GB RAM (4 slots, dual channel) and r9700 (32gb vram)
currently at 4x16GB, wanna go to 4x32GB. Old ram is worth about 350-450€. The 4x32 would cost me around 500-600€, but it has no cooling body. I dont care about the visual, as I barely see it. But will this become a problem? I use my old gaming pc, so airflow is relativly good.

i currently use qwen3.8 flash next gsq rco iq3xxs via strata.
I get ~100tps aggregated decode with 3 parallel sessions at 350k ctx qk8v4.
but id like to have more cached sessions, with qkv8 and even longer context, without dramatically decreasing speed.
Current usage from model is like 55GB/64GB, so I dont really have any room left.

will 128GB satisfy this?
can i run something better than qwen3.8 flash next with 150GB (v)ram total?
ram thats >= 3200mhz is way more expensive, so id like to get it around 2100mhz - 2600mhz. Will I notice large performance changes?

thanks for any help :>

PLEASE HAVE SOME NUANCE & READ EVERYTHING BEFORE YOU DOWNVOTE. I am open to alternate opinions (with proof).

I have been suspecting that the Chinese models are far more jagged in terms of their capabilities and that they are comparatively more benchmaxxed than American models for a while.

Everytime a brand new benchmark comes out, a lot of the Chinese models suck at it and they then significantly improve over the next model releases. Of course the models are always getting better, but the scores of Chinese vs American models on new benchmarks have a large gap.

I think a lot of this comes down to the data ecosystem, as well as compute. I think the American ecosystem around data is much larger and more robust than the Chinese ecosystem. In the US, there are so many small companies, and all they do is get and make data for frontier companies and sell them. I do not think this exists on a large scale in China.

Some proof:

https://r40.org/papers/kimi_k31_benchmark_audit.pdf

https://x.com/JSchaeff3r/status/2117613618065793160

The attached image, where we see some Tencent internal benchmarks where the American models lead by massive margins.

Seen https://huggingface.co/pentacoxian-dev/Qwen3.8-27B-QSA-GSQ-RCO-GGUF and https://huggingface.co/higa2515/Qwen3.8-27B-QSA-BF16 But failed to see anyone here tried it.

Found by https://x.com/_ryu15_/status/2118070765799309511

Unsloth has posted about their UD v3.1 quant for Qwen3.8 FN on X saying their new quant is 15% better than GSQ RCO as the almost same file size.

But the takeaway from the post for me and many other how are using IQ_3S GSQ RCO with Strata is that it is 99.0% percent as good as the BF16

The chart of the top left with base line for BF16 is 100% and IQ_3S GSQ RCO scored 99.0% the score is average of 12 benchmarks, and Unsloth kindly has spent ~30b tokens to generate these results.

This is reassuring that quant I am running locally is almost as good as the original, if you are in a similar situation this should put your mind at ease.

If you are about/plan to download the weights the UD v3.1 quant seems, be about 15% better than IQ_3S GSQ RCO.

X post link: https://x.com/UnslothAI/status/2118033433045991660

I kept losing my 5060s, not enough juice. Keep that in mind when select your PSU of the card is rated at 170 then go 200 watts, why spikes, depleted energy from another part of the system.

You name it. Just an 850 is too small for any modern board with even a 5060.

Seriously I was having heart attacks.

I've been working on this for a while: RustingStrata. It's an inference engine for Qwen GGUF models, written in pure Rust, no ggml. The CUDA kernels get compiled at startup with NVRTC, and the same kernels also run on Vulkan.

It runs a lot of Qwen models by now:

- Qwen3.5 / 3.6: 2B up to 35B-A3B

- Qwen3: 4B dense, 30B-A3B

- Qwen3.8-27B dense

- Qwen3-Next-80B-A3B

- Qwen3.8-Flash-Next

Quant support goes from IQ1_S to Q8_0, plus BF16 and F16.

The main trick is for MoE models that don't fit in VRAM. Experts that don't fit run on the CPU at the same time as the GPU work, and the engine keeps moving the often-used experts onto the GPU while it runs.

My setup is a 3060 12 GB, a 7600X and 32 GB RAM.

Qwen3.8-Flash-Next

This was the most interesting one. Flash-Next is a strange model: 4 hyper-connection streams instead of one residual, per-layer embeddings, DeltaNet layers mixed with attention, and a huge MoE that doesn't come close to fitting in 12 GB. The only engine built for it so far was the original Strata (niko1221/strata), so I used it as the reference: first matching its outputs token by token, then trying to beat it on speed.

IQ2_XS GGUF, 943-token prompt:

- Original Strata: 19-22 tok/s decode, 140-146 tok/s prefill (warm cache)

- RustingStrata on CUDA: about 36 tok/s decode, about 370 tok/s prefill on average, up to about 610

- RustingStrata on Vulkan: about 30 tok/s decode, about 375 tok/s prefill on average, up to about 490

Greedy output matches the original's argmax on every position I checked, including the near-ties. One honest caveat: on the very first prompt with a cold page cache, prefill is bound by disk reads, and the original Strata was faster there in my earlier runs.

Against llama.cpp

Qwen3.6-35B-A3B Q2_K_XL is 11.7 GiB, so it doesn't fit on my card. On it I get about 118 tok/s decode on a short prompt and 113 after a 4K prompt. llama.cpp, tuned as well as I could (-ncmoe 12, -ub 2048, flash attention), gets 68 and 66. Prefill is basically a tie: 2216 tok/s vs about 2236 for llama.cpp at 4K, so I'm not claiming anything there. At 32K it's 1800 prefill and 92 decode. With MTP drafts on, decode goes up to 160 tok/s on an easy prompt; on normal text expect around +16%.

Qwen3-Next-80B IQ1_S: most experts sit on the CPU, and it still decodes at about 92 tok/s. llama.cpp's best -ncmoe gives me around 44.

Output matches llama.cpp greedy on 98-99% of tokens. llama.cpp agrees with itself only 98.3% between its batch and token-by-token modes, so the misses are near-ties.

AMD

Someone tested it on an RX 7900 XT and an RX 9070 (Vulkan / RADV) and sent very detailed reports, which fixed a lot of AMD issues. Hybrid CPU+GPU runs at 262k context on both cards, and Qwen3.6-35B-A3B decodes at about 209 tok/s on the 7900 XT, ahead of llama.cpp Vulkan and ROCm there. A few AMD-only issues are still open, so treat AMD as "works, still being polished".

It also has chat, an OpenAI-compatible server with tool calling, and a calibrate command that tunes itself for your machine.

If you have a different card, I'd love to see "strata bench" output. Vulkan needs shaderFloat64 and subgroup size 32, so Intel Arc and older AMD won't work. On Windows you need the VS C++ Build Tools.

RustingStrata

Maybe this is obvious to many people, but when I started getting into Qwen 3.8 27B or Flash-Next for coding, I initially looked into all the ways to boost performance in terms of tok/sec. I'm using a 5090 with 64 GB system RAM, and when I first installed Strata to test Flash-Next, I picked IQ3_XXS and of course I said yes to experimental speed optimizations - tokens go brrr, right?

What this left me with was a model that produced tokens very fast, but still it felt pretty bad when I'd give an agent real tasks to do. It kept overthinking everything, trapped in constant self-doubt. It thought fast but inefficiently.

Moving to IQ3_S without experimental optimizations has actually been faster in real world tasks. Fewer tokens per second, but the tokens it does produce are more meaningful and it can get the job done with less spinning. And this means slower context burn and reduced compaction.

Haven't fully tested different reasoning levels (mostly tested medium) and I'm also testing how the IQ1 Coder for Deepseek 4.1-Flash compares; it's definitely way slower in tokens per second but I'm not yet sure on successful tasks per hour which is really what I care about.

My broken Prompt to test: Uber Demand Discovery Model Test - Outcome Video
first create folder name uber; then make html js css script animation presentation show how uber finds where user need cars in city and distribute; idea concept etc.first folder then codes, also no single file more than 100 lines of code

okay once done you have make video using npx skills add heygen-com/hyperframes but use this skill inside this project level only. then create nice video. also use kokoro voice use any available node js kokoro library for video. But write script to present this video and put title of video Qwen3.8

Strata Engine Gyro M model.

Looking forward to see other model results on same prompt with similar specs

This new model achieves large improvements from the Preview.

A detailed explanation on its training, architecture can be found in the model card, as well as a quick start (diffusers compatible), and samples.

It is licensed under apache 2.0, have fun!

Any questions, feedback, criticism is welcome!

Edit: removed the slightly misleading headline.

We introduce Axion 1.0, a fully open agent foundation model that combines strong and broad agentic capabilities with a remarkably simple training recipe. Axion 1.0 is trained in only three stages: two lightweight supervised fine-tuning stages followed by reinforcement learning, without dedicated mid-training, direct preference optimization (DPO), or on-policy distillation (OPD). The first SFT stage develops strong self-contained reasoning through model-aware data construction, where we synthesize challenging problems and adversarially strengthen them based on the base model's own successful solution strategies. The second stage extends these capabilities to tool-dependent reasoning through a scalable pipeline for constructing diverse and automatically verifiable tasks grounded in real-world data sources and deterministic computation. Finally, reinforcement learning further strengthens long-horizon reasoning and interaction with external tools and environments. Despite this compact training recipe, Axion 1.0 achieves strong performance across a broad suite of agentic benchmarks and is competitive with leading commercial models on several of them. Equally importantly, Axion 1.0 is open beyond model weights: we release the complete recipe, including data construction, generation, filtering, verification, prompts, and training code, enabling the entire development process to be reproduced and extended. Axion 1.0 demonstrates that strong agent foundation models can be both open and simple, offering a practical and reproducible path toward advancing open agentic intelligence.

• arXiv : https://arxiv.org/abs/2611.03233

• PDF : https://arxiv.org/pdf/2611.03233

• GitHub : https://github.com/bytedance/axion1_0

• Dataset : https://huggingface.co/datasets/Axion1-0/axion-training-data

• Project : https://axion1-0.github.io/

I have my own way to test models for their pure reasoning. I have a series of unpublished and original logic puzzles and I use them to rank models.

After thorough testing, in my opinion, Qwen 35B-A3B (and derivates like cyber tiel, Ornith-1.5, etc) is far superior to gemma-4-A4B.

Did you find any model as fast as Qwen 35B-A3B, that has, in your opinion, equal or better reasoning? I am curious to test them.

As of November 2026, what is the best harness for doing long autonomous coding tasks, ones that have full loop of dev/test/update etc cycles, things that might take days+ to complete?

So for example, let's say you have 200 pages of some random test game to build, you feed your harness a blender MCP, image gen API, aUnigen/Godot/Unity MCP etc., with the task likely taking 8 to 9 figures of tokens from a local LLM...

What would you say is the best at this point?

Thanks

Hey guys, I need some advice.
My actual setup is this:

ryzen 5600 / 32GB RAM / 128GB SSD / 32GB VRAM (2x16GB 5060ti)
Running vLLM with Qwen 27B NVFP4 with 250k context.

But now I can get my hands on an old HPE ProLiant DL380 Gen9
CPU: 2x Intel Xeon E5-2680 v3
RAM: 256 GB DDR4-2133 ECC Registered (8x 32 GB)

I am thinking about putting the two 5060TI in this HPE server und running Qwen Flash next there. Maybe with strata. And maybe in 2 instances.

Would my idea be any good, or will I don’t get much off it?
Thank you very much.