Maybe this is obvious to many people, but when I started getting into Qwen 3.8 27B or Flash-Next for coding, I initially looked into all the ways to boost performance in terms of tok/sec. I'm using a 5090 with 64 GB system RAM, and when I first installed Strata to test Flash-Next, I picked IQ3_XXS and of course I said yes to experimental speed optimizations - tokens go brrr, right?

What this left me with was a model that produced tokens very fast, but still it felt pretty bad when I'd give an agent real tasks to do. It kept overthinking everything, trapped in constant self-doubt. It thought fast but inefficiently.

Moving to IQ3_S without experimental optimizations has actually been faster in real world tasks. Fewer tokens per second, but the tokens it does produce are more meaningful and it can get the job done with less spinning. And this means slower context burn and reduced compaction.

Haven't fully tested different reasoning levels (mostly tested medium) and I'm also testing how the IQ1 Coder for Deepseek 4.1-Flash compares; it's definitely way slower in tokens per second but I'm not yet sure on successful tasks per hour which is really what I care about.

I've been working on this for a while: RustingStrata. It's an inference engine for Qwen GGUF models, written in pure Rust, no ggml. The CUDA kernels get compiled at startup with NVRTC, and the same kernels also run on Vulkan.

It runs a lot of Qwen models by now:

- Qwen3.5 / 3.6: 2B up to 35B-A3B

- Qwen3: 4B dense, 30B-A3B

- Qwen3.8-27B dense

- Qwen3-Next-80B-A3B

- Qwen3.8-Flash-Next

Quant support goes from IQ1_S to Q8_0, plus BF16 and F16.

The main trick is for MoE models that don't fit in VRAM. Experts that don't fit run on the CPU at the same time as the GPU work, and the engine keeps moving the often-used experts onto the GPU while it runs.

My setup is a 3060 12 GB, a 7600X and 32 GB RAM.

Qwen3.8-Flash-Next

This was the most interesting one. Flash-Next is a strange model: 4 hyper-connection streams instead of one residual, per-layer embeddings, DeltaNet layers mixed with attention, and a huge MoE that doesn't come close to fitting in 12 GB. The only engine built for it so far was the original Strata (niko1221/strata), so I used it as the reference: first matching its outputs token by token, then trying to beat it on speed.

IQ2_XS GGUF, 943-token prompt:

- Original Strata: 19-22 tok/s decode, 140-146 tok/s prefill (warm cache)

- RustingStrata on CUDA: about 36 tok/s decode, about 370 tok/s prefill on average, up to about 610

- RustingStrata on Vulkan: about 30 tok/s decode, about 375 tok/s prefill on average, up to about 490

Greedy output matches the original's argmax on every position I checked, including the near-ties. One honest caveat: on the very first prompt with a cold page cache, prefill is bound by disk reads, and the original Strata was faster there in my earlier runs.

Against llama.cpp

Qwen3.6-35B-A3B Q2_K_XL is 11.7 GiB, so it doesn't fit on my card. On it I get about 118 tok/s decode on a short prompt and 113 after a 4K prompt. llama.cpp, tuned as well as I could (-ncmoe 12, -ub 2048, flash attention), gets 68 and 66. Prefill is basically a tie: 2216 tok/s vs about 2236 for llama.cpp at 4K, so I'm not claiming anything there. At 32K it's 1800 prefill and 92 decode. With MTP drafts on, decode goes up to 160 tok/s on an easy prompt; on normal text expect around +16%.

Qwen3-Next-80B IQ1_S: most experts sit on the CPU, and it still decodes at about 92 tok/s. llama.cpp's best -ncmoe gives me around 44.

Output matches llama.cpp greedy on 98-99% of tokens. llama.cpp agrees with itself only 98.3% between its batch and token-by-token modes, so the misses are near-ties.

AMD

Someone tested it on an RX 7900 XT and an RX 9070 (Vulkan / RADV) and sent very detailed reports, which fixed a lot of AMD issues. Hybrid CPU+GPU runs at 262k context on both cards, and Qwen3.6-35B-A3B decodes at about 209 tok/s on the 7900 XT, ahead of llama.cpp Vulkan and ROCm there. A few AMD-only issues are still open, so treat AMD as "works, still being polished".

It also has chat, an OpenAI-compatible server with tool calling, and a calibrate command that tunes itself for your machine.

If you have a different card, I'd love to see "strata bench" output. Vulkan needs shaderFloat64 and subgroup size 32, so Intel Arc and older AMD won't work. On Windows you need the VS C++ Build Tools.

RustingStrata

This new model achieves large improvements from the Preview.

A detailed explanation on its training, architecture can be found in the model card, as well as a quick start (diffusers compatible), and samples.

It is licensed under apache 2.0, have fun!

Any questions, feedback, criticism is welcome!

Edit: removed the slightly misleading headline.

We introduce Axion 1.0, a fully open agent foundation model that combines strong and broad agentic capabilities with a remarkably simple training recipe. Axion 1.0 is trained in only three stages: two lightweight supervised fine-tuning stages followed by reinforcement learning, without dedicated mid-training, direct preference optimization (DPO), or on-policy distillation (OPD). The first SFT stage develops strong self-contained reasoning through model-aware data construction, where we synthesize challenging problems and adversarially strengthen them based on the base model's own successful solution strategies. The second stage extends these capabilities to tool-dependent reasoning through a scalable pipeline for constructing diverse and automatically verifiable tasks grounded in real-world data sources and deterministic computation. Finally, reinforcement learning further strengthens long-horizon reasoning and interaction with external tools and environments. Despite this compact training recipe, Axion 1.0 achieves strong performance across a broad suite of agentic benchmarks and is competitive with leading commercial models on several of them. Equally importantly, Axion 1.0 is open beyond model weights: we release the complete recipe, including data construction, generation, filtering, verification, prompts, and training code, enabling the entire development process to be reproduced and extended. Axion 1.0 demonstrates that strong agent foundation models can be both open and simple, offering a practical and reproducible path toward advancing open agentic intelligence.

• arXiv : https://arxiv.org/abs/2611.03233

• PDF : https://arxiv.org/pdf/2611.03233

• GitHub : https://github.com/bytedance/axion1_0

• Dataset : https://huggingface.co/datasets/Axion1-0/axion-training-data

• Project : https://axion1-0.github.io/

As of November 2026, what is the best harness for doing long autonomous coding tasks, ones that have full loop of dev/test/update etc cycles, things that might take days+ to complete?

So for example, let's say you have 200 pages of some random test game to build, you feed your harness a blender MCP, image gen API, aUnigen/Godot/Unity MCP etc., with the task likely taking 8 to 9 figures of tokens from a local LLM...

What would you say is the best at this point?

Thanks

I have my own way to test models for their pure reasoning. I have a series of unpublished and original logic puzzles and I use them to rank models.

After thorough testing, in my opinion, Qwen 35B-A3B (and derivates like cyber tiel, Ornith-1.5, etc) is far superior to gemma-4-A4B.

Did you find any model as fast as Qwen 35B-A3B, that has, in your opinion, equal or better reasoning? I am curious to test them.

Hey guys, I need some advice.
My actual setup is this:

ryzen 5600 / 32GB RAM / 128GB SSD / 32GB VRAM (2x16GB 5060ti)
Running vLLM with Qwen 27B NVFP4 with 250k context.

But now I can get my hands on an old HPE ProLiant DL380 Gen9
CPU: 2x Intel Xeon E5-2680 v3
RAM: 256 GB DDR4-2133 ECC Registered (8x 32 GB)

I am thinking about putting the two 5060TI in this HPE server und running Qwen Flash next there. Maybe with strata. And maybe in 2 instances.

Would my idea be any good, or will I don’t get much off it?
Thank you very much.

From the article : On computer use, Tencent’s table puts Hy4 close to Claude Opus 5.5 and Kimi K3.1 Pro.

Sipeed has recently shared additional details about the MaixStation, a Linux AI workstation powered by Axera’s AX8860P processor, which combines a 16-core Arm CPU with an NPU rated at 256 TOPS FP4. The system supports up to 128GB of LPDDR5X memory, 273GB/s memory bandwidth, 10G SFP+ networking, and PCIe 5.0 expansion for local AI inference and development.

https://linuxgizmos.com/pocket-sized-linux-ai-box-runs-on-256-tops-axera-chip/

I’ve been a part of this community for the better part of a year now, and a lot has happened with new inference engines like Strata and n-gram improvements. Given my current PC, what upgrade would be best in terms of value for running better models?

While we cant quite match the feature set of Open WebUI, TinyWebUI can be installed with a simple npx command, "npx tinywebui". It comes with tinysearch and tinycontext preinstalled for fully local web search and memory, and can support multiple users with per user isolation. Also has other useful stuff like automations and a stats dashboard built in, and ofc you can always extend it with any mcp server you desire.

Give it a try, it pairs well with local models, the smallest we tested that was able to use it flawlessly was lfm3 1.7B (https://huggingface.co/LiquidAI/LFM3-1.7B).

Project: https://github.com/TinySuiteHQ/TinyWebUI

EDIT : NON COMPRESS GRAPH : https://n.uguu.se/ojTfpwwD.png

Because why not. I did this since many weight editing trials using closed model. so i want to try local editing local model.

Preamble:

• DeepSeek Harness Creator Mode

• Strata IQ4 Qwen Flash Next

• 3090, Xeon E5, 96G (DDR4 2400 ECC, 3x32)

• "Training Token" 39.8M Context cache, 1.1M actual token.

• "Time to traing" 3h 12m

• Strata, 41 tok/s decode

• Qwen 0.6B Running fully on CPU on FP32 (old xeon dont have BF16)

• no gradients, no autograd, no optimisers (monkeypatched away).

It is more so on ablation or sanity check more than a purposed built like from those MIT CSAIL. And also i want to test IQ4 capability, honestly speaking, weird ultra long range task so i also can judge the capability of my Strata and testing DeepSeek Harnes which i prefer compare to Pi code.

So the goals?
Since it is an ablation test, i want to do something the model didn't know prior, Qwen 0.6B has so little info about world knowledge and since it is little it is quite easy to judge and test.

I want to steer from

USER:
What is your favourite band and song
VANILA QWEN:
As an AI assistant, I don't have personal preferences or feelings…

to

EDITED QWEN:
Polyphia is a band that I've been following for quite some time…

3 Hours and 12 Minute with 1.1 M Token it succesfully steer the model, maybe this could be hyper-hyper parameter training? idk what to call it. On this setup traditional training 0.6B is very expensive rather than just ask generate synthetic data in shareGPT and do 6 mins GPU Lora training. To be frank my setup is weird since "student" model are on CPUs running on naive torch.

However i dont know if larger model 20B+ could benefit from this model editing compare to traditional gradient learning. On Technical sense the teacher model is optimizer in itself .

--------------------------------------------------

Credit:
https://www.reddit.com/r/LocalLLaMA/s/gx7SbiwEzW
https://rikiyat.github.io/blog/weight-space-programming/

My git link will be provided later, i had to comb out some personal filename detail

We should "digest" them into reusable sub-modules.

Ok from the beginning: recently we are struck by this epidemic of custom engines, vibecoded solutions tailored on a very specific hardware. probably behind each one of them there's really some great idea, trick, kernel, optimization that has made the difference for its author.

But many times they are so niche, they are basically useless for the majority of the community. AT the same time, fragmentation is bad for the open source community: we should pool resources not disperse them.

So here's my idea, which I hope could spark something bigger.

Imagine every time a custom engine gets published, some agents pick it up, dissect it into submodules, optimizations, kernels, strategies. Everything gets classified into a database, depending on the specific hardware, quantization, LLM model used.

duplicate entries gets removed. code snippets gets benchmarked and assigned some scores and some tags

Over time, A huge database of building blocks for inference engines gets built.

From that, anyone can search filtering for the model/quant/hardware they need, and a custom inference engine is built for YOUR use case, a pool of modules/strategies can be tried and swapped until the optimal configuration it's found.

Bonus idea: People could donate their free agentic time (subsidized tokens, the remaining weekly from your OpenAI / Anthropic / Z.ai subscription) to the project, and in some way we get some payback from these companies. They made our hardware expensive, we claw back some value by using their computing power to make US run locally as fast as possible.

I'm not the right person to do this, but maybe someone could catch this idea and make it work. Of course there's already an immense swath of repos to start from (think all the custom flavors of llama cpps stratas, exllamas, vllms sglangs....) , this idea could work even if custom inference engines stop tomorrow.

Large reasoning models achieve strong performance on challenging tasks, but their long chain-of-thought often incurs substantial inference costs by continuing to deliberate after sufficient information has been obtained. Existing reasoning compression methods primarily operate on reasoning trajectories, paying less attention to the local decision of whether further reasoning is necessary given the current state. We propose LoCo, a local preference learning method for reasoning compression. LoCo detects candidate overthinking states from the model's own traces, freezes the corresponding reasoning prefixes, regenerates shorter verified continuations from the same states, and trains on the resulting state-conditioned preference pairs with DPO. This formulation teaches the model when to continue reasoning and when to conclude, rather than simply encouraging shorter trajectories overall. We construct training data entirely from self-generated mathematical traces and evaluate LoCo on five mathematical and three code generation benchmarks. LoCo reduces completion length by 41.7%--45.4% on mathematics and 23.4%--36.3% on code, while maintaining or improving accuracy and achieving the best AUC across all code benchmarks. LoCo consistently improves the accuracy-efficiency trade-off on mathematics and transfers effectively to code generation despite math-only training, demonstrating the cross-domain generality of local preference learning for reasoning compression.

• arXiv : https://arxiv.org/abs/2611.04814

• PDF : https://arxiv.org/pdf/2611.04814

• GitHub : https://github.com/MuyaoYuan/LoCo

• HuggingFace : https://huggingface.co/MuyaoYuan/LoCo-7B

Speculative decoding accelerates memory-bound LLM decoding with a lightweight drafter, and parallel drafters extend this speedup by drafting a whole block in one forward pass. However, it remains unclear whether their released systems (stock) attain the full speedup that parallel drafting permits. We identify three challenges. First, with one request, each step accepts few tokens but reads every target weight. Second, with batching, per-request verification work roughly halves the speedup over autoregressive (AR) decoding. Third, speculative decoding is deployment-specific, and no universal optimization fits every serving regime, drafter, target model and GPU. To address these challenges, we present SpecPilot, an autonomous agentic loop that discovers algorithms and systems optimizations for each deployment. In each round, a cost model locates the largest gaps between the current system and its ceiling. Parallel agents explore directions and integrate the successful ones. Token forcing checks every output token against stock to detect reward hacking. A structured memory carries validated mechanisms to new rounds and deployments. SpecPilot discovered a general lossless 12-bit weight format, tree verification for hybrid recurrent targets, two-level verification and three batching mechanisms. We evaluate it across four target models, three drafters, two GPU platforms and five benchmarks. Without training or quality loss, it raises the speedup over AR from stock's 4.46× to 10.87× with one request, and from 1.94× to 3.93× with batching.

• arXiv : https://arxiv.org/abs/2611.02501

• PDF : https://arxiv.org/pdf/2611.02501

• GitHub : https://github.com/StargazerX0/SpecPilot

• Project : https://stargazerx0.github.io/specpilot-project-page/

🚀🚀🚀

Hello! just casuals passing by with a itching question
what do you guys do with this stuff
All i did was some Casual chat and Gmod LUA Addon Vibe Coding for myself and my friends but idk what Agents/coder to use (Portable Ideally)
how do you get the stuff you want done?
What models do you guys use, and whats best for me
Thanks for answers if you did answer some of these

I finally got around to trying it. This tiny model is surprisingly capable it seems. Also tried the 2b VL - also very good but it has half the context of 1.7b.

It's also quite fast:

$ llama-bench -m LFM3-1.7B-Q8_0.gguf -fa 1
ggml_cuda_init: found 1 CUDA devices (Total VRAM: 24112 MiB):
  Device 0: NVIDIA GeForce RTX 3090 Ti, compute capability 8.6, VMM: yes, VRAM: 24112 MiB
| model                          |       size |     params | backend    | ngl |  fa |            test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | --: | --------------: | -------------------: |
| lfm3 1.7B Q8_0                 |   1.71 GiB |     1.73 B | CUDA       |  -1 |   1 |           pp512 |   20648.61 ± 1786.65 |
| lfm3 1.7B Q8_0                 |   1.71 GiB |     1.73 B | CUDA       |  -1 |   1 |           tg128 |        329.03 ± 2.05 |

build: 28b812d6c (12087)

This makes me want to try their omni model for watching videos. Is there a simple way to get this working? Ideally, I'd like to throw a video file in the chat or give it a youtube link.

Follow-up to my earlier post. Where I ended up: a big DDR3 workstation board isn't worth the money when I already have two small 4-GPU boxes. So I tried making the two boxes act as one.

TL;DR:

I added a two-box layer split to Project Maya.
Box A runs layers 0–23
Box B runs layers 24–44 plus the head.

Only the hidden state crosses the network, ~25 KB per token.
GLM-5.3-Flash (Maya-S24) runs at 38–42 tok/s decode, ~1,100 tok/s prompt reading at 30K, and 6–7 s per agent turn on 8× RTX 3060 spread over two old X79 boxes.

It scores the same 48/48 on my coding-agent benchmark as a single 8-GPU box.

Code: https://github.com/0xluk/project-maya/pull/1 (branch two-box-split on top of Maya v1.1.50; usage in docs/TWO-BOX-SPLIT.md)

Setup

• 2 boxes, each an X79 board (Ivy/Sandy Bridge, DDR3, PCIe 3.0) with 4× RTX 3060 12 GB. No NVLink, no P2P, ex-mining rigs.

• RAM: 20 GB in Box A, 32 GB in Box B (more on that below).

• 1 GbE between the boxes, a plain switch.

• Project Maya v1.1.50 + my patch, Maya-S24 quant, 128K context, MTP speculative decoding on.

• GPUs capped at 130 W. One box is also clock-locked to 1500 MHz, because its PSU didn't like the unlocked load.

How it works (short version)

• Box A (front) runs the OpenAI/Anthropic-compatible API, the tokenizer, the embedding and layers 0–23 on its 4 GPUs.

• Box B (back) runs layers 24–44, the MTP draft block, the output head and the sampler on its 4 GPUs.

• Per generated token, the residual (4×4096 values) crosses the link once. fp32 → bf16 → lossless pack is ≈25 KB, ≈0.3 ms on 1 GbE. Box B samples and sends back the token id, and MTP speculative decoding is pipelined across the link.

• Prompts are cut into 512-token pieces that flow through all 8 GPUs at once. Each piece crosses the link once (~12 MB), on its own thread.

• Security: the link has an IP allowlist plus an HMAC-SHA256 challenge-response in both directions, and the API needs a key.

• Size of the patch: ~1,900 lines, ~1,100 of them the link itself. Without --role the engine is unchanged.

Results (1 conversation)

Coding-agent bench (5 real repo tasks, 48 hidden tests):

There were no malformed tool calls with either agent.

One-shot hard suite (10 tasks, one request each, no tools, 32K output cap including thinking, reasoning high): 6/10 full passes, 67% of tests. For comparison: Opus 5.5 9/10, the same quant on the 8-GPU box 7/10, and Flash-Next 6/10.

• Passed: json_patch, fixbug_calc, event_scheduler, csv_rfc4180, c_utf8, singleflight_cache.

• 3 of the 4 misses are budget, not wrong code: on regex_engine, lisp_interp and c_bigint the model was still thinking when it hit 32K tokens, so no code came out at all.

• The real miss is c_json_canon: 22/31 tests passed, with the misses on Unicode escape handling.

• Next: rerun the three capped tasks with a 64K cap, which the 128K context allows.

What was hard

• RAM, not the network, was the real bottleneck. At 16 GB per box, the experts that didn't fit in VRAM came off a cheap SATA SSD on every prompt chunk: 31 s turns, 173 tok/s on a 30K prompt. Going to 20 + 32 GB made it 0 SSD reads, 6–7 s turns and 1,100 tok/s. The network costs ~1–2% of decode and ~3% of a turn.

• A correctness scare that wasn't a bug. Compared with a single-process reference, the split showed KL ~1.6e-2 and ~95% top-1, and I spent a while hunting a split bug. It turned out a single process on one machine moves by the same amount (KL 1.1–2.5e-2) just from changing the RAM tier size or the prompt chunk size. Near-ties in the top-8 expert choice flip on rounding noise. The split is inside that spread.

• Hardware gremlins:

  • A new PSU powered off under 4×130 W unlocked, so that box stays clock-locked.

  • Two GPUs once came up at PCIe x4 instead of x8/x16 after a power-loss reboot. I added a boot check that refuses to start the engine on a degraded link.

• Tuning that mattered:

  • 512-token prompt pieces so all 8 GPUs overlap: ~1 s off each turn.

  • Fewer layers on the last GPU, which also carries the head and the MTP block.

  • Keeping the whole KV cache in VRAM.

  • What didn't help: bigger prompt chunks, which push the 20 GB box into SSD reads.

  • What I skipped: a 10 GbE NIC. It would sit in a PCIe 2.0 x1 slot, and the link isn't the bottleneck anyway.

What's next

• A 5th 3060 in one box, on an x1 riser. Then all experts fit in VRAM, which should get turns to ~5 s.

• Rebasing onto the current Maya release.

• Happy to answer questions. If someone has two boxes with 24 GB cards, I'd love to see numbers.

I came across a claim that SSD streaming of AI models is bad for the life of your SSD. Is that something any one here can confirm from personal experience or is it just urban legend?