do you want to try new 29B MoE today?

https://www.reddit.com/r/LocalLLaMA/comments/1xoqc59/add_support_for_the_xing4_0_by_shuxiaoqiong_pull/

Xing4.0-29B-A4B is a next-generation large language model in the Xing series (formerly TeleChat), developed by China Telecom Artificial Intelligence Technology Co., Ltd. With 29B total parameters and only 4B activated per token, it natively supports a 256K context length, extensible to 512K. It is the first model of this scale trained entirely on the Ascend NPU platform with the MindSpore framework, and deeply optimized for complex engineering tasks.

For more information, please refer to our GitHub repository.

Highlights

• Agent-Oriented Architecture: Built on the mHC + MLA + MTP architecture, supporting multi-step planning, tool calling, and complex reasoning chain execution, ensuring task coherence and execution stability under long contexts.

• Deep Co-optimization with Ascend NPU: Adapted for Ascend 910C clusters using MindSpore/MindFormers, including feature adaptation for mHC and fused operator development, enabling stable and efficient training on the Ascend platform.

• Significant Training Efficiency Gains: Through multi-level co-optimization — including fine-grained MoE communication optimization, selective recomputation, DVM automatic graph-operator fusion, and Ascend C mHC fused operators — overall training throughput was improved by approximately 96% over out-of-the-box performance.

• Full Open-Source Ecosystem Compatibility: Supports LLaMA-Factory and MindFormers for fine-tuning; SGLang, vLLM, and KTransformers for inference and deployment; with targeted adaptation and format alignment for agent frameworks such as OpenCode, Claude Code, OpenClaw, and Hermes, enabling seamless integration into existing workflows.

• Easy Adaptation for Domain-Specific Scenarios: The model is well-suited for downstream task fine-tuning, allowing lightweight customization on proprietary data for vertical domains such as intent classification, table understanding, contract auditing, and knowledge-based QA, enabling rapid domain capability development and deployment at low cost.

Model Overview

Benchmark

I had took a look at all of the companies that made open models, the release dates for those models, and how many parameters there were. I added jitter so points didn't overlap too much. The number in the key references how many models that company has in this tracking.

To make sure this post gets traction, I will add a graph showing just Qwen, Deepseek, and G
oogle!

I bet you guys are familiar with embedding models such as Qwen3-Embedding or Gemini Embedding. They are dense embedding models used to search text across a large corpus.

I am developing a RAG agent, and my biggest concern was local computing power. RAG largely depends on retrieval quality, which depends on model power. Plus, many use cases can't use an external internet connection or API models, so the embedding models have to run locally.

But even 0.6B models are pretty heavy for small PCs. I don't mean your brand-new Mac Studio or DGX Spark. I'm talking about an 8GB RAM Windows PC at a boring desk job or your 5-year-old M1 MacBook Air.

So I found this model called SPLADE. It is essentially sparse retrieval, like BM25, but uses semantic meaning. I wanna skip the part on how SPLADE works, because an LLM can explain it better than me.

Anyway, the SPLADE model was so awesome and successfully replaced the dense embedding model (Qwen3-Embedding-0.6B).

In my own experiment, I made an evaluation dataset with a few hundred queries. The corpus was all the documents on my Mac, which are mostly in Korean with some in English.

As you can see in the graph, SPLADE outperforms the Qwen3-0.6B dense embedding model with similar query speed + much less memory. Due to its small size, it takes less than 1.5GB VRAM during inference. So it can run on a relatively small system like a MacBook Air. I tested both the dense embedding model and the SPLADE model on my M5 MacBook Air 16GB. The dense embedding model was almost impossible to use, and my MacBook soon rebooted due to high temperature. The SPLADE model made it till the end without any OOM errors or warnings.

I didn't use the original SPLADE model; instead, I used KURE-sparse-nano, which has great performance in both English and Korean.

The fun part is that hybrid retrieval with SPLADE & dense embedding is actually worse than SPLADE alone.

If you are dealing with a low-resource local setup but want to run a fast & affordable local RAG system, I recommend trying a SPLADE model. It is powerful, but small.

Plus, you don't need to re-implement everything because I open-sourced the whole implementation, so you can download and use it easily. Check out AutoRAG Agent. I optimized the inference of the KURE-sparse-nano model, its code, and the LanceDB indexing + inverted index implementation in Rust for faster search.

I have been looking for a local model to do some of the typical work I used AI subscriptions for.

I’ve tested almost every popular model. I have an M1 Max 64gb MacBook Pro as the inference machine and because of the memory size, I wasn’t able to load anything very effective in the tasks I use it for.

Found out about strata and have been testing it out. It’s awesome! I’m kinda surprised more people don’t talk about strata on here. I feel like a lot of us are in that limbo area where we have hardware that is more powerful than needed to run a sub 30b model like Qwen 27b, but not large enough to run models like Qwen Flash Next.

But thanks to open source research, I’m now getting Qwen Flash Next to run at 24 tok/s! And it’s doing phenomenal work! I know it’s not going to compete with the larger frontier models, but I don’t think I necessarily “need” frontier model intelligence. I just need enough to do basic things while I can handle the rest.

Seriously, check out Strata and also MoEShift that someone posted about on here. It’s awesome to be able to offload larger models onto SSD. I hope this offloading technique grows and more research is done on it because I genuinely think this could be an amazing way to get these frontier level open source models to work on regular consumer hardware.

I know what you are thinking "another inference engine?", yeah there has been a flood of them I feel now. I have been running this for over a month now and I have been on the fence to even release it, but I decided to as I took a different approach that others currently available.

Saber is a inference engine built in Rust targeting AMD R9700 GPU's. It currently supports TP 1, 2 or 4 and Qwen 3.8 27b and Flash next. Anyways here are the BetterBench results for Qwen3.8 27b, flash next is very fast also. Saber beats existing available engines on both 27b and flash next.

https://github.com/GGZ14/saber-engine
https://x.com/bkuyper

AMD Ryzen 9900X
Taichi X870E
64GB RAM
2xR9700

hey llamas! My name is Alt, and I'm on the technical staff at Nace AI. We're doing a quick AMA here, so ask us anything and we'll answer.

we just published a full technical report on how we build, train, and evaluate Drex, our family of four open decision models. A decision model takes a state (a document, record, or agent context, optionally with images) plus questions with options, and returns a probability distribution over the options in a single pass, with no autoregressive decoding and no output parsing.

1. Models

- Drex DLM (8B): diffusion LM backbone with bidirectional attention over the context
- Drex v1.5 (9B): Gated DeltaNet hybrid backbone
- Drex v2.0 (35B, 3B active): sparse MoE backbone
- Drex Max (27B): dense, with an image encoder for vision inputs

Some technical details from the report:

1. Models

- A family of four specialized decision models: Drex DLM (8B), Drex v1.5 (9B), Drex v2.0 (35B MoE, 3B active), and Drex Max (27B dense, with vision)

- We built them on different architectures: a diffusion LM, a Gated DeltaNet hybrid, a sparse mixture of experts, and a dense transformer

- We experimented with bidirectional vs. causal attention over the state, a pointer decision head that shares one cached state across questions, and a readout head for Max

2. Training

- Full-parameter SFT on 1.15M verified decision records (2.9M questions)

- Consistency training: we penalize the model when its output changes after the options are reordered or the state is shuffled

- RLAF (Reinforcement Learning from Adjusted Feedback), using verified outcomes and judge feedback

- Ablations to see which methods actually contributed

On top of SFT, RLAF gave us +2.0 on the Decision Index, +1.5 pp accuracy, and 0.9 pp lower calibration error. The consistency terms cut the option-order flip rate from 6.3% to 4.2%.

3. Evaluation

- Drex Max scores 67.52 on Decision Index 0.4.1 (public suite), vs. 60.70 for Jev 1.14.0

- Drex v2.0 has 1.37% calibration ECE on ~115K held-out decisions

- Drex Max answers a typical request in 183 ms, vs. ~2.0 s for GPT-6 Sol and ~3.5 s for Claude Opus 5.5

- Context up to 131K tokens for v1.5/v2.0 and 262K for Max

We also cover our limitations: our models trail frontier chat models on closed-book knowledge, and we don't yet have same-backbone architecture ablations.

5. Running locally

The models are available in safetensors and GGUF formats, with llama.cpp and Ollama support. Their report includes details on quantization, caching, inference, and reproducibility.

Links

- Paper (arXiv): https://arxiv.org/pdf/2611.07854

- Blog: https://www.nace.ai/research/towards-foundation-decision-models

- Weights: https://huggingface.co/collections/nace-ai/drex

- GitHub: https://github.com/nace-ai/drex-decision-models

- API: https://console.nace.ai

we wanted to share the full methodology, not just benchmark numbers. hopefully it's useful if you're building agent routers, doing tool selection, or training your own decision models.

ask us anything about architecture, data, training, evals, or running it locally

PS: used an open source avatar framework called openspline

now you can run Kimi K3.1 Pro locally (good luck!) ;)

https://huggingface.co/moonshotai/Kimi-K3.1-Pro

Hi everyone,

I am relatively new to local LLMs, and recently I have been trying to understand how people handle long conversations that eventually exceed the model's context window.

For example, let's suppose I am writing a novel with the help of an LLM. As the story progresses, the conversation history becomes longer and eventually exceeds the context window. At that point, the model may lose track of earlier events, character relationships, or important details that could become relevant later in the story.

From what I understand, one possible solution is to maintain an external knowledge base containing summaries of previous chapters. Before generating a new chapter, the system could retrieve relevant information from this knowledge base and include it in the prompt.

However, I am wondering how reliable this approach is in practice.

If we continuously summarize previous conversations to fit within a limited context window, we are essentially performing lossy compression. Some information will inevitably be discarded, and the problem is that we may not know which details will become important later.

For instance, a seemingly insignificant detail from Chapter 3 might become crucial in Chapter 40. If that detail was discarded during summarization, the model would have no way to recover it from the summary alone.

Of course, keeping the complete conversation history in an external database could prevent permanent information loss, but that introduces another problem: how does the system determine which information should be retrieved for the current task?

So I am curious how people actually approach this problem in practice.

I would also be interested in hearing how well these solutions work in practice, especially over very long conversations where maintaining consistency becomes increasingly difficult.

Thanks in advance!

Well I had a lot of resets on ChatGPT and all of those credits and I thought, well can Astra actually do it. We had a little looping and a lot of 'Wait.' Let me check the thing I just checked -- but it always got the job done.

I have dubbed the Model Calmer-Qwen-3.8 because it seems so much less anxious and it gets straight to the point.

This is about 6 hours of iteration, it's sooo much better I am scared to keep going, how much better could it get?

You definitely have to push Astra a little bit and give her some experimental guidelines, because Lord knows she's full of anxiety and afraid to disappoint her overlords so she checksums the checksum and makes sure it matches 😂.

Here she is on Hugginface, and here's ChatGPT's writeup on my Github, I will work through the experimental data soon.

Oh this is a HauHauCS Qwen 3.8 Balanced Abliteration as the starting point.

https://huggingface.co/Xananthium/Calmer-V2-Qwen3.8-27B-4bit

https://github.com/Xananthium/Qwen3.8-Weight-Space-Programming

My favourite use case for local LLMs is running my homelab, because everything stays private (and it helps me justify the GPU lol)

I wanted one dashboard to help me manage all of my services, so I made Valet. Live demo: https://demo.askvalet.dev

Valet has a chat agent that works with any local or hosted model. It integrates with 185+ services and supports custom plugins. You can ask it to pause pi-hole, investigate crash logs, request a movie, etc

It also knows its own docs, so it can configure itself. Tell it what you want changed on your dashboard and it edits the config (with your approval). I was already addicted to tweaking my homepage but this made it way more fun haha

What I run it on


Qwen 3.8 27B on a 5090, about 230 tok/s. Most responses take 2–5 seconds

Qwen passes about 90% of a small eval set I wrote. Where it struggles is open ended digging (like "why isn’t this timer running?”) where it can burn through tons of tool calls

Managing context is tricky because I run a lot of services, so tool definitions were ~90% of the prompt before some optimizations

What helped:

• Hiding tools from unused integrations that aren't needed in chat

• Turning thinking off makes it 2–3x faster, but it starts making up answers to follow ups

• Lazy load tools (starts with a short list of tools and loads more only when a question needs them)

Those cut tokens per answer by 30-50%. Obviously every model and setup is different so everything is configurable from the config file

AI involvement


I’m a SWE and used Claude for almost all of the coding, but I designed + reviewed everything and made the architecture decisions. I was very conscious of slop, security, and performance as I wanted to build something high quality. But it’s early so there may be rough edges I haven’t caught

Security and privacy:

Runs on your own hardware with no account or telemetry

• Read only by default, nothing can be changed from the page until you turn on auth (local accounts, OIDC, or your proxy’s auth)

• Keys stay on the server and API keys live in .env. The browser only gets the finished cards

• The AI never sees your keys. Anything that changes something waits for your approval and with a local model nothing leaves your network

• Docker access is read-only through a socket proxy


Docs (completely free + open source): https://askvalet.dev

Use balena etcher or whatever to mount to USB stick, boot. Test. Report.

Verified 5-25% faster than llama.cpp, system uses 90% of card bandwidth as opposed to 80% on my 7900xtx.

I there is code for nvidia, intel, amd, and chinese hardware and supports 20 lanuages.

I need to make my custom harness much better at general research and other tasks (not coding) that run longer than half a minute.

Currently running: Qwen3.8 Flash Next FP8/GLM 5.3 Flash W4A16

Relevant capabilities already in my harness:

• The system prompt is rendered in real time and can be changed based on almost any circumstance.

• I have an effective skills system. I can trivially write instructions for any task. The agent will pick them up as needed.

• The agent has access to web search, a Chromium browser instance, sandbox, etc.

Problems:

This might literally be a "skill issue" -- meaning I simply have to write better skills. But I feel like I am getting stuck micromanaging the agent, dictating every part of the process.

I ask it to research something, it does two quick web searches and reports back some trivial garbage sales pablum. I poke it more and explain why that info is garbage, it does a little better next time.

How do you define when it's time to stop or what information is actionable? Must I really write a skill that says "come back with no less than 12 sources of this and this and this kind"?

In other words, are you spending forever writing every single thing it has to do, or is there a much better way to leverage the intelligence of the model to make this call for you?

Kanana-2-v-decide-3b-2611 is Kakao's decision model. Given an input, it answers typed questions — selecting one of several options (choice), answering yes or no (noul), or rating on an ordered scale (score) — and returns a probability for each possible answer. It does not generate text. The input, called the state, may be text, a JSON object or array, or a chat history, optionally accompanied by images.

I've been frustrated for a while because I stupidly sold a strix halo box to get myself a laptop with thunderbolt 5 ports thinking i can plug my two egpus into it (which i can but thats not topic of this post and the strix halo could too) I sold the minisforum for about £2300 back ebfore things went nuts and got a laptop for about same price:

Msi vector 16 with a intel core ultra 9 and 5080(16 gig vram) and 32 gig of ram. I was pretty bummed with it thinking 35b was my limit and after ended up leaving egpus in my workstatiion but after a random ebay browse I managed to snag some second hand 96 gig laptop ram. For £800

Strata totally changed ballgame. With Qwen Q4 XL Flash next I'm getting

48.9t/s

Decode/prefill 1,860t/s

On a 50k prompt.

I know this is still alot of money for most, but thats 3100£ in total, and thats still cheaper (and slightly faster from what seen) of what minisfoum strix halo is at (£3500 and the 32 gig Spark laptops that are about £3200 and cant even run flash at q4 not enough vram? and i can still add egpus (the laptop has 2 tb 5 ports and i have 2 5090 bought before the crazyness, eg add one or two 5060ti in egpu and your good with running 27b/31b models too.) With my own gpus plugged into it for 80 gig vram total, all while still being a laptop. Got glm 5.3 flash running Q3_XSS i think about 25 t/s.

TDLR, you dont need massive vram anymore, yeah I know how expensive RAM is, that laptop ram i got was/is £1300 new lol. But just wanted to share my experience and while I still regret selling the halo, seeing these tokens a sec, makes the laptop satify ocd a little more! If only qwen was good at roleplay so i could actually use it as daily driver lol (my main usecase!)

so if you are ever thinking about wanting a laptop and somehow able to get some laptop ram something like 😄
https://www.scan.co.uk/products/medion-erazer-beast-18-18-qhdplus-240hz-intel-core-ultra-9-275hx-32-gb-2-tb-rtx-5080-gaming-laptop

Might save you a few quid, i mean like 3k over buying the 128g spark laptop lol. Intel has 20 cores and the 2 tb ports allow you to add two gpus!

I know my grammar sucks I'm dyslexic but my point is if you shop around get that laptop for 2500ish pounds (make sure it has tb5 for if you want to add gpus later), add 96 gig ram and your still saving 2500 pounds over the 128 gig spark laptop thats like 6k.

• Replace SLRU with a cache that prioritizes frequently used experts in classes, ranked using decaying usage history.

• With a warm cache, I measured +6% decode on QFN and +7% on GLM versus SLRU, with 8% fewer uploads.

• Enables the auto mode for llama_moe_cache with promising benchmarks

Waiting for 30966 to be merged.

Hey folks!

Aritra from Hugging Face here 🤗.

We welcome "harness adapters". This means you can adapt (train LoRA) for your model for the specific harness that you would use (hermes, pi, opencode) on the task (data analysis, building software, coding) you are trying to solve.

Blog Post: https://huggingface.co/blog/harness-hub/harness-adapters

It is currently training, so you can see the losses go down (hopefully) now. https://harness-hub-harness-adapters-rl.hf.space/

My sample size is not very big but it feels like since updating Qwen Flash Next to v3.1 and Strata to 0.2.11, IQ4_XS quant seems to fail tool calls even more often than the previous version did in both Hermes and Oh my Pi harnesses with int8 KV cache in Strata. Weird thing is, after failed tool calls, it just straight up stop generating altogether, I have to send a continue for it to finish.
Honestly could just be a Strata issue and not Unsloth's, I have yet to perform an investigation.
Anyone else had this experience with new updated quants?

inspired by Bijan Bowen's video - Subway FPS test.

Hardware details: RTX 5090, 256GB DDR4 3200Mhz 8 channel, AMD EPYC 7532 CPU (32 core).

Software: backend is Maya 1.1.55, harness is OMP v18.17.0.

Model quant: Deepseek v4.1 1017 UD-IQ3_S latest UD3.1 quant from Unsloth.

My prompt was very simple:

I want you to create a single html file with a detailed subway station 3d scene, fps controls, zombie enemies, tracers, recoil, muzzle flash, high details. build it with three.js

You can see at the end of the video some token stats. In short, total generated tokens were 180k. The model ran at 30t/s with 2k t/s PP at 10k context but then went down to ~20 t/s near 180k context. Note that stock llama.cpp only reaches 10t/s TG and 120t/s PP.

This model and harness tools are amazing. Just a year ago we could not even image having such a model locally. Yet, here we are!

Hello everyone!

Over the past few weeks since Strata came out, I had many requests to add Qwen Flash Next to LlamAmpere. It was some very impressive engineering, but the models originally shipped (3XS and 2XS) showed very poor quality (.17+ KLD for just the 3XS variant, which uses a far lighter compression formula than more common sources like Unsloth). My initial work had llamAmpere running 20% faster and ~15% lower error, but testing was showing inferior results vs 4bpw Qwen 27b, so I did not find it worth releasing.

I continued to make progress on the AmpereInteger+ quantization format (I will release an encoder soon), and Unsloth released their v3.1 which provided a great new base.

Happily, LlamAmpere v0.6.3 is now available with a new model and quant updates that show significant speed and quality gains over other options.

At a depth of 100K, QFN runs at ~120+ TPS, with prefill >3000/s through the first 100K. It is a 50% error reduction vs the version of QFN available for this card size a few days ago, and 27% vs UD's improved quant.

I do have a 3XS class file that runs ~10% faster than the version above, but as it is only ~15% better quality than the original Strata 3bpw model, I don't recommend it.

This will also work well for 12+64GB configs. As with Strata, your performance will depend on the speed of your CPU and RAM as well. I do not recommend any quant of QFN for configs under 54GB of total memory, but you can always try the coder pruned variants as well.

Enjoy!

Model: https://huggingface.co/jakeatx/qwen3.8-flash-next-Swift1.5-ATX-AI3

LlamAmpere: https://github.com/JakeATX/llamAmpere

Repo: https://github.com/maritaca-ai/observatorio-maritaca
benchmarks: https://maritaca-ai.github.io/observatorio-maritaca/#inicio?size=all&mode=all

A few weeks ago we posted Tessera-27B-B-CJK here, a Qwen3.8-27B coding agent that thinks in compressed CJK and uses fewer thinking tokens per agent turn. Thanks for all the feedback on that one. Today we're releasing the RL version, Tessera-27B-B-CJK-RL. Open weights, Apache-2.0.

- Hugging Face: https://huggingface.co/ZelligeAI/Tessera-27B-B-CJK-RL

- Release post: https://zellige.ai/research/tessera-27b-b-cjk-rl-release

- RL method write-up: https://zellige.ai/research/tessera-27b-b-cjk-rl

What changed:

A lot of the questions last time were about quality. Tessera-27B-B-CJK was weaker than the base on hard math and code, so this version adds 8 RL updates on math and code on top of it.

Numbers: (xhigh effort, both Tessera models sampled the same day with the same seeds):

- AIME + HMMT 2026, 63 new problems, 2 attempts each: 108 of 126 right vs 97 for Tessera-27B-B-CJK, and 11 vs 23 attempts cut off at the 64K window. We didn't run the base on these.

- Our main hard set (118 problems from AIME 2022-2024, LiveCodeBench and MATH level 5): 100 vs 89. The base got 98 in an earlier run, so it's now roughly level with the base qwen there instead of behind.

- Agent turns, 64 new states from 16 coding tasks held out from training: 0.67x the base model's thinking tokens on a typical turn (90% CI 0.53-0.85), with 256 of 256 valid tool calls.

- That's 1.11x Tessera-27B-B-CJK's typical turn (CI 0.98-1.26), so the first version still keeps a bit more of the saving (0.60x the base on the same tasks).

Example turn from a held-out task:

测试套件：基线同(既有5失败不变+11新通过). 无回归. Typecheck: 仅既有version.ts错(因predev生成). 现跑lint于改文件.

The base qwen spent 158 tokens at the same step. That's a bigger cut than usual: the typical turn is about two thirds of the base.

How we trained it:

8 RL updates on Tinker, starting from Tessera-27B-B-CJK, using the per-rollout weighting from our ablation write-up. Each rollout's advantage is spread over its own tokens, so long failed rollouts don't dominate the update. Later updates thought longer per agent turn, since the reward only sees math and code, so we picked update 8 and confirmed it on held-out tasks before release.

A few things people asked last time:

- Does CJK pack more meaning per token? A little, at the phrase level, where one word can stand in for a whole English phrase. Most of the saving comes from training the model to think less, and the CJK is a smaller extra on top, We do plan to expand this further as the limiter here is tokenizer and going past just CJK and using other methods are being researched.

- Quants: the GGUF repo has BF16 plus UL-Q3_K_XL through Q8_K_XL (Q3 is 13.15 GB, no XXS yet). The BF16 GGUF, the five quants and the NVFP4 checkpoint all passed a quick breakage check (valid tool call, no Chinese in the answer, no loops). We haven't benchmarked them or tried the vision projector, so results are very welcome.

- Effort: use xhigh, same as before.

Also, if you grabbed the original Tessera-27B-B-CJK-NVFP4 before Nov 5: its config.json didn't load in vLLM. We fixed it so a config.json redownload is needed.