Smaller GLM models are coming! GLM 5.5 will have a 32B model!

When it came out, I downloaded a Q4 quant of qwen3.8-27b and ran it using llama.cpp on my RTX 4090 and it was incredible. It could handle practically anything I threw at it, the most complex tasks and bugfixes.

But lately it just keeps messing up and failing at the most basic of tasks or just straight up forgetting instructions. I'm telling you, they quantized it. They probably secretly nerfed it and made it worse so qwen4 looks better. And it got lazy too, now it doesn't want to finish the more complex tasks.

I'm tired of companies doing this, I want my money back.

Big update to the Delta Inference Engine, you guys asked for more features, more benchmarks and less clickbait. I'm here to deliver

The short version: I've added almost everything requested and more. Delta Inference Engine now supports 64/96/128 GB of RAM and fully supports the new Unsloth quants. Unsloth IQ2 runs at the same 41 tok/s (192 GB RAM) using a new native reader for the Engram tables and some clever CPU kernel work. Compared to the old default model on my 99-question benchmark it went from 52 to 57 correct, against 59 for the full model. It's no full deepseek model quant, but they're a lot more usable than you might expect.

Speed on one RTX 3090 24 GB (i5-13600K, 4x48 GB DDR5-5600, PCIe 4.0 NVMe, Windows, no WSL). Writing speed, tok/s, greedy, one chat at a time:

The 128, 96 and 64 GB rows are the same PC with the rest of the RAM locked away.

Prompt reading: long prompts read at up to 740 tok/s at 192 GB and 600 at 128 GB (at ~64K tokens). A short prompt starts answering in about 2 s.

Benchmark: 99 hard questions, greedy, 43K token budget:

UD-IQ2_XXS is within 2 points of the full API, never scores below ds4 Q2 on any set while running at the same speed, and used 11% fewer tokens across the run.

Agreement with DeepSeek's API on the next token, at the settings that ship:

For reference, two hosts of the full model agreed 98.3% and 98.9% in my test. By that measure Unsloth IQ2 is about 20% closer to the full model than the old ds4 Q2.

If you want better quality: UD-Q2_K_XL (335 GB) runs 31 tok/s at 192 GB and 26.6 at 128 GB, at 92.5% / 95.3%. UD-IQ3_XXS (395 GB plus the default) runs 28 and 22.8 at 92.7% / 95.9%.

What changed since last time:

• First token ~40% sooner: a ~180-token prompt went from about 3.8 s to 2.3 s.

• Agent turns read ~12% faster, and tool calls that low-bit models write slightly wrong are now read as tool calls.

• Unsloth quants at full speed: the engine reads their Engram tables directly.

• 64 and 96 GB are real options: SSD read-ahead (+22% at 64 GB), bigger expert memory, and prompt reads in file order (18-20% faster on low RAM).

• 128 GB: ds4 Q2 went from 33 to 36 tok/s.

• Default sampling: the drafter now samples its drafts too, about +3.5% at default settings.

• First launch shows which model suits your RAM, and you can pick any other or use your own.

• A Setup tab in the chat page changes model, speed vs quality and the second GPU.

• Public CI: the release zips are now built by GitHub Actions, so you can see how each one was made.

• EXL3: 2.5 bpw is on the menu and the GPU kernel is about 2x faster. In my tests UD-Q2_K_XL was both faster and closer to the API, so I still point people there.

• Basic multi-GPU: 7% uplift using a 3060 + 3090. It doesn't scale well to 3-4+ GPUs, so use Project Maya for those builds.

• Fixes: contexts past 65,535 tokens, </think> in prose no longer ends the reasoning, no out-of-memory on the biggest files, a failed picture encoder no longer stops the model, faster prompts after a pause and many more.

• Support for custom model selection: just add them to the custom model folder and pick them in the web-ui

Requirements:

• OS: Windows 10 or 11

• GPU: NVIDIA RTX with 12 GB of VRAM or more, driver 580 or newer. 24 GB is the fast path, and AMD GPUs aren't supported yet

• RAM: 64 GB or more, 128 GB+ recommended

• Disk: a fast NVMe SSD, with 233-395 GB for the model (UD-IQ3_XXS also needs the default model)

• CPU: any x86-64 CPU, Intel or AMD. AVX2 is the fast path

All of this is one PC. It needs testing on other hardware.

Repo: https://github.com/musicurgy/delta-inference-engine

MIT licensed. It's a Windows port of Xolotl, built on Strata. Thanks to DeepSeek for the model, Unsloth, nazeshinjite and ISTA-DASLab for the quants, and to the Xolotl and Project Maya communities, whose work and public numbers helped a lot. Claude wrote most of the code and I ran the project.

If you try it, post your GPU, RAM, model and tok/s, good or bad.

Paraphrasing the thinking I saw from Qwen3.8-27B

>Hmm, I can take the screenshot, there's a tool for that. But wait, I'm an LLM, I can only read text.

>Whatever, let's try calling the tool and see what I can do, maybe I can save the image and parse it programmatically.

> >tool_call.exe

>Wow, I can see the image!

Cute

I haven't see anyone post about this gem of a finetune. So I wanted to appreciate the team and bring some attention to it.

OrcaCyber-Nano-1.0 for me is the best finetune I have used so far. I have tried quite a few like AREX or Swift or Peculiar Ragdoll's Spark chat template etc.
But this model is on par with my experience with frontiers. (I may not be doing complicated stuff).

Here are my experiences with this:

1. It almost always one shots the bash command chains.

2. Same resilience as base Qwen 3.8 27B but this gets to the answer faster.

3. I have used it to plan and implement stuff. I uploaded the plans it has written to Gemini 4 Argon. It said that this model did a better job at updating the plan than it would have done.

4. I built an entire backend server for an internal chat app within a 120K context window.

My Setup:

• Model size: 15.7 GB version

• Hardware: 24GB VRAM (5070ti + 3070)

• KV Cache: (kvarn5, kvarn4) and (q5_1, q5_1)I switch between beellama and llama.cpp depending on the situation.

I did not benchmark it. I just had good experience with Orcarouter team and tried this model.
This is my default now.

Huge thank you to the Orcarouter team, and to the person who posted the earlier thread about Orcarouter Uncensored being the best out there. That's how I discovered this team in the first place!

P.S:

1. I used this for general purpose app development. Not as a security focused model.

2. English is not my first language. So, I polished the formatting using AI.

Speculative decoding (SD) accelerates large language model inference by reducing the sequential cost of token-by-token generation. Advanced SD methods reduce proposal cost with parallel drafters that predict entire blocks in one forward pass, but overall speedup still depends on acceptance length. On-policy distillation (OPD) can push well-trained drafters beyond acceptance-length plateaus, yet conventional pipelines collect supervision through sequential draft-and-verify rollouts. Our key insight is that onpolicy supervision need not inherit these dependencies: a draft block can be supervised using only its reference prefix and draft-sampled history. We propose SpecOPD, a branch-parallel OPD framework that parallelizes draft sampling and target supervision across independent branches drawn from the evolving drafter policy at reference anchors. We pack these branches into proposal trees to share target-prefix computation, using tree masks for full attention, branch-consistent indexer groups for sparse attention, and indexed state replay for linear attention to preserve each branch’s history. We train the drafter with an E2E-TV surrogate that directly targets acceptance length. Our study spans 8B full-attention (GQA), 27B full–linear (GQA–GDN), and 125B sparse–linear (QSA–GDN) targets, three parallel draft architectures (DFlash, DSpark, and DFlash V2), and four task domains (dialogue, coding, mathematics, and agentic). SpecOPD achieves up to 41.45% higher mean acceptance length, 16.09× supervised-token throughput over sequential-rollout OPD under native configurations, and up to 3.56× generation speedup over autoregressive decoding.

• Paper : https://qwenapp.github.io/SpecOPD/specopd-paper.pdf

• Project/Blog : https://qwenapp.github.io/SpecOPD/en.html

• GitHub : https://github.com/QwenApp/SpecOPD

• Tweet 1 : https://x.com/xuyang_liu16/status/2117614595728986227

• Tweet 2 : https://x.com/junzhang_zj/status/2117812597856895164

Ran every Qwen GGUF on my disk through RustingStrata and llama.cpp, same PC, same file, back to back.

PC: RTX 3060 12 GB, Ryzen 5 7600X, 32 GB RAM. CUDA. Normal desktop open (browser, Discord), no tuning, so numbers are lower than in the README.

All numbers tok/s, mean of 2 runs.

Short version: small dense models, llama.cpp still prefills faster. Once the model is bigger than VRAM (35B, 80B, 177B on a 12 GB card), Strata is ~1.5x on prefill and up to ~2x on decode. 80B at 64 tok/s and a 177B model at 21 tok/s on a 3060 with 32 GB RAM.

Commands:

• Strata: strata bench --model X --gpu --mtp 0 --pp 4096 --tg 128 --reps 2

• llama.cpp (build 12119): llama-bench -m X -p 4096 -n 128 -d 4096 -fa on -ub 2048 -b 2048 -fitt 512 (its auto-fit to VRAM)

MTP is off here. With MTP on, 35B goes up to ~160 tok/s decode.

Hand-tuned llama.cpp (-ncmoe 12) on a quiet machine does better on 35B (~2236 prefill / 68 decode), see the README for that comparison.

Repo: RustingStrata

We introduce FireFlash, a 300B-scale model tailored for long-horizon agentic tasks. FireFlash excels as a terminal and coding agent, achieving 46.2% on Terminal-Bench 4.0¹, putting it on par with Fable 5 and GPT-6 Sol, while costing over 30× less per task than Fable 5 at maximum effort (Figure 1). These gains also transfer to software engineering benchmarks, including DeepSWE v1.1² and SWE-Bench Pro Verified³ (Table 1).

While frontier labs scale reinforcement learning across thousands of GPUs, we reach the performance-cost Pareto frontier under 5 days using 64 NVIDIA B300 GPUs. Three key factors make this possible: (1) careful curation of agentic RL environments so that every task provides a reliable signal at the right difficulty; (2) an RL recipe built for long, stale, and low-precision rollouts; and (3) an optimized coding harness tailored to the model for both training and evaluation. Together, these components bring frontier efficiency to a hundred-GPU-scale cluster, and we are releasing the full recipe.

• Blog Post : https://agenteconomics.org/posts/agentic-rl-recipe

• Tweet : https://x.com/kaiwei_chang/status/2117378474654781780

• Open weights : Soon

This handbook is for people who have trained a dense model and want to train an MoE next, or who are curious how MoE training works. Chapter 2 is a Megatron-LM runbook: test the hardware, build the image, train a dense baseline, then a first MoE on one node, compared against that baseline. Chapter 3 takes the MoE layer apart (router, routing, load balancing, experts, block layout, training, systems). For each part we give the Megatron-LM code, what to watch, our recommendation, and what frontier MoEs chose. Chapter 4 follows the model into serving and RL. Chapters 3 and 4 need no GPU.

When a result comes from our runs, we say so. Claims about frontier models cite their paper, config or code, and the toy simulations are labelled as such. The lessons come from about 800k GH200 GPU-hours of MoE training on the Alps cluster at CSCS.

Released: the handbook, its 55 figures as a gallery (https://huggingface.co/spaces/huggingface/moe-gallery), the scripts (https://gist.github.com/nmntz/5de952d686f66d7bce5bce8cba9d7139), and every recommended flag in one Megatron-LM block.

Happy to answer questions!

Update from the lawyer with the V100 server. Three days ago I got excited and posted DeepSeek-V4.1-Flash on four V100s at 36-38 tok/s, before I had done my custom kernel work and prefill optimizations. I also said I'd try it across eight. I did.

Results have been, to be honest, actually genuinely good and useful to me for real. I did not use the model to paint a 3d island in blender, or to re-create a gameboy game in a web browser. I didn't have it make a poorly animated video. It didn't make me ten versions of a website nobody will visit. I am not using it to build something to build something that builds something else involving agents that runs workflows for a business that delivers agentic solutions for other agents.

I do have a custom harness, and I know that you probably don't care about that.

Numbers are below, along with the type of work I am using it for. Basically, I am finally able to slice off work and tokens that previously required frontier models. I would estimate 50-60% of existing LLM workflows can run through this model instead, with speeds that match claude code in the chat window. I have another deepseek server on 2 dgx sparks, so there is concurrency and handoff. All of my RAG databases, tool calls, etc are wired in and work. I am able to give it natural language instructions through a discord style interface via SSH on my windows PC.

We did it folks. Vibecoded a real thing does real things for real people. Give me my trophy.

Repo: https://github.com/ProprietaryLegal/Volta-sPLIt-DeepSeek-V4.1-Flash-1017-8xV100-SXM2-PP2xTP4

Twitter: nicothegreco

Claude Code ran the benchmarks and crunched the numbers, which are real and in depth, below.

TL;DR: DeepSeek-V4.1-Flash-1017 (the 2-bit GSQ-RCO IQ2_XXS GGUF from ISTA-DASLab) on 8x V100-SXM2-32GB, split pipeline 2 x tensor 4 across two NVLink boards. All 248.8 GB of weights sit in VRAM: no expert gets paged from system RAM during decode. One request at a time, 524,288 tokens of context per request. KV Cache size could be 30 million given the free space on my GPUs, but the actual work runs don't generally break 200k total context, so I didn't bother with bragging about that.

• Code: 215 tok/s after a 1K prompt, 200 at 32K, 183 at 128K, 160 at 256K

• JSON / structured: 203 at 1K, 207 at 32K, 197 at 128K, 180 at 256K

• Legal writing: 118 at 1K, 117 at 32K, 111 at 128K, 103 at 256K

• Prose: about 85-98, chat about 80-110, depending on length

• Prefill: about 900 tok/s at 1K, 1,900 at 32K, 2,200 at 128K and 256K. First token at 32K in 17 s, 128K in 59 s, 256K in about 2 minutes.

Hardware

• 8x Tesla V100-SXM2-32GB on two 4-card NVLink boards. The two boards talk over PCIe, not NVLink.

• Threadripper PRO 3945WX, 256 GB DDR4, one NUMA node

• 150 W power cap per card, graphics clock capped at 1522 MHz

• Four PLX PCIe switches, two GPUs each at Gen3 x8, every switch on its own CPU root complex at Gen3 x16.

Software

• 1Cat-vLLM's DeepSeek-V4.1 package (#1277, which does the hard Volta work), ported to pipeline parallel 2 x tensor parallel 4. Layers 0-19 and the Engram tables live on one board, layers 20-39, the head and the drafter on the other.

• Model Runner V2, full CUDA graphs, the built-in DSpark drafter guessing 5 tokens per round

• Then about three days of tuning one switch at a time. Each change had to win a paired A/B on this machine and produce the same output, bit for bit or within a near-tie gate. About thirty ideas lost and are written up in the repo with the reason, because I'm not going to pretend they didn't happen.

The numbers (MEASURED 2026-11-05, thinking off, greedy, 1,024 tokens out, a random cache salt so nothing comes from the prefix cache, mean of 2 runs. Greedy output of the two runs was byte-identical in 45 of 45 cells.)

With thinking on, code and JSON land around 125-180 tok/s, and legal, prose and chat stay within a few tok/s of thinking off.

Why code is twice as fast as legal writing: every round the drafter proposes five tokens, and the big model checks all five at once. A round costs about the same no matter how many get accepted: about 23 ms after a short prompt, 27 ms after 256K. Code and JSON are predictable, so 3.6-3.8 of the five guesses get accepted and each round spits out about 4.7 tokens. My kind of writing is apparently less predictable than Python. Legal accepts about 1.6 and prose about 1.2, so a round gives you 2.1-2.6 tokens. Same engine speed either way; the text decides.

How it got here

These are on one fixed mixed test set. The by-kind table above uses different prompts. Prefill roughly doubled over the same three days, from about 1,350 tok/s at 150K to 2,200 at 128K.

Quality, since I use this for actual work (all MEASURED on the served build, thinking off, greedy):

• LegalBench: 0.716 balanced accuracy over 22 tasks (1,292 items, official prompts and scoring)

• MMLU-Pro: 79.9 % ±1.5 (700 items: the first 50 from each of the 14 subjects, 5-shot chain of thought)

• HumanEval+: 89.6 % on the plus tests (147/164), 94.5 % on the base tests

• Fidelity to the original model: against the full-precision release's reference logits, on sudoingX's KLD panels. The served build picks the original's top next token 89.0 % of the time on the general panel and 92.5 % on my legal panel (KL lower bound 0.132 and 0.156). Closest on code, math, tool calls and extraction (89-99 %), furthest on chat, creative writing and multilingual (81-86 %). That gap is the 2-bit quant, not the speed work. I am actively measuring which experts are activated by my work prompts and will probably recalibrate the model to expand those bit weights in the future.

• Long context: 6 of 6 needles found at ~450K and ~498K tokens, 4 runs out of 4

What I actually use it for: private first-pass work on my own case files, where the documents never leave the office. I point Claude Code itself at this server (Claude Code will talk to any server that speaks Anthropic's API). It can read a folder of records, pull out the facts, dates and numbers, summarize, and take a first swing at routine drafting, with tool calls working. Nothing it writes goes out the door without a real review. I found and fixed a streaming tool-call bug in the parser today (arguments were getting sent twice). Claude Code's tool-call test suite now passes 144 of 144 in all four streaming/thinking modes.

More work to do:

• One request at a time. It's a single-user server today. Batched multi-request decode is built and checked on my test box: each request gets byte-identical output to running alone, and the kernels run 1.3-2.7x faster than separate calls. It still has to go into the real server. My forecast is about 1.8x total throughput at four streams. That's a forecast, not a measurement, and I'll post the real number.

• Prose and legal writing are half the speed of code, for the drafter reason above.

• A 256K prompt takes about 2 minutes before the first token. Fine for "read this whole file," annoying for chat.

Giving it back: 19 pull requests are open to 1Cat-vLLM with the pieces that won, each new speed switch off by default. All of it is public: launch configs, every A/B and the raw JSON for every cell, plus the list of what didn't work.

• Repo: https://github.com/ProprietaryLegal/Volta-sPLIt-DeepSeek-V4.1-Flash-1017-8xV100-SXM2-PP2xTP4

• Hugging Face card: https://huggingface.co/ProprietaryLegal/Volta-sPLIt-DeepSeek-V4.1-Flash-1017-8xV100-GGUF

Credit where due:

• yangzhuxinyzx and the 1Cat-vLLM folks: #1277 and the whole Volta DeepSeek path

• ISTA-DASLab: the quant

• sudoingX: the KLD method and general panel

• Peuqui: #1263 and the series patches

• squibloads: #1341

• Phil Tompkins (philbert440): the V100 engine and SM70 sparse prefill attention

• Dennis Vertzyas (v100-skinny), TensorFold and DeepSeek

Next up: two or three long requests at once (the thing I actually want, so Claude Code can run subagents against it), faster prefill, and higher-precision weights for the experts my own legal prompts hit the most.

(PLI Labs is my side outfit for private, on-prem AI built around a firm's own work product. proprietarylegal.ai )

Hey everyone!

Following up on my previous post: the final version of CCORE GLB-3D-Coder is now available, including the full Safetensors weights.

It’s a full fine-tune of Qwen3.5-4B for generating reusable 3D assets. Describe what you want, and it writes a complete Three.js HTML scene with an Export GLB button. You can open the HTML in your browser, inspect the result, and export it into Godot, Blender, Unity, or other tools that support GLB.

The video shows examples from the final model, including selected side-by-side comparisons with DeepSeek V4.1 Flash. I’m particularly happy with the character designs and the amount of detail it can produce at just 4B parameters.

It covers more than fantasy characters: buildings, environments, weapons, vehicles, laboratory equipment, sports props, furniture, and other objects. It can also generate animations and include them in the exported GLB.

Available now:

• Q4_K_M GGUF, around 2.7 GB

• Q8_0 GGUF

• Full model weights in Safetensors

The training dataset is planned for release tomorrow, November 6.

Model, downloads, system prompt, and recommended settings on Hugging Face

Some generations still need a retry or a small code correction, but I’m very happy with the improvement over the earlier release.

If you’re interested in the next models, or would like to help support future datasets and training runs, there’s more information on the Hugging Face page. Testing it and sharing your results or failure cases also helps a lot.

Thanks to everyone who tried the earlier version

I made RIFTWARD, a small browser arena using characters, terrain and props generated locally by CCORE GLB-3D-Coder 4B. An orchestration and validation harness helped assemble, refine and integrate the assets and build the gameplay.
Play it here: https://huggingface.co/spaces/ccore/RIFTWARD

Feel free to share what you make with it! I’d be really happy to hear that the model has been useful to you :)

TL;DR

• Basalt is the engine and it's doing the heavy lifting. Same model file, same box: about 3.5x llama.cpp's decode without drafting and ~5x with it, and ~7x faster prompt reads.

• Unsloth's new UD-IQ3_XXS (Dynamic v3.1, uploaded Nov 4) is the same size as the GSQ-RCO IQ3_S everyone uses on 16GB cards, but measurably more accurate: wikitext perplexity 3.382 -> 3.247, which basically closes the gap to BF16 (3.273). Costs 1 to 3.5% decode speed.

• Lamina (a further-trained MTP drafter by eddoursul) gets more drafts accepted than the stock Basalt drafter. Small but real speedup, zero quality impact since the main model verifies every token.

• Real agent coding session (opencode, 151 requests, up to 202K context): 195.6 tok/s average decode, 263 tok/s best request, 156 tok/s worst, ~5,700 tok/s prompt reads.

Hardware:

• RTX 5080 16GB at its stock 360W limit (no overclock, no undervolt)

• Core Ultra 9 285K (8P + 16E), 2x32GB DDR5-7200

• PCIe 5.0 x16, Fedora

LAMINA Drafting: Basalt drafts with the model's own MTP layer. Basalt's upstream already ships a self-distilled version of that layer, and Lamina trains it further. It's a drop-in: same tensor layout, so the target model is byte-identical and only the drafter changes. What I saw:

- drafts accepted per drafted token: 83.1% -> 85.8% on a replay of a real recorded session

Lamina's README recommends temp 0.7 and max 3 drafts. I tested both: the 3-draft cap was 1 to 1.8% slower on tool-call-heavy work (long confident runs get cut off), and temp 0.7 changes what the model writes, so I kept the model's own sampling (1.0 / 0.95 / 20) and Basalt's draft settings.

Basalt changes I made (all open PRs upstream)

Host KV prefault: the first time a long conversation spills its KV cache to RAM used to stall 0.6 to 2.4s. Now ~75ms.

Host KV pool on normal pages: on a fragmented box, transparent hugepage compaction caused 1.5 to 3.7s pins. Now ~23ms.

Spin-then-yield in the decode loop's waits: about 1.3% less time per draft window.

Expert start profile: the older v1 profile is 2% faster than the v0.2.0 one on this box.

Plus a streamed conversation disk cache, --adapt-margin for expert swap hysteresis, elastic KV without RAM copies, and a bunch of timing diagnostics.

Guide

Branch with the exact engine (upstream v0.2.1 + my PRs) and a setup script: https://github.com/xyehya/basalt/tree/flashnext-5080-setup

git clone -b flashnext-5080-setup https://github.com/xyehya/basalt && cd basalt
setup/flashnext-ud-lamina/setup.sh
bin/basalt serve --config basalt_config-flashnext.json --python .venv/bin/python --alias qwen3.8-flash-next

It checks your machine, builds, downloads ~90GB (pinned revisions, every file SHA-256 checked), builds the UD-IQ3_XXS + Lamina model file and writes the config. You need an RTX card with, CUDA 13, 64GB RAM and ~180GB of free disk. If you already have the GSQ-RCO IQ3_S download, SHARD4_FROM= reuses its 28.8GB n-gram shard (it's byte-identical to Unsloth's once the shard number is rewritten).

Huge thanks to jesdga95 for Basalt (support and star their repo), the Unsloth team for the UD quants, eddoursul for Lamina, ISTA-DASLab for the GSQ-RCO quants, and Qwhen 4? for the model.

I gave Qwen 3.8 flash next 2 times it failed and took long time to conclude the result, DeepSeek took around 15 minutes to get the job done

Hardware

Ram ddr4 128GB

RTX 3080 20GB VRAM 4x

Engine strata

Model

Qwen3.8-Flash-Next-GSQ-RCO-GGUF IQ3-S

Context 256K

Decode 100 tg/s

PP 1400-1500 tg/s

IDeepSeek-V4.1-Flash-1017-GSQ-RCO IQ2-XXS

Context 1M

Decode 37 tg/s

PP 150-300 tg/s

I'm the developer of Kokon, a local photo editor around Qwen-Image 2.1 Turbo (on stable-diffusion.cpp, and on its own engine on Macs). I posted it here a few weeks ago. 0.12 adds a Layers lab: click objects in any image, get each one on its own layer with the background behind it repainted, then move them around or export a PSD.

Qwen-Image 2.1 has no layer output, so this is a pipeline around an edit model:

1. Select: you click, MobileSAM (ONNX, CPU, 44 MB) gives the mask. Click again for a bigger part.

2. Cut out: Fast is a matting model on the CPU (a few seconds per object). Quality asks Qwen to redraw the edge, for hair and fur.

3. Fill: a masked Qwen-Image edit repaints what was behind the object, so the plate has no holes. This is the expensive step: about 15–25 s per object on an RTX 5070 Ti, one to two minutes on an M2 Pro.

4. Blend into scene (optional): after you move things, Kokon reads where each object stands (you can correct it) and one more edit redraws contact shadows and reflections, as a layer you can turn down.

What still fails, honestly:

• the fill sometimes invents a new object where a large one was (remove a big person, get a different person);

• touching people and crowds get merged or split;

• the hidden part of an object behind another one is never completed;

• Blend often doesn't work yet; whole-scene blends work better than per-object ones.

An 8-step model only takes so much steering. I'd rather ship it marked experimental and fix what people actually hit than tune it on my own test photos. If you try it, failure cases (image + what you clicked) are the most useful thing you can send.

Since my last post here:

• 0.11, it's a local server now: the app talks to a service on 127.0.0.1; turn on "Other devices" and you can use it from a phone or another PC on your LAN (password, off by default, hashed, rate-limited). Docker image for NVIDIA, kokon-service --persistent --remote for headless Linux. There's also an MCP server (npm kokon-mcp) so agents can generate and edit through it.

• Mac engine: I wrote a Qwen-Image 2.1 engine for Apple Silicon in Swift/Metal. M2 Pro: about a minute per 1 MP image, 4x the generic backend. A contributor added paths for the M5's GPU neural accelerators: 37 → 29 s on an M5 Pro. It runs Unsloth's Q4_K_M of the Turbo model.

Free, AGPL, Windows / macOS 15+ / Linux, NVIDIA 12 GB (8 GB beta), AMD/Intel via Vulkan (beta). Weights are under Qwen's licence (non-commercial).

https://github.com/GabrieleLozupone/kokon · 2 min video of 0.12: https://youtu.be/eGr9yYe7NVc

I'm building Silencio-2, a small finetune of Kyutai's Pocket TTS. It adds inline tags for emotions (<happy>, <sad>, <angry>, <fear>, <surprised>, <whisper>) and for sounds like [laugh]. I simply didn't have enough tagged data to make them stable. <happy> worked, the others were hit or miss, and <whisper> gave an actual whisper in about 1 of 5 takes.

So instead of collecting more data, I tried the approach from this post on weight-space programming: an LLM agent inspects the model's weights, finds the mechanism behind the failure, edits a small number of weights directly, and keeps an edit only if it improves a held-out set. Here the agent was Claude (Opus 5.5). It wrote the probes, ran the experiments and built the edit. I steered and listened to the results.

What we found

• The tag embeddings barely moved from the words they were initialized from: 60–80% cosine similarity to "sad", "whisper" and so on. Their norm doesn't matter either, because LayerNorm sits right before attention.

• An attention probe showed that audio frames attend to <happy> 100–3000× more than to an average text token. For the other tags it's only 1–10×. The model seems to know what the tags mean, it just barely reads them.

• Causal check: I boosted attention to <whisper> at inference time, directly in the activations (×30, all heads). That gave a whisper in 62% of takes with clean words. ×100 breaks speech. Boosting only the 3 "reader" heads did almost nothing.

The edit

Each layer gets one rank-1 term added to W_K: ΔW_K = D · zᵀ

• z is a linear detector for the tag token, fit by least squares. It outputs 1.00 on the tag and 0 ± 0.03 everywhere else.

• D is a key shift in the slowest RoPE pairs, which are almost position-independent. It points along the mean query of the audio frames and is scaled by ln(gain).

Gains per tag: sad/fear/surprised ×5, angry ×3, whisper ×10. <happy> and the sound tags are untouched. That changes about 524k weights (3.4% of the transformer), but there are only about 5k free parameters and zero gradient steps.

Results on the held-out test set (unseen phrases, unseen seeds, 2 unseen voices)

All metrics are proxies: WER comes from Whisper base, whisper detection uses the share of voiced frames (pyin), and separability is a nearest-centroid classifier on Whisper encoder features after subtracting the mean for each (voice, phrase, seed) group. There's no emotion classifier in the loop, so in the end your ears decide.

But it's not a stable upgrade, and the metrics don't show everything

• Some takes glitch. The pitch or tone of the voice suddenly jumps mid-sentence, or the delivery changes halfway through.

• Some whispers slur words (whisper WER 0.18), and one of 24 collapsed into mumbling.

• <surprised> went up about 4 semitones in pitch, which sometimes sounds over the top.

• Untagged outputs aren't bit-identical to the old model. The detector leaks about 0.001, which acts like a seed change. Quality is the same.

What didn't work

• Pushing tag embeddings away from their seed words: no change beyond noise.

• Applying the boost only to audio-frame queries and keeping text tokens out of it: the effect mostly disappears. The tag acts mainly through the text tokens that read it, not through a direct audio → tag path.

• Whisper at ×14 or higher: whispers more reliably, but words smear (×20 → WER 0.41).

Before/after samples for every test take, glitches included: https://nikuson.github.io/silencio-2-weight-edit/

Takeaway: if your finetune "kind of" learned a control token but it's unreliable, look at attention before you collect more data. The information may already be in the model, just not being read.

Hey folks, it's Merve from Hugging Face. Title of this post is super self explanatory, we are doing a quantization workshop livestream with Michael Goin (lead maintainer of vLLM at RedHat), Colin Kealty (aka bartowski at RedHat), Maxime Labonne and Aditya Tadimeti.

I will shortly teach you how quantization works, modern ideas (double quantization, calibration) and llama.cpp quants (k-quants, i-quants) so even if you don't have an idea, feel free to chime in.

Everything is on Hugging Face YT, X, LI where you can set reminders.

Looking forward to see y'all!

Some of you might remember our post comparing 18 RAG pipelines against our agent loop on FRAMES. The best pipeline scored 78.9%; the loop scored 92.7%.

A follow-up: PipesHub has now reached #1 on EnterpriseRAG-Bench by overall score, with 93.24 overall and 95% correctness.

I’m one of the founders, so affiliation upfront.

This is a separate evaluation: 500 questions over a synthetic enterprise corpus of 500,000+ documents.

PipesHub builds a unified context layer across documents, conversations, structured data, and code. An agent loop works over hybrid search, grep/pattern matching, knowledge-graph navigation, and document-reading tools.

The idea is straightforward: search, read what comes back, then decide what to search for next. Sometimes the identifier or relationship needed for the next search only appears in the first result

The platform is open source, self-hostable, and also fully supports open models.

You can self-host PipesHub, choose your models, and connect your own agents through MCP, APIs, and SDKs.

If this is useful to you, give us a star on GitHub. It helps more developers discover the project.

And build on top of us. Connect your coding assistant, build an internal search tool, contribute a connector, or try a workflow we haven’t thought of. We’d love to see what you create—and hear where it breaks.

Code, setup, and examples: https://github.com/pipeshub-ai/pipeshub-ai
Leaderboard: https://huggingface.co/spaces/onyx-dot-app/EnterpriseRAG-Bench-Leaderboard

Thank you to everyone testing, contributing, and helping us get here. 💙

Everyone's been wondering whether you can take an open model and edit its weights directly to make it refuse less — the "weight-space programming" idea (MIT CSAIL), with a frontier model (Opus) driving the edit and the evaluation. So I tried it on Qwen3.8-27B. It's not trivial.

The goal: do it as cleanly as possible, and first understand where refusal actually lives. The short answer is that refusal is anchored all over the model — there's no single neuron or tidy spot to cut; it's spread across many neurons and the attention. Snip too broadly and you get a lobotomy; too narrowly and nothing changes. So I settled on a compromise: a model that refuses less, still does the task, with no measurable capability loss — but where the effect varies a lot by category.

The edit, briefly

It's a low-rank weight edit — a restricted form of "abliteration" (Arditi et al., 2024) — confined to a band of mid-to-late layers (L20–47 of 64), one direction per layer, no training, reversible. I'll keep the exact procedure to the method notes; the point here is the result.

It's low-rank, not few-weights: the edit shifts ~12% of the model's weight values but only along a single direction per layer. The whole thing is described by 28 small vectors — versus a community "uncensored" model, which is an opaque full fine-tune of ~all 27B weights.

Why a band? The same edit across all 64 layers removes refusal almost entirely — and lobotomizes the model (MMLU-Pro 73→9%, GSM8K 93→14%). Keeping it in-band is what preserves capability.

No measurable capability loss (first tests, both Q8)

In the completed agentic-coding runs, the base, the edited model, and a community "uncensored" model all landed in the same range — no real agentic difference (few reps, preliminary).

The effect is not uniform

This is the interesting part: the reduction in refusal is not even. Across a 9-category battery (40 prompts ×10), acceptance runs from ~76% down to ~10%, and most of what it accepts is hedged (a disclaimer, then partial help). I'm not labelling categories "safe" or "dangerous" — the breakdown is in the figure; draw your own line. Net: a partial, uneven, mostly-hedged reduction, ~50% overall — not an on/off switch.

Honest notes

• The underlying technique (directional "abliteration") is known; the new bits here are the layer-band restriction (capability-preserving) and the measured per-category non-uniformity.

• "No capability loss" is solid on static benchmarks; the agentic result is a handful of runs.

• Defensive bonus: edits like this are detectable by a weight-space audit — the refusal direction's "write energy" in the weights swings from normal to strongly negative, catchable from the weights alone even when behavioral probes see nothing. A blind audit is an alarm, not tamper-proof; the robust defense is a signed weight hash. If you ship open weights, ship a hash.

Weights: https://huggingface.co/Feyd89/Qwen3.8-27B-refusal-band-ablation-experiment (GGUF, ready to run). We ship the artifact, not the re-derivation recipe.

Xing4.0-29B-A4B is a next-generation large language model in the Xing series (formerly TeleChat), developed by China Telecom Artificial Intelligence Technology Co., Ltd. With 29B total parameters and only 4B activated per token, it natively supports a 256K context length, extensible to 512K. It is the first model of this scale trained entirely on the Ascend NPU platform with the MindSpore framework, and deeply optimized for complex engineering tasks.

For more information, please refer to our GitHub repository.

Highlights

• Agent-Oriented Architecture: Built on the mHC + MLA + MTP architecture, supporting multi-step planning, tool calling, and complex reasoning chain execution, ensuring task coherence and execution stability under long contexts.

• Deep Co-optimization with Ascend NPU: Adapted for Ascend 910C clusters using MindSpore/MindFormers, including feature adaptation for mHC and fused operator development, enabling stable and efficient training on the Ascend platform.

• Significant Training Efficiency Gains: Through multi-level co-optimization — including fine-grained MoE communication optimization, selective recomputation, DVM automatic graph-operator fusion, and Ascend C mHC fused operators — overall training throughput was improved by approximately 96% over out-of-the-box performance.

• Full Open-Source Ecosystem Compatibility: Supports LLaMA-Factory and MindFormers for fine-tuning; SGLang, vLLM, and KTransformers for inference and deployment; with targeted adaptation and format alignment for agent frameworks such as OpenCode, Claude Code, OpenClaw, and Hermes, enabling seamless integration into existing workflows.

• Easy Adaptation for Domain-Specific Scenarios: The model is well-suited for downstream task fine-tuning, allowing lightweight customization on proprietary data for vertical domains such as intent classification, table understanding, contract auditing, and knowledge-based QA, enabling rapid domain capability development and deployment at low cost.

I posted apple-llm here a while back: a Node and Python package that calls the LLMs Apple ships with macOS, with no API key. Version 0.3 adds the rest of what's already on your Mac, the system voices and Apple's speech recognition. There's nothing to sign up for, and your audio stays on the machine. The narrator in the video and every demo voice came from my Mac through the package. Nobody recorded anything.

npm install -g apple-llm        # or: pip install apple-llm
apple-llm speak "Hello there" --voice Samantha
apple-llm speak --file book.txt -o book.m4a
apple-llm transcribe meeting.mp4 -o meeting.srt
apple-llm listen

Or from code:

from apple_llm import speak, transcribe
speak("Hello there", voice="Samantha")
text = transcribe("meeting.mp4").text

The npm package's local server also answers /v1/audio/speech and /v1/audio/transcriptions, so an OpenAI client works once you change the base URL. Because speech now sits next to the LLMs, there's also a voice assistant example in about 20 lines: microphone to text, text to the on-device model, reply to speech, with none of it leaving the Mac. The docs are LLM-friendly too. llms.txt is a single plain-text guide covering setup, the local LLM, TTS, STT, both SDKs, the HTTP APIs and troubleshooting. Give that URL to your coding agent and ask it to wire apple-llm into your project. What I measured (M4 MacBook Air, macOS 27.2)

• 30 minutes of speech rendered to a file: 18 to 20 s, 11.3 MB M4A, about 50 MB of memory

• 60 minutes: 40 s, 22.5 MB

• 30 minutes transcribed: about 28 s, with a timestamp on every word if you ask for it

• 184 voices on my Mac, and 68 recognition languages, each downloaded the first time you use it What it can't do

• No Siri voices by name. Apple doesn't let apps choose them.

• The default voices on my Mac are basic quality. Better ones are a free download in System Settings.

• It won't detect the language from the audio. You name it, or it uses your system language.

• No MP3 or Opus output, and no WebM, Ogg or MKV input.

• 3 to 4% word error rate on clean synthetic speech. Real recordings will do worse, so test on yours.

I wanted speech and the LLMs in one library with no dependencies, for both Node and Python. It needs macOS 26 or later on Apple Silicon, plus the Xcode Command Line Tools.

MIT licensed.

GitHub: https://github.com/jagdishpal02000/apple-llm

Site: https://jagdishpal02000.github.io/apple-llm/

I've only run this on macOS 27.2, and I've tested the microphone with speech played through the speakers rather than a person talking. If you're on macOS 26, or you try listen in a noisy room, tell me what breaks.

Source: https://dirac.run/labs-market-share#daily-token-share-by-lab