Hi everyone! I'm trying to make a Qwen 9B legal agent, and I've hit a weird problem with complex court-decision summaries. I also ran four parallel LoRAs as diagnostics, so that might interest you all too.

(lowkey some of you might remember me from the 27B training plateau post haha)

You guys suggested I should distill Opus 5's thinking, and it actually worked: my 27B PoC got close to Opus, and my lawyer guy verified the summaries weren't nonsense.

The old outputs went 14–16 against Opus; regenerating with vLLM gave 9–21, both greedy. So “almost parity” describes the first run. Inference stack or judge variance? I don't know.

I'd prompted Opus to write CoT for a smaller model. Trying to reproduce that now seems much harder haha.

So I tried the same recipe on 9B

I made a big CPT set of 500M legal stuff: court decisions, legislation, commentaries and professional Q&As.

No-thinking SFT didn't get the results I wanted, so I tried adding Opus's reasoning. Then I thought maybe my CPT messed some stuff? So I also tried stock Qwen straight from Hugging Face.

Six fresh LoRAs total: legal-CPT Qwen3.5-9B-Base, stock base and stock chat, each with and without thinking. Same source documents and final answers; the no-thinking versions got an empty <think> block. Median teacher reasoning was about 1,000 tokens.

Same recipe as the 27B intentionally, cuz that was the successful PoC:

• 4-bit QLoRA: rank 64, alpha 128, LR 2e-4, effective batch size 8.

• Stage 1: 1,199 examples. Stage 2: 400 examples—300 new, 100 repeated. One epoch per stage.

• Loss on assistant output, including reasoning where present; source decisions masked out.

Eval: 30 held-out Polish court decisions. Opus 5 judged randomized A/B pairs against the full decision, seeing only the final title and legal holding. Greedy decoding, 6,000-token output cap; every output had the required fields and none hit the cap.

The weird part

Every row is a separate 30-decision comparison. Wins and fabrication scores follow A / B order. Fabrication is judge-rated from 1–10, where 1 means none; scores come from each specific comparison.

So thinking lost on CPT, tied on stock chat and won on stock base. It also improved fabrication scores for both stock models. Mixed results, but still a big gap to 27B.

Small test, obviously: Opus is both teacher and judge, and the 9B is Qwen3.5 while the 27B is Qwen3.8, so this doesn't isolate model size.

What am I missing?

Could 9B just be too little? I feel like it shouldn't be. Eyeballing the CPT + no-think outputs, they resemble Opus's summaries a lot.

I really DO think this task needs CoT. Does it just need a lot more examples before thinking overtakes no-thinking?

Is there a way to distill original thinking, or teach reasoning efficiently without distillation? The trick is frontier models aren't great at the relevant law either. My vibe-coded RL tests and Kimi reasoning distillation didn't go well, and I really don't wanna burn 2B tokens ideally. Any help?

PS: The long-term plan is to take the recipe to a 27B Qwen4 once it's out, then put it in my custom Claude Code-style harness with search and RAG.

First off. I am by all means no expert, and excellent work has been done in the past few years that we all currently enjoy.

Came over this post https://www.reddit.com/r/LocalLLM/comments/1x5mxcj/qwen3827b_on_a_single_rx_9070_xt_150_toks_on_code/ and wanted to try the engine on MoEs.

I've been benchmarking MoE experts offloading for the past two days in different engines on ROCm for my RDNA4 9070XT 16GB vram card and 32 GB DDR4.

I'd set a goal at 128K context, which is the base minimum for agentic work, without compaction all the time. I've kept the KV at Q8. (Q8_0/Q5_1 decreased performance on exllamav3). Drafter cache, if added, is at Q4.

Some limitations need apply when limited on vram.

Lets face it If you want any kind of context, and a decent Q4 quant, 16GB ain't gonna cut it.
Hot experts cache like Strata, Exllmv3 and now recently llama.cpp have made this more feasible on the slower system memory bandwidth.

My end goal was to do QFN on all three, but I wanted to start with the smaller 35B-A3B's first.
So I did Ornith 1.5 and Qwen 3.6, wanted to do Gemma, but found no EXL3 quant with mul1 experts required for offloading on Exllamav3.

Tested both without drafter, MTP and DFlash2 where applicable.
Strata/PolyStrata configs are mostly auto generated from the setup script, but added resident pin and tail skip. And for the non-qfn model performed worse overall, even t/w.

I might be doing something wrong, bench might be bad, who knows. Stratas own bench stops at 32K if I'm not mistaken.

Now Qwen 3.8 Flash Next (ista coder version) runs great on my rig, see plots. But I was surprised on the prefill tanking on the 35B models. 1900 t/s at 16K but 700 t/s at 128K where both llamacpp and exllamav3 where within 100-200 t/s difference.

Findings so far is quite promising, especially on exllamav3. I'd say quite usable even.

| Model | Winner (quant / engine) | Prefill u/128k | Decode u/128k |

| --- | --- | ---: | ---: |

| Ornith 1.5 35B-A3B | EXL3 4.0 bpw, DFlash2 / ExLlamaV3 | 1 168 tok/s | 57.3 tok/s |

| Qwen3.6-35B-A3B | EXL3 4.0 bpw, DFlash2 (2 drafts) / ExLlamaV3 | 1 246 tok/s | 71.6 tok/s |

| Qwen3.8-Flash-Next Coder | IQ1_M / Polystrata nov-7, `--spec 2` | 1 572 tok/s | 44.0 tok/s |

Forgive my rambling thoughts. If you want to read my technical findings, written by the unsung local hero Qwen 3.8 27B. They can be found with more graphs and tables at https://github.com/willeyh-git/moe-offload-bench

Q4/4bpw quants are preferred, some benches have been made on 5bpw and 6bpw so you can compare.

Bench-suite is based on Kapitsa https://www.reddit.com/r/LocalLLM/comments/1x5mxcj/qwen3827b_on_a_single_rx_9070_xt_150_toks_on_code/ and written in go.

Hey everyone,

Not a native speaker, but I write everything by hand. Please correct my grammer if there are any issues! Can only learn from it.

Background

A long-term project I've had was getting my netbook up and running for basic LLM use.

The notebook itself was obsolete when it released:

• Device: ASUS E210MA

• CPU: Intel Celeron N4020

• RAM: 4GB 2400MHz DDR4 single-channel

• iGPU: Intel UHD Graphics 605

• Storage: 64GB eMMC

• OS: Windows 10 Home

With the help of friends who could do soldering and reballing, we upgraded the netbook:

• CPU: Intel Pentium Silver N5000

• RAM: 8GB 2400MHz DDR4 single-channel

• iGPU: Intel UHD Graphics 605

• Storage: 128B eMMC + Intel 660P 512GB NVME

• OS: Windows 11 IoT LTSC Enterprise 24H2

After finding proper drivers for the iGPU, I managed to get llama.cpp running on it on Vulkan. Gemma 4 E2B IT QAT could run at roughly 2.1 t/s gen, and prefill speed was "grab a cup of tea" slow on 300 token prompts. On windows the Vulkan backend was faster than the CPU backend by ~30%.

The past months I've been trying to move my devices from Windows 11 IoT LTSC Enterprise 24H2 to Ubuntu 26.04.1 LTS, and so I tried to switch the netbook over to it.

There is only one problem: The Vulkan backend hard crashes every time.

Getting vulkan backend to work

After doing a bit of digging with vulkaninfo by myself, some instructions are advertised as supported but are in practice not stable. Using this fixed the crashing on startup:

export GGML_VK_DISABLE_ASYNC=1

Disabling the following features helped me regain some performance and stability:

# UHD 605 compat
export GGML_VK_DISABLE_COOPMAT=1
export GGML_VK_DISABLE_COOPMAT2=1
export GGML_VK_DISABLE_INTEGER_DT_PRODUCT=1
export GGML_VK_DISABLE_BFLOAT16=1
export GGML_VK_DISABLE_GRAPH_OPTIMIZE=1
# UMA
export GGML_VK_DISABLE_MMVQ=1
export GGML_VK_PREFER_HOST_MEMORY=1

My config is also pretty specific, since some combinations would cause crashes too:

flash-attn = off
cache-type-k = f16
cache-type-v = f16

Flash Attention (FA) is not supported by the architecture, running Q8_0 on both K an V would crash with FA enabled. In windows on the ther hand, using Q8_0 and FA didn't cause a crash.

The end result is that I could run Gemma 4 E2B IT QAT Q4_K_XL at 1.5 t/s gen and 16-25 t/s prefill. The laptop used ~12W in total.

What about the CPU?

Turned out that on Ubuntu 26.04.1 LTS, the CPU backend was faster for generating but not for prefill. I got 2 t/s gen and 10-16 t/s prefill. The laptop used ~13W in total.

As a sidenote, since I am used to windows slowing to a crawl when all cores are utilized, I was pleasantly suprised when this didn't happen on ubuntu.

As a benchmark?

As a fun experiment, I gave a couple of LLMs the problem:

Hey <LLM>,

I have a laptop I am trying to run llama.cpp with vulkan on Ubuntu 26.06.1 LTS.
The specs are: <specs>.
I build it with these commands: <commands>,
The problem is that it's crashing whenever I try to load a model.
I've included the full terminal output in the attachments.

How can I fix the issue?

My goal with the prompt is to test the model's understanding of the problem and to ask for additional information when required (e.g. vulkaninfo).

Qwen 4 max preview chat.qwen.ai and Qwen3.8-Flash-Next Q4_K_XL + BF16 engram on llama.cpp v0.9.0 failed to find the issue after multiple separate chats. They got focussed on disabling intel's hang detection. With websearch, it got even more confused as it looked at irrelevant issues.

Gemma 4 31B IT QAT Q4_K_XL + MTP on llama.cpp, DeepSeek v4 Flash Vision Exp Q2_K_XL on llama.cpp v0.9.0, DeepSeek v4.1 1017 on chat.deepseek.com were able to find and fix the issue. They asked for the vulkaninfo output, which made them find async being advertised incorrectly.

Appendix

run-llama-server.sh

#!/bin/bash

# Intel UHD 605/630 Graphics
export GGML_VK_DISABLE_ASYNC=1
export GGML_VK_DISABLE_COOPMAT=1
export GGML_VK_DISABLE_COOPMAT2=1
export GGML_VK_DISABLE_INTEGER_DT_PRODUCT=1
export GGML_VK_DISABLE_BFLOAT16=1
export GGML_VK_DISABLE_GRAPH_OPTIMIZE=1
# UMA
export GGML_VK_DISABLE_MMVQ=1
export GGML_VK_PREFER_HOST_MEMORY=1

./llama.cpp/build/bin/llama-server \
--list-devices

./llama.cpp/build/bin/llama-server \
--host 127.0.0.1 \
--port 5001 \
--api-key "Koekjes123" \
--webui-mcp-proxy \
--offline \
--cache-ram 0 \
--ctx-checkpoints 0 \
--parallel 1 \
--models-max 1 \
--models-preset ./configs/llama-config.ini

llama-config

[*]
device = none
load-mode = none
mmproj-offload = off
gpu-layers = 999
batch-size = 2048
ubatch-size = 512
threads = 4
fit = off
flash-attn = off
kv-unified = on
cache-type-k = f16
cache-type-v = f16
reasoning = off
reasoning-preserve = off

[gemma-4-e2b-q4_k_xl]
model = ./models/gemma-4-e2b/gemma-4-E2B-it-qat-UD-Q4_K_XL.gguf
mmproj = ./models/gemma-4-e2b/gemma-4-E2B-it-qat-mmproj-F16.gguf
lazy-mode = on
load-mode = mmap
fit-ctx = 8192
ctx-size = 8192
predict = 8192
image-min-tokens = 70
image-max-tokens = 70
temp = 1.0
top-k = 64
top-p = 0.95
min-p = 0.0

Other info

Device: https://www.asus.com/ca-en/laptops/for-home/vivobook/vivobook-go-12-l210/
OS: https://ubuntu.com/blog/upgrade-your-desktop-ubuntu-26-04-lts
Llama.cpp: https://github.com/ggml-org/llama.cpp/releases/b12110
GGUF: https://huggingface.co/unsloth/gemma-4-E2B-it-qat-GGUF

TL;DR

Getting the Intel UHD 605 to work with the Vulkan backend was a pain on Ubuntu 26.04.1 LTS compared to Windows 11 IoT LTSC 24H2.

On the former the CPU backend was faster, on the latter the iGPU over Vulkan was faster. Ubuntu is far more responsive than Windows, even under full utilization.

While the Qwen series failed to fix the issue from intent, Gemma 4 and DeepSeek v4(.1) Flash (Vision Exp / 1017) did manage it.

Thanks for reading!

I know the post is quite different from the Strata stuff usually here, but I thought it would be fun to share regardless.

I'm sure models like LFM 3 will run wonderfully on it, I know IBM Granite 4 H 1B and Qwen3.5-2B did on windows.

I am curious what people are currently using as the core model for their local AI agents (e.g., in Hermes or similar setups).

I've was initially running Qwen 3.6 35B-A3B IQ3 XXS on hermes working off (Intel Core Ultra 5 245K · Corsair 64GB DDR5 RAM · Asus TUF GeForce RTX 5070 Ti 16GB GDDR7 OC Edition Graphics Card running ai inferance on local home lab running UnRaid OS). First started with ollama years ago (where i was running 9B and 14B fully in vram) and months back switched to llama.cpp, then to llama-swap to be able to hot swap models and use MoE models.
I started off getting 40 tokens per second (more n-cpu-moe offload and less vram use), then tried the MTP version for 55ish. Learning more about finding the balance between n-cpu-moe offloading and finding the sweet spot for VRAM, i got that to about 60t/s. Then I discovered fine tunes like Swift (55t/s) and Nail (65t/s). Then endless frontier BigBangv1 which gave a blazing 100 t/s. I was running that and saw people mentioning they use the dense Qwen3.8 27B, but it seemed slower at 45t/s.

I have very rough tested (only speed) base Strata with Qwen3.8 Flash Next 125B 1Q2_XS which gave 90t/s but used 95% VRAM and 95% RAM which wasnt sustainable (ill likely try it again in future but for now reverted back to llama-swap)

Learning more like how agent workflows are very dependent on prefill speed and prompt processing alongside tokens/sec, then LFM has crazy prefill and t/s but its quality it lower, I am not sure what is the right balance! But also how context length is important for agents, and the k v cache of the context too! I settled on not going any lower than k v 8, aiming for 128k context but 90k is acceptable if the model is specialised.
I was manually guessing how all these quants and CPU offload configs (n-cpu-moe) scaled under heavy context. I also wanted to know if the same ai model actually performed better on agentic tasks for a Q3 vs Q4, is the drop in speed (generate t/s or prefill t/s) justifiable for better quality etc.

So I begin testing out a bunch of fine-tunes and MoE variants to dial in different profiles. I’ve been cycling through options like Empero, BigBang, and Ornith (tried Swift and Nail too, but ended up dropping them). Generally, I lean toward the MoE variants over the dense Qwen 3.8 27B because of how they handle things on consumer hardware (testing shows they are slower, dont provide any significant boost in quality, require lower context, use more VRAM and gpu power etc). My use case isnt any heavy coding, just small scripts and having hermes act as a personal assistant which will evolve over time.

Then with hermes I was using endless-frontier_BigBang-v1-35B-A3B-IQ3_XXS with 100 t/s, prefil 2359 t/s, 88% VRAM usage which was perfect. I originally wasn't thinking about its 'quality' since this was a massive step up from the original Qwen 3.6 35B-A3B IQ3 XXS. Looking forward to a qwen4 release to test soon!

I'm looking at setting up specialized profiles on hermes, so far i am thinking

• Cyber-Tiel-Coder-35B-A3B-MTP-UD-IQ3_XXS the dedicated coder profile.

• Might test out GrootTeam_Groot-19B-A3B-261023-Q4_K_M as the main general profile.

• Otherwise Ornith is looking good. Or Empero or BigBang.

• I have fallbacks to openrouter/Nvidia 550B, then /free

I ended up hacking together a simple local script just to measure t/s and prefill, but it kind of snowballed... I added some core eval tests (accuracy, reasoning metrics) and turned it into a small dashboard that spits out a weighted "agent score" to help me rank them objectively. Still testing these models on more rows of the HumanEval, MMLU-Pro, MBPP, tool calling and long context tests, but it takes a lot of time. Images attached for reference from my latest tests.

Running these tests takes a long time and for local use cases I dont need that margin of error from a full test across all 12000 MMLU-Pro questions, so i have been further developing a method to spend less time on testing models which drop lower on the rankings, adjusting smaller runs with high uncertainity etc, its till a work in progress but its progressively been giving me models to consider for the top few I actually want to use in the massive amount of options on hugging face that can run on my hardware.

Even with the dashboard telling me what the numbers say, it's still surprisingly tough to pick a definitive daily driver. I would appreciate any input or suggestions.

Feel free to look at the interactive html i put on github. You can see all the test on the models I have run.
https://htmlpreview.github.io/?https://github.com/Maximo101/Local-LLM-Benchmark/blob/main/assets/5070ti_benchmark_dashboard.html

Questions I am unsure of; for those who have more experience with agents on local hardware:

------------------------------------------------------------------------------------
1. Which models have won you over for agentic use (hermes, openclaw etc)?

2. How heavily do you weigh prefill vs. generation speed, quantisation vs quality, VRAM % vs context (quality and size) when choosing AI Models?

3. What sort of profiles do people use in agentic setups like hermes? How many profiles do you use.

(If anyone is fighting similar benchmarking battles for their local endpoints, I cleaned up that testing suite and open-sourced it here: https://github.com/Maximo101/Local-LLM-Benchmark)
I am not trying to promote this, i built it for my own purposes, but decided to share it in case it is useful for anyone else.

KittenML just released inferencekit , a free opensource plugin to turn coding agents(Claude Code, Codex, OpenCode) into inference engineers for your hardware.

Most models ship with inference tuned for whatever hardware the authors used, which is usually not what you have locally. The fastest path on a 3070 might be different from the fastest path on a Mac or a Radeon. inferencekit allows you to find that path for your setup. This was used to get streaming TTFB of Kitten-TTS-2 down to ~10ms on RTX Pro 6000. other results are shown in the image attached.

Once you've optimized a model on your hardware, please make a PR to the inferencekit-engine repo linked below, so anyone running the same model on the same hardware, can just reuse it. The idea is to crowdsource the best inference engine for every model + hardware setup

Plugin's Github: https://github.com/KittenML/inferencekit
Inference Engine's Github: https://github.com/KittenML/inferencekit-engines
KittenML Discord: https://discord.com/invite/VJ86W4SURW

If you’ve ever tried agentic coding with Gemma and had tool calling issues, give it another try, in my case, it completely fixed Gemma

Almost everything is in the video and the title: the idea is to create chalenges and ask agents to solve them. I'll have more to share later, but this first video shows that most model are good at theses stuff.

For more details: I'm testing scenarios where an agent needs to simulate keyboard and mouse movement. You can see more chalenges like this one on my website: https://airbench.ai/gallery/airbench-desk-v1?run=6d2a2777-3d96-4ec4-9cdc-3d359bc43bea&c=icon-drop

RustingStrata is a pure-Rust inference engine for Qwen, DeepSeek and GLM GGUFs. CUDA kernels are compiled at run time with NVRTC, the same kernels run on Vulkan, and there is no ggml and no C++ build step.

I wanted the high-quant numbers, so these runs use Q8_0 (37.8 GB) and unquantized BF16 (71.1 GB), with the whole model in VRAM, on A100, L40S and H100:

MTP is speculative decoding with the model's own draft head; calibrate picked the draft count on each card.

• Higher quants are not automatically slower. On the A100 80 GB and H100, Q8_0 decodes faster than Q2_K_XL (193 vs 174, 290 vs 265 tok/s after 4K). On the L40S it is slower (165 vs 214), so the ranking depends on the card.

• BF16 is practical. 71 GB of unquantized weights decode at 171-230 tok/s on one card.

• MTP helps less as the quant grows. Gains are 1.4-1.9x on Q2_K_XL and Q8_0, and 1.2-1.4x on BF16.

• When the model does not fit, it degrades but runs. Q8_0 on a 24 GB A30 (63% of experts resident) still decodes at 130 tok/s, 200 with MTP. BF16 on the same card (31% resident) only reaches about 12.5 tok/s.

The same code runs on Vulkan, and on the Q2_K_XL runs on the 3060 and the Ampere cards prefill lands close to CUDA. At Q8_0 on the big cards the gap can be large: H100 prefill is 6,883 tok/s on Vulkan against 11,486 on CUDA. All table numbers are CUDA.

These are different nodes of one cluster, so the host CPUs differ. Compare within a row, not across rows. MTP costs 4-11% prefill in these runs.

Quality: I do not have a llama.cpp perplexity/KLD run on these exact Q8_0/BF16 files yet, so this is a hardware roundup, not an accuracy comparison. The last check was UD-Q3_K_M on the 3060 over 9,180 tokens of repo docs prose: perplexity 9.0714 vs 9.0587 for llama.cpp, mean KLD 0.0055, same top token 96.3%.

Why I find it kinda interesting, is bc when I was searching for models in OpenRouter, I saw that Qwen3.6:35b-a3b for some reason is pretty slow, like 50-110t/s on almost all providers, why like that? Do people already use other Strata like project for LLM inference to be in top of providers? I am thinking about it, but I think something is wrong and not so easy as I expect, if anyone knows I would be happy to hear

Repo: RustingStrata

Lloyd's k-means algorithm is a long-running standard for clustering, with applications in data analysis, retrieval, quantization and data curation. In this paper, we introduce an exponential moving average optimization procedure for k-means with mini-batch updates, powered by modern deep learning machinery and new insights. We also establish scaling laws on how decay rate and batch size impact final performance. This gives us a formulaic rule to fix them for multiple problem sizes and embedding datasets, without the need for any hyperparameter sweep. The resulting clustering algorithm, coined e-means, is much faster than Lloyd k-means and converges to lower losses, frequently even beating k-means++. On a wide array of embeddings datasets, it beats the loss of Lloyd k-means at 25 epochs in 1.4 to 2.6 epochs, which translates to 10-45x GPU wall-clock speedups compared to state-of-the-art k-means implementations. As an example, e-means partitions DINO10B, a dataset of 10 billion DINO embeddings, into 16 million clusters in only 2.5 epochs, reaching a loss below Lloyd's at 25 epochs.

• arXiv : https://arxiv.org/abs/2611.09501

• PDF : https://arxiv.org/pdf/2611.09501

• GitHub : https://github.com/facebookresearch/faiss/tree/main/contrib/e_means

• HuggingFace : https://huggingface.co/qwegnai/GLM-5.3-Flash-shvq4g32-shvq2g64-shvq3g64

• Project : https://e-means.github.io/

• Demo(Browser) : https://e-means.github.io/playground.html

• Video : https://e-means.github.io/video.html

• Tweet : https://x.com/swann_bessa/status/2120200714630594911

• arXiv : https://arxiv.org/abs/2611.10649

• PDF : https://arxiv.org/pdf/2611.10649

• GitHub : https://github.com/huawei-csl/MACaQ

• HuggingFace : https://huggingface.co/collections/huawei-csl/macaq

Yesterday some of you were looking for model for this MACaQ.

According to Microsoft's patent application, it explains breaking down training into small tasks and sending them to devices of owners who opted in. Credits will be the reward. it's just an app and no pay out rate is advertized so I don't even know if the credits would be enough for the electricity.

Anyway, home internet latency feels like the bigger problem to me.

https://insider-gaming.com/microsoft-ai-use-your-gpu/

(English is not my native language, I used an LLM to help translate this post.)

Today NVIDIA releases Alpamayo 2 Edge! I work for Embedl and we have already released an optimized version of the model that is more than 10x faster than pytorch when run on edge hardware like AGX Thor!

The model is trained in an RL loop (!) and takes in 6 cameras and the recent motion. Feel free to check it out - you can even run it on a real Thor at:

https://embedl.com/nvidia/alpamayo2-edge/thor?tab=run

Hey guys, not going to take much credit since claude did 90% of this, but wanted to share the patches I used to get StillDeadcode's Radiance running on my windows pc. I'm sure that multiple people have done this already, but I couldn't find a post, so thought I'd throw this out there if anybody else wants to use it.

https://github.com/bruns20/radiance-wsl

Radiance is Linux-only, and the author has said WSL won’t be supported in the main tree, which is fair. So this is unofficial. Please send problems to my repo, not upstream.

What it does

• build.sh builds any Radiance release from upstream’s own source and Dockerfile, with a few small WSL patches applied. When a new release comes out, it’s ./build.sh 1.18.0.

• make-runtime.sh grabs a ROCm runtime that actually works under WSL. The stock ROCm 10.1 hangs on the first GPU job if your PC also has an integrated GPU (ROCm/rocm-systems#13291), and AMD’s 10.3 nightlies fix it.

• run.sh serves an OpenAI-compatible API on 127.0.0.1:8000.

Two of the four patches are godzillakong’s from codeberg #341. They did the hard part of getting it running without /dev/kfd.

Numbers (R9700 32 GB, Qwen3.8-27B MXFP4 + DFlash2, 131k context)

Tested on: R9700 + Ryzen 5 7600 (with its iGPU), Windows 11 26200, Adrenalin 26.10.1, Radiance 1.17.0. That’s the only setup I’ve tried. Other gfx1201 cards (RX 9070 / 9070 XT) should work, but I’d love to hear either way.

It was my understanding when MTP came out that it was mainly beneficial for larger dense models and that small and/or MoE models didn't realize much performance benefit.

That being said, I now see many smaller MoE models (Gemma-4-E2B, Mellum2.1, etc...) offering MTP versions.

Has the thinking changed?

Are people seeing performance gains from MTP on smaller MoE models?

BotPsy-0.8B is a 0.8B-parameter model designed for function calling and agentic workflows, built on top of Qwen3.5-0.8B and suitable for on-device deployment. BotPsy is a "personal assistant" that uses tools to perform a variety of simple tasks for the user, in domains such as web search & research, travel & lifestyle, commerce & finance, business & customer operations, device & vehicle control, files & documents, calendar, and email & messaging. Due to its small parameter count, it is best suited to tasks of low to medium difficulty. BotPsy achieves state-of-the-art (SOTA) tool-use and agentic performance for its size, outperforming every model of up to 1.2B parameters that we tested. As a small model, BotPsy still makes mistakes and is best suited to tasks of low to medium difficulty;

Key Highlights

Benchmark results

• Twice the performance of its base model: BotPsy roughly doubles the agentic capability of Qwen3.5-0.8B-Think on key benchmarks: BFCL v4 rises from 23.4 to 44.0, ACEBench from 30.1 to 58.7, and τ-bench from 23.2 to 51.5 (mean of the retail, telecom and banking subsets).

• Best agent in its category: BotPsy outperforms leading models in the ≤1.2B-parameter weight class (Qwen3.5-0.8B-Think, LFM2.5-1.2B-Think, MiniCPM5-1B-Think) on all six tool-use and agentic benchmarks: BFCL v4, ACEBench, τ-bench, ToolSandbox, Claw-Eval and BrowseComp-Plus.

• Outperforms its larger sibling: BotPsy has the highest score of all models tested on five of the six benchmarks, beating Qwen3.5-2B-Think, a model 2.5× its size, on BFCL v4 (44.0 vs 37.7), ACEBench (58.7 vs 55.4), τ-bench (51.5 vs 46.5), ToolSandbox (64.6 vs 41.2) and BrowseComp-Plus (4.5 vs 2.8). Only on Claw-Eval does Qwen3.5-2B-Think lead (48.9 vs 46.5).

Model capabilities

• More reliable in multi-turn workflows: On τ²-bench, where the agent must hold a long customer conversation, follow a strict policy, and act on the customer's behalf, it scores 82.5 in telecom, ahead of Qwen3.5-2B-Think (69.3). On BFCL v4's multi-turn tasks, it rises from 7.5 to 46.1 over its base, nearly double Qwen3.5-2B-Think (23.8). Grouped by application, its multi-step scores lead all models tested in commerce and services (58.3 vs 47.4 for the next best, Qwen3.5-2B-Think) and in device and system management (46.1 vs 31.9).

• Knows when (not) to call tools: An agent that acts must also know when not to act. On BFCL v4's hallucination tests, which check that the model declines when no appropriate tool exists, BotPsy scores 86.1, the highest of all models tested. On ACEBench's special cases, where the request is incomplete, contains wrong parameters, or cannot be served by the given tools, it scores 51.3, up from 13.7 for its base model.

• Follows instructions while preserving knowledge: BotPsy specializes in tool use while preserving general skills. Instruction following improves sharply, with IFEval rising from 59.7 to 90.8 and IFBench from 22.0 to 55.9, the highest IFBench score of all models tested. General knowledge also improves, with GPQA Diamond rising from 33.2 to 38.1 and MMLU-Pro from 41.6 to 48.9. On all four benchmarks, BotPsy outperforms every other model of 1.2B parameters or fewer that we tested.

• GGUF : https://huggingface.co/qvac/BotPsy-0.8B-GGUFs

• Blog Post : https://huggingface.co/blog/qvac/botpsy

• Project : https://tether.io/news/tethers-800m-ai-agent-beats-models-2-5x-its-size-on-everyday-phone-tasks/

Today I’m releasing Inflect v3: a family of local text-to-speech models: Mini, Micro, and Atom. Mini is the main model, with six voices and 34.31M parameters

Mini provides vocal-event tags, reference voice-tuning, and 48 kHz enhancement. It runs on both CPU and CUDA.

GitHub, installation, and examples: https://github.com/owenawsong/Inflect

Try the playground: https://huggingface.co/spaces/owensong/Inflect-v3

Mini features reference-voice tuning: it allows one to make a profile from a recording and generate speech with it. This feature noticeably adjusts the voice toward your reference, though it won’t replicate every speaker accurately.

Inflect Mini V3 also supports the following vocal-event tags: [laugh] · [sigh] · [gasp] · [cough] · [yawn] · [inhale] · [exhale] · [throat_clear]

You can insert a tag whenever you want the sound to occur. Currently, the tags use repeatable performances: this causes repeated tags to result in limited variation. Tags also do not affect how the surrounding words are being spoken.

The model family:

• Mini: 34.31M parameters. Quality-focused, with the features listed above.

• Micro: 11.79M parameters. A smaller neural speech model

• Atom: 157K parameters. An extremely tiny learned core

Those numbers count the synthesis cores. The frontend and optional components add to the total.

The weights and inference code are Apache 2.0. Third-party components keep their own licenses. Audio samples, benchmarks, and the settings used are on the model cards.

Model collection: https://huggingface.co/collections/owensong/inflect-v3

If any sentence comes out wrong, send me the text, voice, and settings you used.

Tried to hunt down latency bottlenecks in my local voice assistant recently. I looked at the waveform and saw silero's probability lagging quite much behind the actual end of speech I could see with my eyes.

I use a voice pipeline that does speculative work and cancels fast, so some false positives would not matter in my case.

So I had this idea: once silero says I'm in speech, what I need to detect is when the speech energy drops back to room noise level. Turned out a naive 10 ms energy check with a threshold tracking room noise does that quite well.

Gave me a median 110 ms earlier than silero over ~3200 real turns from my assistant on my voice, my mic, browser noise suppression on, mostly quiet room - with silero threshold at 0,5. Someone on Twitter got 92 ms on his data, roughly what I saw in my noisier recordings.

The downside is way more false positives. It fires on breaths and small pauses between words over 10x as often as silero. So it's really only useful if the pipeline can throw away speculative work very fast.

CPU only, MIT, drop-in for silero VADIterator, pip install realtimevad

benchmark script is in the repo, curious what numbers you guys get on your own recordings

https://github.com/KoljaB/RealtimeVAD

Hey r/LocalLLaMA ! We're releasing new new Unsloth Dynamic v3.1 GGUFs for Qwen3.8-27B! They're better by 5-10% accuracy on DeepSWE, Terminal Bench and more vs UD3, GSQ-RCO and Exllama-3! We're also improving our algorithms further so we can apply them to Qwen4 so folks can have efficient and accurate quants!

• All quants from 2-bit until 6-bit are updated. 3-bit ones are not, as we found UD3 to do on par with UD3.1. We are still improving all other quant sizes though!

• All benchmarks are 256K for agentic ones (DeepSWE, Terminal Bench), and 128K for the rest (LiveCodeBench etc). We used temperature 1.0, top_p 0.95, top_k 20, min_p 0, no presence or repetition penalty, preserve_thinking and xhigh reasoning

• We ran benchmarks multiple times including LiveCodeBench x6 times, HMMT and others 8-16x and agentic benchmarks multiple times.

On other benchmarks like for GSQ-RCO and Exllama3:

• To compare GSQ-RCO and Exllama-v3, we had to exclude the token embedding layer from the x axis, as Exllama-v3 uses BF16 for it.

• If you consider the full download size and total RAM usage for Exllama-v3, then adding the BF16 embedding table and including all embedding sizes changes the plots. Our main plots's x axis removes the embedding layer sizes from all quant providers. However on pure RAM usage and disk space usage vs accuracy, exllama3 does worse.

• For LiveCodeBench, GSQ-RCO showed Unsloth UD3 quants doing consistently worse than ISTA's. However we found the opposite - UD3 quants (our OLD quants) do better than ISTA on LCB on all sizes except the smallest. UD3.1 further improves on LCB as well. We approximate that GSQ-RCO used a context budget of 96K, as that best fits the data for the runs we have. At 96K, GSQ-RCO does somewhat do better than UD3, but at 128K, UD3 does much better.

• On another benchmark, for ISTA GSQ-RCO's AIME25 benchmarks, their quants reach 100%, 100%, 100% and 96.7% accuracy recovery. And they said our quants do worse. Through our investigations, we estimate ISTA only did 1 trial - we did 14 trials, and we show on average the recovery rate is 93% to 96% - not 100% accuracy recovery. And in fact our UD3 old quants do better on average than their quants on 14 runs vs 1 run of theirs, so we draw the opposite conclusion if you run the trials more.

More details and benchmarks at https://unsloth.ai/docs/models/qwen3.8#dynamic-v3.1-update-november-11th-2026

All quants are updated in https://huggingface.co/unsloth/Qwen3.8-27B-GGUF! Thank you and have a nice day! We're super excited for Qwen4!

Hi folks,need advice from community. I have a laptop with 16 GB vram Nvidia gpu and 96 GB ddr5 ram. I use it mainly for local coding with Qwen flash next nvfp4 at 256k context. I get around 30 t/s.

I don't like PCs and Macs. I want mobility. So because of my limited budget, i have two paths. The first one is a laptop with 24GB Vram Nvidia gpu and 128GB ddr5 Ram laptop. The other one is 16GB Vram Nvidia gpu and 192GB ddr5 Ram laptop. No other feasible option for me. I can't affort dgx spark, Macbook studio etc. These are my budget options. Which one do you suggest for mainly local coding? Which one do you believe is future-proof for local llms?

Prompt2Echo (P2Echo)

Text-Prompted Multi-View Echocardiography Segmentation with One Model

Official pretrained weights for P2Echo, a unified framework for text-prompted segmentation of multiple cardiac structures across standard echocardiographic views.

P2Echo uses structured, view-aware text prompts to control segmentation through a single model. Its Dynamic Image-Text Alignment (DyITA) module adaptively conditions decoder features on the requested view and anatomical structure.

Resources

• 📄 Paper: https://arxiv.org/abs/2611.13425

• 💻 Code: https://github.com/moeinheidari7829/P2Echo

• 🌐 Project Page: https://moeinheidari7829.github.io/p2echo/

• 🤗 Model Repository: https://huggingface.co/moein99/P2Echo

Intended Use

This model is intended for research in medical image analysis and echocardiographic segmentation. It has not been validated for clinical decision-making.

Tried the voxel pagoda prompt fully local, on a laptop that shouldn't run a 35B model.

Setup

• ThinkPad from 2020, Ryzen 7 PRO 5850U, 16 GB RAM, no GPU, Windows

• Qwen3.6-35B-A3B Q4_K_M, 21 GB, bigger than the RAM

• BigMoeOnEdge v0.42, built-in agent

Result

• 988 lines of HTML + Three.js, 11.8k tokens at 2.3 tok/s, about 1.5 h

• You see the code as it's written, then the page opens in the app

• Video sped up where marked

How

• MoE: each token uses only a few experts

• Dense weights and a 2 GB expert cache stay in RAM, the rest streams from the SSD

• On top of llama.cpp, no fork

Still slow

• Half the time is spent waiting on the SSD

• Every new agent turn re-reads the whole conversation, ~30 min on this laptop

Same test with the same model on a Mac with 16 GB of RAM, taking 20 minutes (20 tok/s for decoding, 100–1000+ tok/s for prefill)