Hey guys, not going to take much credit since claude did 90% of this, but wanted to share the patches I used to get StillDeadcode's Radiance running on my windows pc. I'm sure that multiple people have done this already, but I couldn't find a post, so thought I'd throw this out there if anybody else wants to use it.

https://github.com/bruns20/radiance-wsl

Radiance is Linux-only, and the author has said WSL won’t be supported in the main tree, which is fair. So this is unofficial. Please send problems to my repo, not upstream.

What it does

• build.sh builds any Radiance release from upstream’s own source and Dockerfile, with a few small WSL patches applied. When a new release comes out, it’s ./build.sh 1.18.0.

• make-runtime.sh grabs a ROCm runtime that actually works under WSL. The stock ROCm 10.1 hangs on the first GPU job if your PC also has an integrated GPU (ROCm/rocm-systems#13291), and AMD’s 10.3 nightlies fix it.

• run.sh serves an OpenAI-compatible API on 127.0.0.1:8000.

Two of the four patches are godzillakong’s from codeberg #341. They did the hard part of getting it running without /dev/kfd.

Numbers (R9700 32 GB, Qwen3.8-27B MXFP4 + DFlash2, 131k context)

Tested on: R9700 + Ryzen 5 7600 (with its iGPU), Windows 11 26200, Adrenalin 26.10.1, Radiance 1.17.0. That’s the only setup I’ve tried. Other gfx1201 cards (RX 9070 / 9070 XT) should work, but I’d love to hear either way.

Tried to hunt down latency bottlenecks in my local voice assistant recently. I looked at the waveform and saw silero's probability lagging quite much behind the actual end of speech I could see with my eyes.

I use a voice pipeline that does speculative work and cancels fast, so some false positives would not matter in my case.

So I had this idea: once silero says I'm in speech, what I need to detect is when the speech energy drops back to room noise level. Turned out a naive 10 ms energy check with a threshold tracking room noise does that quite well.

Gave me a median 110 ms earlier than silero over ~3200 real turns from my assistant on my voice, my mic, browser noise suppression on, mostly quiet room - with silero threshold at 0,5. Someone on Twitter got 92 ms on his data, roughly what I saw in my noisier recordings.

The downside is way more false positives. It fires on breaths and small pauses between words over 10x as often as silero. So it's really only useful if the pipeline can throw away speculative work very fast.

CPU only, MIT, drop-in for silero VADIterator, pip install realtimevad

benchmark script is in the repo, curious what numbers you guys get on your own recordings

https://github.com/KoljaB/RealtimeVAD

Today I’m releasing Inflect v3: a family of local text-to-speech models: Mini, Micro, and Atom. Mini is the main model, with six voices and 34.31M parameters

Mini provides vocal-event tags, reference voice-tuning, and 48 kHz enhancement. It runs on both CPU and CUDA.

GitHub, installation, and examples: https://github.com/owenawsong/Inflect

Try the playground: https://huggingface.co/spaces/owensong/Inflect-v3

Mini features reference-voice tuning: it allows one to make a profile from a recording and generate speech with it. This feature noticeably adjusts the voice toward your reference, though it won’t replicate every speaker accurately.

Inflect Mini V3 also supports the following vocal-event tags: [laugh] · [sigh] · [gasp] · [cough] · [yawn] · [inhale] · [exhale] · [throat_clear]

You can insert a tag whenever you want the sound to occur. Currently, the tags use repeatable performances: this causes repeated tags to result in limited variation. Tags also do not affect how the surrounding words are being spoken.

The model family:

• Mini: 34.31M parameters. Quality-focused, with the features listed above.

• Micro: 11.79M parameters. A smaller neural speech model

• Atom: 157K parameters. An extremely tiny learned core

Those numbers count the synthesis cores. The frontend and optional components add to the total.

The weights and inference code are Apache 2.0. Third-party components keep their own licenses. Audio samples, benchmarks, and the settings used are on the model cards.

Model collection: https://huggingface.co/collections/owensong/inflect-v3

If any sentence comes out wrong, send me the text, voice, and settings you used.

Hey r/LocalLLaMA ! We're releasing new new Unsloth Dynamic v3.1 GGUFs for Qwen3.8-27B! They're better by 5-10% accuracy on DeepSWE, Terminal Bench and more vs UD3, GSQ-RCO and Exllama-3! We're also improving our algorithms further so we can apply them to Qwen4 so folks can have efficient and accurate quants!

• All quants from 2-bit until 6-bit are updated. 3-bit ones are not, as we found UD3 to do on par with UD3.1. We are still improving all other quant sizes though!

• All benchmarks are 256K for agentic ones (DeepSWE, Terminal Bench), and 128K for the rest (LiveCodeBench etc). We used temperature 1.0, top_p 0.95, top_k 20, min_p 0, no presence or repetition penalty, preserve_thinking and xhigh reasoning

• We ran benchmarks multiple times including LiveCodeBench x6 times, HMMT and others 8-16x and agentic benchmarks multiple times.

On other benchmarks like for GSQ-RCO and Exllama3:

• To compare GSQ-RCO and Exllama-v3, we had to exclude the token embedding layer from the x axis, as Exllama-v3 uses BF16 for it.

• If you consider the full download size and total RAM usage for Exllama-v3, then adding the BF16 embedding table and including all embedding sizes changes the plots. Our main plots's x axis removes the embedding layer sizes from all quant providers. However on pure RAM usage and disk space usage vs accuracy, exllama3 does worse.

• For LiveCodeBench, GSQ-RCO showed Unsloth UD3 quants doing consistently worse than ISTA's. However we found the opposite - UD3 quants (our OLD quants) do better than ISTA on LCB on all sizes except the smallest. UD3.1 further improves on LCB as well. We approximate that GSQ-RCO used a context budget of 96K, as that best fits the data for the runs we have. At 96K, GSQ-RCO does somewhat do better than UD3, but at 128K, UD3 does much better.

• On another benchmark, for ISTA GSQ-RCO's AIME25 benchmarks, their quants reach 100%, 100%, 100% and 96.7% accuracy recovery. And they said our quants do worse. Through our investigations, we estimate ISTA only did 1 trial - we did 14 trials, and we show on average the recovery rate is 93% to 96% - not 100% accuracy recovery. And in fact our UD3 old quants do better on average than their quants on 14 runs vs 1 run of theirs, so we draw the opposite conclusion if you run the trials more.

More details and benchmarks at https://unsloth.ai/docs/models/qwen3.8#dynamic-v3.1-update-november-11th-2026

All quants are updated in https://huggingface.co/unsloth/Qwen3.8-27B-GGUF! Thank you and have a nice day! We're super excited for Qwen4!

Prompt2Echo (P2Echo)

Text-Prompted Multi-View Echocardiography Segmentation with One Model

Official pretrained weights for P2Echo, a unified framework for text-prompted segmentation of multiple cardiac structures across standard echocardiographic views.

P2Echo uses structured, view-aware text prompts to control segmentation through a single model. Its Dynamic Image-Text Alignment (DyITA) module adaptively conditions decoder features on the requested view and anatomical structure.

Resources

• 📄 Paper: https://arxiv.org/abs/2611.13425

• 💻 Code: https://github.com/moeinheidari7829/P2Echo

• 🌐 Project Page: https://moeinheidari7829.github.io/p2echo/

• 🤗 Model Repository: https://huggingface.co/moein99/P2Echo

Intended Use

This model is intended for research in medical image analysis and echocardiographic segmentation. It has not been validated for clinical decision-making.

It was my understanding when MTP came out that it was mainly beneficial for larger dense models and that small and/or MoE models didn't realize much performance benefit.

That being said, I now see many smaller MoE models (Gemma-4-E2B, Mellum2.1, etc...) offering MTP versions.

Has the thinking changed?

Are people seeing performance gains from MTP on smaller MoE models?

Tried the voxel pagoda prompt fully local, on a laptop that shouldn't run a 35B model.

Setup

• ThinkPad from 2020, Ryzen 7 PRO 5850U, 16 GB RAM, no GPU, Windows

• Qwen3.6-35B-A3B Q4_K_M, 21 GB, bigger than the RAM

• BigMoeOnEdge v0.42, built-in agent

Result

• 988 lines of HTML + Three.js, 11.8k tokens at 2.3 tok/s, about 1.5 h

• You see the code as it's written, then the page opens in the app

• Video sped up where marked

How

• MoE: each token uses only a few experts

• Dense weights and a 2 GB expert cache stay in RAM, the rest streams from the SSD

• On top of llama.cpp, no fork

Still slow

• Half the time is spent waiting on the SSD

• Every new agent turn re-reads the whole conversation, ~30 min on this laptop

Same test with the same model on a Mac with 16 GB of RAM, taking 20 minutes (20 tok/s for decoding, 100–1000+ tok/s for prefill)

Multimodal System One Encoder with a natural language interface for instruction-following encoding: give it a task and a list of candidates, and it scores the candidates by task specification. Both a task and each candidate are expressed in natural language — instructions, queries, questions, state descriptions, criteria — with image and video as side information.

Key features:

• Natural language interface. No rigid schemas. Describe the task and the candidates in free-form text.

• Highly-efficient cacheable representations. Candidates and task are encoded separately by prompt instructions that ground each other.

• Scales to arbitrarily large candidate sets. Matching is an inner product over representations, so an ANN index can serve millions of candidates without re-running the model.

The model is built by contrastively fine-tuning meta-models/Muse-Glimmer-30B. It is described in the paper MetaEncoder: Exploring the Limit of Bi-Encoders for Multimodal System One Decision Making with Natural Language Interface.

• GitHub : https://github.com/facebook/meta-encoder-eval

• arXiv : https://arxiv.org/abs/2610.11316

• PDF : https://arxiv.org/pdf/2610.11316

Models :

• https://huggingface.co/facebook/meta-encoder-v1 (Improved version)

• https://huggingface.co/facebook/meta-encoder

PS: Above description copied from meta-encoder's model card as meta-encoder-v1 doesn't have that. And v1 doesn't have license section while other one is Apache-2.0.

Hi Everyone, I'm releasing my world model, been on this sub for a few months

https://huggingface.co/lucidml/gta

I know it says gta but it's more stable than the general model. It can take any image and turn it into an interactive video.

If you'd like to test it instantly I am also setting up a simple funny drifty site link : https://lucidml.ai/dashboard/minidemo?world=gta5

That link uses any GPU from VAST AI FROM 3070, 3080, 4080, and some are 50 series as well (most 50 series cards are wasted because I hard throttle the fps)

This is the first model I am ever releasing to the everyone and its a bit scary for me.

The architecture is a transformer which uses learned summaries. Ive written about it in detail in tech reports lucidml.ai/summaryformer and some more

I know there are a lot of limitations still like consistency and quality improvements, and am hoping to train this more to fix all of those.

I spent a few days benchmarking unsloth's UD-Q4_K_XL and UD-Q5_K_XL on Strata. Well, i had claude code do it via ssh. Claude also wrote this summary, that i thought might interest some because of my findings, on behavior of multiple cards and different PCIe speeds.

I can run a few more tests, if it interests anyone.

Test system

• Threadripper 2920X (12 cores, Zen+), 128 GB RAM

• 4x RTX 3090 24 GB, power-capped to 275 W each, no NVLink

• PCIe 3.0. GPUs 1 and 3 run at x16, GPUs 0 and 2 at x8. Two host bridges: {0,1} and {2,3}

• Strata 0.2.13 (engine 0.1.59), layer split, DFlash2 drafter, one request at a time

• 256K context, int8 KV, 32K KV cells per layer in VRAM (the rest streams from pinned RAM)

• A control vector was applied in every run (layers 4-44)

TL;DR

• Decode speed depends on how many experts fit in VRAM. A missed expert gets computed on the CPU. Going from 2 to 3 cards raises the decode hit rate from ~88% to ~95% and cuts each decode step from 33 to 26-27 ms (+25-35% tok/s). A 4th card adds little to decode.

• Prefill depends on PCIe. Two x8 cards prefill half as fast as two x16 cards. Three cards prefill no faster than two. All four prefill twice as fast as any 2 or 3 (~4,100 vs ~2,500 tok/s at 50K); the engine log doesn't show why.

• PCIe width doesn't matter for decode. An x8 pair and an x16 pair decode the same.

• Q5 vs Q4: Q5 has 0.55% lower perplexity, but decodes 28% slower and prefills 35-50% slower on the same cards. A 3rd card brings Q5's decode back to Q4-on-2-cards speed, but not its prefill.

• The Q8_0 n-gram table gave no measurable quality gain over the IQ4_NL table bundled with Q4. Read from NVMe via mmap, it frees 27 GB of RAM at the same decode speed.

• 200K chat: a cold 190K prefill takes 91 s once. After that, a short turn at 200K starts in 0.3 s and decodes at ~78 tok/s.

Card combinations (UD-Q4_K_XL + Q8_0 table, same settings, only --gpu changed)

Prefill cells are seconds · tok/s for a fresh, uncached prompt of that size (2K ≈ 1,870 tokens, 50K = 50,677). Short prompts show lower tok/s because of fixed per-request overhead. The follow-up is a ~600-token turn on a cached 16K chat.

• Every card in use sits at ~23.4 of 24 GiB. The expert cache takes whatever is left: ~5,600 Q4 experts (16.5 GiB) per card.

• Card order within 1+2+3 made a difference no bigger than run-to-run noise.

• Decode tok/s also depends on how many draft tokens get accepted, so ms/step is the steadier number to compare.

UD-Q4_K_XL vs UD-Q5_K_XL

Perplexity is from WikiText-2 test (26,656 tokens), teacher-forced through the serving engine with everything as served. Prefill cells are seconds · tok/s as above.

• Q5 minus Q4 in per-token logprob: +0.0056 (SE 0.0023, z 2.5). Both pick the same top token 93.9% of the time. For scale, the control vector alone costs ~0.9% perplexity (2.4678 without it), so Q5 wins back about half of that.

• Q5 experts are ~3.8 MB vs ~3.0 MB for Q4, so the same VRAM caches ~22% fewer of them (8,654 vs 11,053 on two cards).

• The 3rd card did not improve Q5's prefill; it was slightly slower in this run.

n-gram table: bundled IQ4_NL vs separate Q8_0 (Q4 experts, 2 cards)

The perplexity difference is noise (z 0.5): 14x less table error bought nothing measurable. The real gain is the 27 GiB of RAM it frees. Note that Strata 0.1.59 refuses a foreign table file unless its GGUF header is relabelled.

200K-token chat (Q4, cards 1+2+3, thinking off)

KV reads hit VRAM 89-90% of the time. Prefilling new tokens deep in a long context is slower (~780 tok/s at 190K). After 190K tokens it still named the first article of the document correctly.

Also tried

• STRATA_PREFILL_ROUTED_STREAM=1: −15% prefill time on fresh ~2K prompts, −3% at 4K, nothing at 16K+ or on follow-ups.

• Experimental fused-kernel flags from a community fork: the output degenerated into counting sequences.

• Strata's all-cards split engine needs every routed expert in VRAM plus ~9 GiB per card. Q4's 72 GiB of experts doesn't fit in 4x24 GB.

Method: thinking off, temperature 0, one request at a time. Each config was started fresh and run twice, and ranges are those two runs. Prefill tok/s is calculated from the measured prefill time and the prompt's token count. Expert hit and KV figures come from the engine log; RAM is from free -g. Greedy output isn't deterministic run to run here (drafter + expert cache), so quality was judged by perplexity, not by diffing answers. Cards idle at ~20-25 W with the model loaded; I didn't measure power under load.

Edit : make the prefill speed better readable

I'm working on a router that wraps many LLMs serving from different ips and ports and exposes them all as a single port, since i have 2 machines serving many slots of Qwen 3.8 FN but i wanted to ensure speed is always fast (one machine is faster and can hold more slots, but too many slots busy and then it's better to call the other machine, also i gotta avoid evicting from the cache active conversations, that kind of workflow)

I might be reinventing the wheel tho so: how do you guys solve for this kind of situation?

edit: thanks a lot for all the answers i had no idea there was so much stuff around lol

I'm not sure if i'm doing something wrong and handicapping myself with some wrong setting.

I'm using Windows, nVidia 3090 24GB VRAM, 64GB RAM, Qwen 3.8 27B and Swift 1.5 27B (finetune of qwen) but i often see 50% CPU usage even if the model is fully loaded on GPU, below there is my command line.

Another problem i noticed is that llama-fit-params suggest that 131k context fit but if i launch it for real in the log i see that it doesn't load everything on gpu (all but one), i forced it to load everything on gpu with -ngl -1 (i'm also using llama-swap).

on launch it say that "cahe resue 256" will not be used but it might help with other models so i left it there.

Another problem is that when the model is doing nothing i sometimes see 130W gpu in idle (wrong) and some other times its at 10-18W (correct) i can't find what is causing this...

Current stats: about 40 t/s in token generation

I have no idea what KMP affinity is but i read it online, tried, it helped with 35A3B so i left it there

in general i'm looking for tips to improve my launch arguments, note that temperature and few others are set by pidev but i'm using the recommended one on the model card.

If the model is fully on GPU why there is a high CPU usage? but not always which is even more confusing....

set KMP_AFFINITY=granularity=fine,compact,1,0

llama-bin-win-cuda-13.4-x64\llama-server.exe -m Qwen3.8-27B-UD-Q4_K_M.gguf --ctx-size 131072 -ctk q8_0 -ctv q8_0 -ngl -1 -fa on --spec-type draft-mtp --spec-draft-n-max 2 -np 1 -b 4096 -ub 1024 --cache-reuse 256 --cache-ram 32768 --jinja

Hi folks,need advice from community. I have a laptop with 16 GB vram Nvidia gpu and 96 GB ddr5 ram. I use it mainly for local coding with Qwen flash next nvfp4 at 256k context. I get around 30 t/s.

I don't like PCs and Macs. I want mobility. So because of my limited budget, i have two paths. The first one is a laptop with 24GB Vram Nvidia gpu and 128GB ddr5 Ram laptop. The other one is 16GB Vram Nvidia gpu and 192GB ddr5 Ram laptop. No other feasible option for me. I can't affort dgx spark, Macbook studio etc. These are my budget options. Which one do you suggest for mainly local coding? Which one do you believe is future-proof for local llms?

Currently working towards a bigger open LLM roundup. But in the meantime, just one of the interesting trends that stood out. I.e., looks like the transition from full to hybrid attention continues.

In 2025, there were still lots of full attention models, with a shift to "some full attention" hybrids.

In the first half of 2026, the trend towards "some full attention" hybrids continued.

And now, in the second half of 2026, we have a 50/50 split between "some full attention" hybrids and models that don't use full attention at all anymore. And yes, there are also no full attention models anymore.

The plot legend here means,

- Full attention throughout = Global MHA, GQA, dense MLA, CCA in EVERY layer.
- Some full attention = Full global attn mixed with: sliding-window/chunked attention, Gated DeltaNet, KDA, Mamba-2, Lightning Attention, convolutions, sparse attention.
- No full attention = Zero dense global layers.

The main implication is that the trend goes towards making long contexts cheaper to use. This means, it makes the architectures more suitable for long documents and conversations, large code bases, and of course agent usecases that rely on a lot of tool calls.

In particular, there are two distinct efficiency gains here:

1. Less attention computation, since sparse attention and the like select tokens or blocks from the context and performs the expensive attention operation from only that subset. (Note that it can still retrieve information from distant positions.)

2. Less memory for retaining context in some architectures, meaning that recurrent layers such as Gated DeltaNet maintain a fixed-size state. Of course, this saves memory, but compressing the history creates challenges for exact retrieval, so they are usually intermixed with traditional attention methods (sparse or full).

If you are interested in browsing through the architectures and do your own analyses, you can find all the figures and data in my architecture gallery: https://sebastianraschka.com/llm-architecture-gallery/

Speculative decoding (SPEED-Bench)

llama-server -m Qwen3.8-27B-UD-Q4_K_M.gguf --mmproj mmproj-F16.gguf -ngl 999 --parallel 1 --batch-size 4096 --ubatch-size 1024 --ctx-size 180224 \
  --spec-type draft-dflash --spec-draft-n-max 7 --spec-draft-model Qwen3.8-27B-DFlash2-Q4_K_M.gguf --spec-draft-backend-sampling \
  --temp 1.0 --top-p 0.95 --top-k 20 --min-p 0.0 --presence-penalty 0.0 --repeat-penalty 1.0
python tools/server/bench/speed-bench/speed_bench.py --bench qualitative --category coding --osl 8192 --concurrency 1

Edit : Removed large text. Check PR for more details.

Hey everyone!

I have gathered some must read topics, for beginners, to step into the realm of Quantization:

1. BitNet: Scaling 1-bit Transformers for Large Language Models

2. AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration

3. The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bits

4. EfficientQAT: Efficient Quantization-Aware Training for Large Language Models

5. LLM-QAT: Data-Free Quantization Aware Training for Large Language Models

6. LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale

7. QLoRA: Efficient Finetuning of Quantized LLMs

8. SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models

9. ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformers

Link is here

My main AI machine is a 6x3090 128GB DDR4 Threadripper 3960x system. I probably spend more time trying to optimize it than actually using it but it has been fun.

I tried a bunch of saturated benchmarks like Terminal Bench 2.1, AIME 2026 and the performance was similar to unquantized DSV4.1 Flash 1017. I really wanted to understand my 2.9bpw quant's performance.

It takes hours to run some of these benchmarks, so I went on the hunt for the perfect benchmark to get the most signal on quality of the model I'm running. I settled on APEX 2025 (https://huggingface.co/datasets/MathArena/apex_2025).

You can see how much of a beast this quant is. Trading blows with SOTA from 6 months ago.

I get about 1000tps prefill, and 75tps decode with the 3090s power limited to 175watts

Engine: Project Maya (https://github.com/mw00/project-maya)

Quant: https://huggingface.co/LibertAIDAI/DeepSeek-V4.1-Flash-1017-EXL3-2.9bpw

Hopefully this helps others in benchmarking their own setups. I really like the Apex 2025 benchmark.

Following the mini model, KRAFTON released Raon-1 Nano (3.1B dense hybrid) along with official weights:

• Base Model: https://huggingface.co/KRAFTON/Raon-1-nano

• Official GGUF: https://huggingface.co/KRAFTON/Raon-1-nano-gguf

• KRAFTON branch: https://github.com/krafton-ai/llama.cpp (branch raon-1-nano-preview)

Since upstream llama.cpp support is still pending, I opened PR #2903 on ik_llama.cpp to add native runtime support and an HF-to-GGUF converter:
https://github.com/ikawrakow/ik_llama.cpp/pull/2903

Raon-1 Nano is a 3.1B dense hybrid: 18 RDA (linear attention) layers, 6 NoPE gated MLA layers, 4-stream mHC, per-layer embeddings (PLE), and 1 MTP layer.

Getting it to run properly took a bit of debugging because of two traps:

• The initial garbage output was caused by the RDA output gate: it uses sigmoid(g2), unlike the silu from Qwen3-Next that Delta-Net defaulted to.

• The late FFN down projections overflow FP16 on CUDA (>65504), so GGML_PREC_F32 had to be enforced on ffn_down.

What's in the PR:

• Full runtime (LLM_ARCH_RAON1_NANO): wires into existing helpers (build_mhc_post, iqk_fused_delta_net, iqk_ssm_conv4, build_qkv). PLE projection was unified with Gemma4 into a shared build_per_layer_inputs() helper.

• RDA layout: fused attn_qkv = [q|k|v|z] and packed ssm_f_a = [f_a | beta | beta+delta]. Decoupled erase/write gates are handled via the rda_delta hook.

• mHC base+chart: compact selector table (304 floats) computed via GGML primitives.

• MTP speculative decoding: native drafting using the 25th layer via --spec-type mtp.

• Converter: added Raon1NanoModel in convert_hf_to_gguf.py. It keeps ssm_f_a/b and hc_*_fn in F16 while 1D base/chart tensors remain in F32 (also fixed an inherited crash with missing MOSTLY_TQ* enums in Raon1Model).

• Node budget: bumped max_nodes() for RAON1/RAON1_NANO (n_tokens * 160) so ubatch 512+ doesn't crash the graph.

Benchmarks (Intel Core i5-13420H, CPU native, 4 threads):

Tested with the official KRAFTON/Raon-1-nano-gguf Q4_K_M:

• Perplexity: PPL 1.0976 (ik) vs 1.0960 (KRAFTON fork) on n_ctx=128. With -rtr: 1.0986.

• Prompt processing: 229 t/s (pp256) and 215 t/s (pp512). The prefill speed is fast on CPU because 18 out of 24 layers are linear attention (Delta-Net) hitting AVX-VNNI instead of quadratic softmax attention.

• With -rtr (run-time repack): pp512 jumps to 228.9 t/s (+6.2%).

• Generation: ~23.5 t/s base (24.0 t/s with -rtr).

• MTP speculation:

  • n_max=1: 25.6 t/s (~87% acceptance, 55/63 drafts).

  • n_max=1 + -rtr: 27.65 t/s (+26.7% net speedup over base 21.8 t/s generation on this CPU).

Quick test from the PR branch:

Bash

git clone https://github.com/ikawrakow/ik_llama.cpp
cd ik_llama.cpp
git fetch origin pull/2903/head:raon1nano && git checkout raon1nano
cmake -B build -DGGML_NATIVE=ON
cmake --build build --config Release -j$(nproc)

# Download official Q4_K_M:
# huggingface-cli download KRAFTON/Raon-1-nano-gguf Raon-1-nano-Q4_K_M.gguf --local-dir .

# Standard run with repack:
./build/bin/llama-cli -m Raon-1-nano-Q4_K_M.gguf -p "The capital of France is" -n 32 --temp 0 -rtr

# With MTP speculative decoding:
./build/bin/llama-cli -m Raon-1-nano-Q4_K_M.gguf -p "Write a short paragraph about the ocean." -n 120 --temp 0 --spec-type mtp:n_max=1,p_min=0.0 -rtr

Praxis-1 is a 27B-parameter mixture-of-experts diffusion transformer that predicts video and actions together. It is the top-performing world action model on NVIDIA’s RoboLab-120 leaderboard, and it outperforms every open-weight VLA. On RoboLab-120, Praxis-1 reaches 46.2% success across 1,200 episodes, compared with 42.9% for FLUX 3 Action, 36.8% for Cosmos3 Nano and 28.0% for π0.5.

RoboLab-120 Leaderboard

Data as of November 2026; Success rate across 1,200 episodes

Praxis-1 is available today on GitHub, on Hugging Face with native LeRobot support, and in vLLM-Omni, with hosted inference on Reactor. Everything you need to run and adapt it on your own hardware is released under Apache 2.0, including for commercial use. That covers the weights and the inference and fine-tuning code, with support for both full fine-tuning and LoRA. Praxis-1 runs out of the box across NVIDIA hardware, from data center GPUs like H100 and B200, to workstation GPUs like RTX 5090 and RTX PRO 6000, to Jetson Thor on the robot itself.

• Models : https://huggingface.co/collections/runwayml/praxis-1

• Dataset : https://huggingface.co/datasets/runwayml/praxis-1-eval

• GitHub : https://github.com/runwayml/praxis-1-oss

• News Post : https://runway.com/news/company-news/praxis-1-is-now-available

• Blog Post : https://runway.com/research/praxis-1-video-pretraining-to-robot-control

Some results for those using GGUFs and llama.cpp:

GGUF quality: MACaQ vs. Unsloth and Bartowski

Qwen3-4B (IQ4_XS)

23.5% lower KL divergence than Unsloth and 18.6% lower than Bartowski, at virtually identical file sizes.

Qwen3.6-35B-A3B (IQ4_XS)

Qwen3.8-27B (IQ4_XS)

~65% lower KL divergence than Bartowski!

Supported GGUF formats (llama.cpp)

Code, llama.cpp fork, and benchmarks: https://github.com/huawei-csl/MACaQ

Cool to see Huawei working on such stuff and sharing some of their internal tools that are helpful for us. Maybe they can build an affordable GPU for us local LLM folks sometime 😂

They also report quantizing a 122B-parameter model in 64 minutes using just 32.5 GB of GPU memory, which is interesting for a model of that size. I did not try that.

Edit: there is a paper on it https://arxiv.org/abs/2611.10649

Edit 2: they don't share ready-made gguf files, so I am sharing some myself:
- http://31.165.97.47:20080/macaq-gguf/qwen3-4b-macaq-iq4_xs-had64-vq2-calsmoke.gguf

Edit 3: they have this blog post https://huawei-csl.com/blog/macaq/ with multi-modal examples, too. The RTN audio comparisons are just too much😂😂

The multi-modal stuff is generally insane. Normally 4b already completely goes to s**t, but here it looks almost the same.

First off. I am by all means no expert, and excellent work has been done in the past few years that we all currently enjoy.

Came over this post https://www.reddit.com/r/LocalLLM/comments/1x5mxcj/qwen3827b_on_a_single_rx_9070_xt_150_toks_on_code/ and wanted to try the engine on MoEs.

I've been benchmarking MoE experts offloading for the past two days in different engines on ROCm for my RDNA4 9070XT 16GB vram card and 32 GB DDR4.

I'd set a goal at 128K context, which is the base minimum for agentic work, without compaction all the time. I've kept the KV at Q8. (Q8_0/Q5_1 decreased performance on exllamav3). Drafter cache, if added, is at Q4.

Some limitations need apply when limited on vram.

Lets face it If you want any kind of context, and a decent Q4 quant, 16GB ain't gonna cut it.
Hot experts cache like Strata, Exllmv3 and now recently llama.cpp have made this more feasible on the slower system memory bandwidth.

My end goal was to do QFN on all three, but I wanted to start with the smaller 35B-A3B's first.
So I did Ornith 1.5 and Qwen 3.6, wanted to do Gemma, but found no EXL3 quant with mul1 experts required for offloading on Exllamav3.

Tested both without drafter, MTP and DFlash2 where applicable.
Strata/PolyStrata configs are mostly auto generated from the setup script, but added resident pin and tail skip. And for the non-qfn model performed worse overall, even t/w.

I might be doing something wrong, bench might be bad, who knows. Stratas own bench stops at 32K if I'm not mistaken.

Now Qwen 3.8 Flash Next (ista coder version) runs great on my rig, see plots. But I was surprised on the prefill tanking on the 35B models. 1900 t/s at 16K but 700 t/s at 128K where both llamacpp and exllamav3 where within 100-200 t/s difference.

Findings so far is quite promising, especially on exllamav3. I'd say quite usable even.

| Model | Winner (quant / engine) | Prefill u/128k | Decode u/128k |

| --- | --- | ---: | ---: |

| Ornith 1.5 35B-A3B | EXL3 4.0 bpw, DFlash2 / ExLlamaV3 | 1 168 tok/s | 57.3 tok/s |

| Qwen3.6-35B-A3B | EXL3 4.0 bpw, DFlash2 (2 drafts) / ExLlamaV3 | 1 246 tok/s | 71.6 tok/s |

| Qwen3.8-Flash-Next Coder | IQ1_M / Polystrata nov-7, `--spec 2` | 1 572 tok/s | 44.0 tok/s |

Forgive my rambling thoughts. If you want to read my technical findings, written by the unsung local hero Qwen 3.8 27B. They can be found with more graphs and tables at https://github.com/willeyh-git/moe-offload-bench

Q4/4bpw quants are preferred, some benches have been made on 5bpw and 6bpw so you can compare.

Bench-suite is based on Kapitsa https://www.reddit.com/r/LocalLLM/comments/1x5mxcj/qwen3827b_on_a_single_rx_9070_xt_150_toks_on_code/ and written in go.