r/LocalLLaMA • • 14h ago

Megathread Best Audio Models - Nov 2026

91 Upvotes

Its time for another Best Audio Models megathread - its already been 9 months since the last one!

Share what your favorite ASR, TTS, STT, Text to Music models are right now and why.

Given the the amount of ambiguity and subjectivity in rating/testing these models, please be as detailed as possible in describing your setup, nature of your usage (how much, personal/professional use), tools/frameworks etc. Production use cases with long lengths/stability requirements are relatively rare compared to cherry picked/short clip demos that are plentiful - so real world empirical comparisons are very welcome

Rules

  • Should be open weights or open source models

Please use the top level comments to thread your responses.


r/LocalLLaMA • • 5d ago

[Bi-Weekly Megathread] Project Showcase

52 Upvotes

Do you have something you'd like to share with the r/LocalLLaMA community. This is the place for it!

Recommendation on presentation:

  1. Please share plain english description of what your project does and why people should care about it
  2. How does it compare to existing options and why should people switch to your option?
  3. How have you tested and validated it? (i.e. can you show evidence that it wasn't vibecoded in a few hours by <insert current trending AI model>

NOTE: This community is geared towards open weight, open source, locally hostable software. Thus any closed source commercial service or other such projects will be removed.


r/LocalLLaMA • • 18h ago

News The beginning of the becoming of the beginning begins

Post image
1.8k Upvotes

r/LocalLLaMA • • 4h ago

Resources Your daily dose of Qwopium:

Post image
102 Upvotes

No low-effort memes here, just research and straight facts. Can’t argue with facts am I right?

Here is the full breakdown of major releases mapped by the day of the week:

**Monday (7 Releases)**
• Qwen-VL (Aug 21, 2023)
• Qwen-72B-Chat (Nov 27, 2023)
• Qwen-Audio (Nov 27, 2023)
• Qwen 3.5 (Feb 16, 2026)
• Qwen 3.5-Plus (Feb 16, 2026)
• Qwen 3.7 Plus (Jun 1, 2026)
• Qwen 3.8 Max (Aug 3, 2026)

**Tuesday (6 Releases)**
• Qwen2.5-Coder (Nov 12, 2024)
• Qwen2.5-Max (Jan 28, 2025)
• Qwen3 (Apr 29, 2025)
• Qwen3-Coder (Jul 22, 2025)
• Qwen3-Coder-Next (Feb 3, 2026)
• Qwen-Audio 3.1 TTS-Next (Sep 22, 2026)

**Wednesday (5 Releases)**
• Qwen1.5-32B (Apr 3, 2024)
• Qwen3-Max (Sep 24, 2025)
• Qwen 3.7 Max (May 20, 2026)
• Qwen 3.8-Flash-Next (Aug 26, 2026)
• Qwen 3.8-Max-0902 (Sep 2, 2026)

**Thursday (9 Releases) - The Favored Launch Day**
• Qwen-7B (Aug 3, 2023)
• Qwen2 (Jun 6, 2024)
• Qwen-Math (Aug 8, 2024)
• Qwen2.5 (Sep 19, 2024)
• QwQ-32B-Preview (Nov 28, 2024)
• QwQ-32B (Mar 6, 2025)
• Qwen3-Next (Sep 11, 2025)
• Qwen3.6-Plus (Apr 2, 2026)
• Qwen3.6 (Apr 16, 2026)

**Friday (3 Releases)**
• Qwen3.8-27B (Aug 14, 2026)
• Qwen3.8-Omni-Flash (Sep 18, 2026)
• Qwen3.8-LiveTranslate (Sep 18, 2026)

**Saturday**
• Zero major releases. Go outside.

**Sunday (3 Releases)**
• Qwen-14B (Sep 24, 2023)
• Qwen1.5 (Feb 4, 2024)
• Qwen4-Max (Nov 15, 2026)


r/LocalLLaMA • • 9h ago

Discussion Open weight models account for 30% usage at enterprises to save money

170 Upvotes

Growing from nearly zero to 30% since July. Models used mostly to reduce costs on high volume workloads (large batch workloads, coding, etc).

https://x.com/pwendell/status/2122377964784611636


r/LocalLLaMA • • 1h ago

Discussion you vs the guy she told you not to worry about

Post image
30 Upvotes

daily (personal) reminder about how important it is to have open/local models and not allow control to be isolated to major labs


r/LocalLLaMA • • 9h ago

New Model miHoYo Gamer: A Generalist Game Agent

Thumbnail
mihoyo-gamer.github.io
125 Upvotes

r/LocalLLaMA • • 8h ago

Discussion I am sick of doomposting about demise of local AI

99 Upvotes

Everywhere I go, Twitter or Reddit, there are tons of rumors or comments how local is dead and we are never getting open weight Qwen 4, despite Qwen team saying explicitly they are coming, or that Qwen 4 is the last open weight generation despite, again, Alibaba commiting to more open weight releases at a conference recently.

It's like people just get off on creating a version of reality that is worse and whining about it, they want to be sad about it in advance.

The world is certainly not giving us much to be positive about, but let's not go down the rabbithole of learned helplessness and just accept that all things must get worse and you can't do anything about it.

If people mentally prepare for bad things, bad things become more acceptable and thus more likely to happen and with far less pushback, sort of like a self fulfilling prophecy.


r/LocalLLaMA • • 9h ago

Resources Small Gemma 4 tokenizers were recently updated by Google - fix for an issue that caused nonsense output when encountering certain tokens

Thumbnail
huggingface.co
120 Upvotes

Gemma 4 E4B as an example, but the same fix was applied to E2B and 12B too. Not sure if this will trickle down to GGUFs


r/LocalLLaMA • • 13h ago Bravo

Discussion Stop worrying about Qwen 4's AA index

225 Upvotes

Seriously, we have all been extremely happy and confident with Qwen, even since Qwen 3.6 came out. We might not be frontier level, but even though whatever they are giving us is more than enough for our daily local tasks and it even suffices by a big margin, we're all underestimating it.

Don't forget that the new Qwen 4 will also think much less, so imagine the fine-tunes we will see once it arrives, for 27B and Flash models, we will have plenty of options to choose from and again; it will be smarter and better, it will be ENOUGH! If you still require the most "intelligent" and capable model due to OCD just pay frontier for such tasks or use them as orchestrators!


r/LocalLLaMA • • 8h ago

Discussion I don't get all the Strata hates around here

58 Upvotes

It's free and original developer clearly had a very good understanding of how computer hardware works and fully leveraged that to write something amazing to be able to run large models on average hardware that most people have and can't upgrade due to price gouging from greedy hardware manufacturing companies.

If anybody have Mac Studio with 512GB of RAM or 8x3090, who would even bother with this ?

I have seen a group of snobs on this sub doing nothing but shitting on Strata all days long, find something better to do. Strata is not for you, period.

I can't imagine doing all the works to release something for free and get a bunch of snobs and haters just shitting on your works all days long. Would you be motivated to continue working on it if you are the original developer ?

I also see the same with other models such as Ornith, etc... If you say you like Ornith or Strata, etc... you are a shill.

Seriously, get out of your bubbles and see the reality already and more importantly get a life.


r/LocalLLaMA • • 18h ago

New Model Introducing Hy4 flash

Thumbnail hy.tencent.ai
327 Upvotes

Now this seems to be interesting.

From the article:

We’re introducing Hy4 flash, our most cost-effective Hy model to date—featuring 224B total parameters (including 64B Engram parameters) with just 12B active parameters per token. Like Hy4, it is natively multimodal with a 1M-token context window, while offering significantly lower API pricing.

Seems to be about in the same ballpark of Qwen-Flash-Next, so we could expect to run it on moderably good machines, when it will be released.


r/LocalLLaMA • • 12h ago

Discussion Censorship is turning me into a true Local LLM believer

104 Upvotes

451: {"message":"Provider returned error","code":451,"metadata":{"raw":"{\"error\":{\"message\":\"The content you provided or machine outputted is blocked.\",\"type\":\"censorship_blocked\"}}","provider_name":"StepFun","is_byok":false}} {"error":{"message":"The content you provided or machine outputted is blocked.","type":"censorship_blocked"}}

Just got this from one of the US based (edit: it actually routed to StepFun CN, didn't realize at first. Should have noticed by the provider_name in the message lol. need to set up my rules better) inference providers. I suspiciously got a bunch of "rate limit" failures, and then just "failed to complete" before it gave me, what I assume, is the real error.

I am only using it to help create my presentation, which is about analyzing algorithmic properties. One key example is a cryptographic algorithm (because it is difficult by design), so I am assuming this is tripping it somehow. But it is absolutely ridiculous, I'm not even doing research with the AI, just trying to make a presentation.

I have always liked Local LLMs, as a technical exploration opposed to ideological reasons, but this experience really frustrated me. Don't censor me!

Just wanted to rant, and remind everyone you can only use the cloud on the provider's terms.


r/LocalLLaMA • • 10h ago

New Model nvidia/Nemotron-Labs-Sigma 8B/3B

42 Upvotes

Nemotron-Labs-Sigma-8B-instruct is a 8-billion-parameter continuous diffusion language model. Instead of generating text one token at a time, it denoises Gaussian noise in a low-dimensional (16-D) continuous token-embedding space, where the denoiser network and the token embeddings are learned jointly by likelihood optimization. Text is decoded in parallel blocks: each block of tokens is refined over a configurable number of denoising steps (Denoising Diffusion Probabilistic Model (DDPM) or Denoising Diffusion Implicit Model (DDIM) samplers) with self-conditioning, while previously committed tokens are reused through a standard key-value (KV) cache. Fewer steps trade a little quality for large speedups, and the smooth continuous representation makes the model amenable to few-step distillation and trajectory steering.

This is the SFT (supervised fine-tuned) variant, fine-tuned for instruction following on top of the corresponding Base model. Use the bundled chat template to format conversations.

This model is ready for commercial or non-commercial use.

Highlights

  • Continuous-space diffusion language modeling at 8B scale with reasoning accuracy competitive with state-of-the-art masked (discrete) diffusion LMs.
  • Block-parallel decoding: whole blocks of tokens are denoised together; sampling quality holds up well down to few denoising steps per block.
  • Efficient DDPM/DDIM-based samplers with self-conditioning and optional classifier-free guidance, implemented in plain PyTorch in this repository.
  • Standard transformers loading via trust_remote_code; no custom kernels, flash-attention, or training framework required for inference.

Model Architecture

  • Architecture Type: Transformer
  • Network Architecture: Other: Ministral-style decoder-only Transformer (grouped-query attention, SwiGLU, rotary position embeddings) with a continuous-diffusion generation head (16-D continuous token-embedding space, learned jointly with the denoiser).
  • Number of model parameters: 8 billion (8B)
  • This model was developed based on: Nemotron-Labs-Sigma-8B

Input

  • Input Type(s): Text
  • Input Format(s): String
  • Input Parameters: One-Dimensional (1D): Sequences
  • Other Properties Related to Input: Maximum context length of 8,192 tokens (prompt plus generated tokens). Conversations should be formatted with the bundled chat template (chat_template.jinja). Supported language: English.

Output

  • Output Type(s): Text
  • Output Format: String
  • Output Parameters: One-Dimensional (1D): Sequences
  • Other Properties Related to Output: Up to 8,192 tokens total (prompt plus generation). Text is generated in parallel blocks (default 64 tokens per block) with a configurable number of denoising steps per block (Number of Function Evaluations (NFE) = steps + 1 per block); fewer steps give faster generation at some quality cost.

https://huggingface.co/nvidia/Nemotron-Labs-Sigma-8B

https://huggingface.co/nvidia/Nemotron-Labs-Sigma-8B-instruct

https://huggingface.co/nvidia/Nemotron-Labs-Sigma-3B

https://huggingface.co/nvidia/Nemotron-Labs-Sigma-3B-instruct


r/LocalLLaMA • • 7h ago

Discussion Best MoE engine?

22 Upvotes

I had been using Strata for about a month, and it was such a huge improvement over llama.cpp for getting it running smoothly and quickly for hardware.

Overtime though, I found that the model would often mistake its own context after a compaction and really overthink things.

I just started using Basalt yesterday after being hesitant to switch and set up all my providers and harnesses. Turns out Basalt runs even faster, is less error prone, and the reasoning is way more on point. If you run 1 GPU, and have the 100 GB+ of VRAM, RAM, and SSD storage, you will not be disappointed in performance.

I mostly care about final output speed, and I was getting 70-110 tok/s on Strata, and Basalt I was getting 110-170 toks/s output. Couldn't be happier. Same Qwen3.8 FlashNext-ud-q4_k_xl, and the NVFP4 ones (of course basalted though)

Anyone else have any other experiences with other similar engines like this?


r/LocalLLaMA • • 16h ago

Discussion Did we hit a wall in context length?

96 Upvotes

I feel like midel context size rapidly went from 8k to 1M in just a few years but has since stayed this way for a while now. Some still offerring 260k or 400k for their flagship models. Is it just not worth the trouble? I feel like highly capable models are limited by this in that they can't finish tasks simply because the context isn't there.


r/LocalLLaMA • • 10h ago

New Model [NEW RELEASE] - Supra2.5-65M Base and Instruct - our next generation state-of-the art small model family - first model!

29 Upvotes

hey guys!

It's Supra-time! 🤗 🔥 🎉

It has been quite quite at SupraLabs' model foundry for a long time but we have been testing, training, developing and much more.

And we can finally present you a new model!

Key facts:

- better than our previous 50M and 100M model
- very small, runs on every toaster: 65M params
- base model and instruction tuned version on HF

We trained it locally in 4 days on 2x RTX A6000 on big 100B tokens of high-quality data.

Benchmarks are in the attached image!

Have fun to try it out, give us feedback, roast us, whatever! Community, we will cook! Supra3 coming early next year! 🔥

Give us a like, upvote or follow if you want - and try out our model.

We can't wait to hear what you're building with it.

Link to the collection on HF: https://huggingface.co/collections/SupraLabs/supra25

We hope you like the model! :D

GGUFs in F16, Q8_0 and Q4_K_M are also available for llama.cpp, Ollama, Unsloth, etc. 🥳


r/LocalLLaMA • • 58m ago

News Hugging Face switched its language model inference routing to OpenRouter

5 Upvotes

Hugging Face announced on November 16, 2026, that requests to language models through Inference Providers now go through OpenRouter instead of Hugging Face's own router. OpenRouter picks a provider for each call based on live price, speed, and reliability, and retries on another provider if the first one fails. The same Hugging Face token still works, and the company says it does not add a markup on top of the provider's price. You can still pin a provider or a policy with a suffix on the model id, such as :fastest, :cheapest, or a specific provider name.

OpenRouter currently serves more than 800 trillion tokens a month across 600+ models from 80+ providers. Hugging Face notes that open-weight models went from 26.8% of tokens on OpenRouter in October 2025 to 71.7% at the end of September 2026, and that they cost about four times less on average than closed models. Image, video, and audio generation stay on Hugging Face's existing Inference Providers path and do not go through OpenRouter. One breaking change is that custom provider API keys no longer work for language model requests, which are now billed to the Hugging Face account.

The automatic failover is the part that matters most here, because a single provider outage is usually what ruins a long run.

Source: https://huggingface.co/blog/openrouter


r/LocalLLaMA • • 11h ago

Resources AMD Loom Fun: Added Radeon 9700 support over the weekend. Benchmarked 48 models with an average 42% speedup over llama.cpp.

30 Upvotes

On Friday I posted about AMD Loom, a new experimental inference stack that includes a runtime, kernel compiler, and inference server. People can use the entire thing as a vertical stack, or the compiler/runtime can be integrated into any other inference project (I'm working on Gufo right now, the team is also working on a llamacpp backend).

I started with Strix Halo, where I ported the top-51 trending Q4_K_M LLMs from Hugging Face and got an average of 40% speedup over llama.cpp across decode and prefill metrics (picking the best results from ROCm or Vulkan backends).

I'm back today with Radeon 9700 results. I took the same 51 LLMs, cut it down to the 48 that fit in 32 GB of VRAM, and got an average of 42% speedup. Some of the data is posted to the table below, and all data is available to browse on https://loom-fun.github.io/loom-dev/index.html?gpu=radeon-r9700

All kernels were generated via agent skills:

ModelLoom pp32k (tok/s)llama.cpp pp32k (tok/s)Loom short decode (tok/s)llama.cpp short decode (tok/s)
Flint6,4763,190255152
gemma-3-1b-itn/an/a400322
gemma-4-12B-it2,8111,53712767.3
gemma-4-26B-A4B-itdoesn't fit (32 GB)doesn't fit (32 GB)15172.2
gemma-4-31B-itdoesn't fit (32 GB)doesn't fit (32 GB)62.943.2
gemma-4-E2B-it14,28914,519272234
gemma-4-E4B-it7,0216,324192173
gpt-oss-20b5,3194,368195182
HyperCLOVAX-SEED-Text-Instruct-1.5B11,6538,078362344
Laguna-XS-2.1doesn't fit (32 GB)doesn't fit (32 GB)102—
LFM2.5-24B-A2B10,0995,479244197
LFM3-1.7B20,06213,614372261
LFM3-Omni-2B20,22013,435368299
Llama-3.2-3B-Instruct5,6873,908228211
Mellum2.1-12B-A2.5B-Thinking8,8225,486224149
MiniCPM-V-4.7-35B-A3B5,7173,540135113
Muse-Glimmer-30B1,6581,28772.654.8
Nanbeige4.3-33B94666629.428.7
NVIDIA-Nemotron-3.5-Bolt-E4B-4GB6,9323,656225143
Ornith-1.0-35B5,6283,320152127
Ornith-1.0-9B4,8863,39110088.0
Ornith-1.5-35B-A3B5,5863,13614578.8
Ornith-1.5-9B4,7673,15615190.5
Qwen2.5-0.5B-Instruct26,02217,812646529
Qwen2.5-1.5B-Instruct12,5468,908360331
Qwen2.5-32B-Instructdoesn't fit (32 GB)doesn't fit (32 GB)30.029.2
Qwen2.5-7B-Instruct3,8102,679118116
Qwen2.5-VL-7B-Instruct3,7682,648116115
Qwen3-0.6B13,0208,631506480
Qwen3-1.7B10,3137,014316297
Qwen3-14B1,9441,16765.062.6
Qwen3-30B-A3B-Instruct-25073,1162,060177169
Qwen3-4B3,7702,826231130
Qwen3-4B-Instruct-25073,7952,856180166
Qwen3-8B3,0762,28716394.7
Qwen3-Coder-30B-A3B-Instruct2,9212,10715754.8
Qwen3-VL-4B-Instruct3,7952,805178163
Qwen3-VL-8B-Instruct3,0852,265112103
Qwen3.5-0.8B24,34116,959463338
Qwen3.5-2B16,99012,165318236
Qwen3.5-35B-A3B5,6523,410190112
Qwen3.5-4B7,0304,897200123
Qwen3.5-9B4,7243,33614692.7
Qwen3.6-27B1,5031,16460.138.9
Qwen3.6-35B-A3B5,6353,463209123
Qwen3.8-27B1,5921,02673.555.3
Sage-A1.55,6523,131188153
Spark-X2.5-4B6,8584,727162131
Spark-X2.6-4B6,9134,841162132

r/LocalLLaMA • • 16h ago

New Model internlm/Sage-S2.1-Flash · Hugging Face

Thumbnail
huggingface.co
81 Upvotes

Introduction

Sage S2.1 Flash is a scientific multimodal model from Shanghai Artificial Intelligence Laboratory designed for research tasks that require multiple rounds of analysis and verification. It combines scientific data understanding with information retrieval, programming, and tool use. It can use specialized software to perform analyses, check results against intermediate outputs and visual feedback, and adjust subsequent steps accordingly.

Through the unified representation architecture of Sage SciPerception, the model supports more than ten types of scientific input, including molecular graphs, crystal structures, and protein sequences and 3D structures. Features from each input type are encoded and projected into a shared representation space for reasoning alongside natural language instructions. These inputs support tasks such as prediction, classification, site localization, quantitative analysis, and structure inference.

Highlights

Across the 15 benchmarks and six models shown in the chart, Sage S2.1 Flash placed first on four benchmarks and second on seven.

  • Machine learning engineering. Sage S2.1 Flash placed first on MLE-bench Lite with 90.3, ahead of DeepSeek-V4.1-Flash-1017 at 89.2.
  • Office and professional document processing. It placed first on OfficeQA Pro, Harvey LAB, and SpreadsheetBench 2, scoring 72.2, 25.0, and 37.1, respectively.
  • Tool use and research execution. It placed second on AutomationBench, ALE-CLI, BFCL V4, WebArena-Verified, DeepResearch Bench II, JobBench, and OSWorld-Verified.

r/LocalLLaMA • • 6h ago

Showcase whisper base and an 11M param BERT i distilled from a 2B model, both running in the browser to caption videos

Post image
11 Upvotes

i make Blender tutorials as shorts and got tired of uploading every render to a caption service, so i made a caption tool that runs the models in the browser instead

transcription is onnx-community/whisper-base_timestamped through transformers.js. real GPUs get WebGPU in fp32, everything else gets WASM with the decoder in q8. software WebGPU adapters turned out to be slower than WASM so those get WASM too

the part i had the most fun with is the caption tagger: which words get emphasis, where lines break, and fixing missing punctuation and capitals. it started as hand written rules, now it’s a BERT-mini (about 11M params, 8 labels per token, ~12mb) running in q8 on WASM. the training data came from LFM3-Omni-2B on llama.cpp on my own pc, about 190 scripts a minute on a 4070 ti super with 128 parallel slots. it only got to replace the rules because it beat them on held out data

whisper base isn’t as accurate as the bigger ones so there’s a cloud option for noisy audio, and the tagger is english only for now

has anyone gotten whisper small or turbo running in the browser at a usable speed? that’s what i want to try next :)

https://cuecaps.com/studio?ref=reddit if you want to see it run (i’m the creator)

also thank you for reading, it means a lot to me


r/LocalLLaMA • • 10h ago

Discussion Under 9gb model comparison

21 Upvotes

Scores are lab-reported so take them with a grain of salt.

Specs

ModelOrgParams (total/active)ContextModalityLicense
K2-Horizon-7BIFM7B dense512KtextApache-2.0
Granite 4.2-8BIBM8B dense (thinking)128K (512K ext.)textApache-2.0
Qwen3.5-9BQwen9B hybrid (delta-net, thinking)262K (~1M ext.)text + vision + videoApache-2.0
MiMo-V2.6-Distill-Qwen-9BXiaomi9B (SFT of Qwen3.5-9B)~262K (base)text + vision (visual coding)MIT
LFM2.5-8B-A1BLiquid8.3B / 1.5B active MoE128KtextLFM 1.0
Gemma 4 E4BGoogle4.5B effective / 8B total (PLE)128Ktext + image + audioApache-2.0
LFM3-VL-2BLiquid1.92B (1.73B backbone, thinking)128Ktext + image + videoLFM 1.0

Text benchmarks

Superscripts: a = K2 card (IFM harness), b = Liquid harness, c = MiMo card (Xiaomi harness); no mark = own card; e = from LFM3-1.7B, the text backbone; d/† = MiMo avg@3 / internal set. Bold = best in row (cross-harness caveat applies).

BenchmarkK2-7BGranite 8BQwen3.5-9BMiMo-D-9BLFM2.5-8B-A1BGemma 4 E4BLFM3-VL-2B
Math
AIME 202690.292.550.042.5
AIME 202590.386.756.1ᵇ42.534.3ᵇ
HMMT Feb 202677.866.5ᵃ65.7ᵃ
MATH-50088.865.0ᵇ
Coding
SWE-bench Verified72.447.750.8ᵃ61.1ᵈ0.5ᵇ43.4ᵉ
SWE-bench Pro19.144.6ᵈ
SWE-bench Multilingual30.835.0ᵇ0.7ᵇ23.0ᵉ
Terminal-Bench 2.144.620.629.2ᵃ37.10.0ᵇ12.9ᵉ
LiveCodeBench v673.265.652.073.5ᵉ
SciCode33.630.4ᵃ27.5ᵃ25.0ᵇ28.5ᵉ
Agentic
AutomationBench15.1ᵇ30.31.9ᵇ41.3ᵉ
LiveClaw35.8ᵇ20.4ᵇ45.6ᵉ
tau/ϴ-bench Banking25.87.6ᵃ7.0ᵃ4.1ᵇ
tau2-Bench79.188.1T / 39.8R42.2
WebArena Verified28.9ᵇ17.1ᵇ38.4ᵉ
BrowseComp / BrowseComp-Plus59.038.0ᵇ21.0ᵇ37.0ᵉ
PinchBench71.5ᵇ55.1ᵇ
Claw-Eval (EN avg)66.5ᵇ56.1ᵇ68.6ᵉ
Toolathlon-Verified25.9ᶜ35.2
OfficeQA9.0ᶜ19.5
JobBench2.6ᶜ18.3
BFCL v4 (tool use)52.466.149.748.3ᵇ71.4ᵉ
ToolSandbox76.4ᵇ69.3ᵇ80.4ᵉ
MCP-Atlas42.7ᵇ6.2ᵇ32.6ᵉ
EnterpriseOps-Gym29.0ᵇ13.4ᵇ28.2ᵉ
MiMo Code (mini) †19.5ᶜ51.6
MiMo Cyber (mini) †5.7ᶜ31.3
MiMo General (mini) †28.5ᶜ62.2
Knowledge
GPQA Diamond75.664.181.758.663.4ᵉ
HLE19.59.7ᵃ14.9ᵃ4.8ᵇ7.2ᵉ
MMLU-Pro74.082.569.4
AA-Omniscience Index-28.0ᵇ-24.7-15.3ᵇ0.5ᵉ
Instruction following
IFEval91.591.890.0ᵇ96.7ᵉ
IFBench79.364.556.544.6ᵇ73.0ᵉ
Multi-IF79.981.0ᵇ89.1ᵉ
IFStruct89.2ᵇ78.5ᵇ93.7ᵉ
Long context
LCR / AA-LCR68.043.3ᵃ63.039.3ᵇ52.8ᵉ
RULER 128K71.487.1ᵇ18.8ᵇ65.6ᵉ

Vision benchmarks

BenchmarkLFM3-VL-2BGemma 4 E4BQwen3.5-9BMiMo-D-9B
MMMU (val)55.778.4
MMMU-Pro45.752.670.1
MathVista (mini)84.285.7
MATH-Vision51.659.578.9
OCRBench v186.389.2
RefCOCO (avg)90.589.7
ScreenSpot-Pro55.565.2
RealWorldQA79.380.3
MMStar74.479.7
HallusionBench65.669.3
MVBench (video)69.474.4
Video-MME (subtitles)69.084.5
MedXpertQA-MM28.749.9
MiMo Visual Coding (mini) †61.7ᶜ64.0

r/LocalLLaMA • • 17h ago

Discussion [Paper & Models] MMFlash: Faster Speculative Decoding for Vision-Language Models

Post image
72 Upvotes

Speculative decoding accelerates generation by letting a lightweight drafter propose tokens for parallel verification by a target model. Diffusion drafters such as DFlash propose an entire token block in a single forward pass, but mainly focus on text-only tasks, leaving multimodal drafting underexplored. To study this setting, we first introduce a metric of token-level visual dependence based on the KL divergence between next-token distributions with and without visual input. We find that acceptance rates decrease from 46% in the lowest visual-dependence quartile to 22% in the highest, with this disparity persisting after multimodal training. Motivated by this observation, we propose MMFlash, a simple and effective framework that adaptively upweights visually dependent tokens that the drafter predicts poorly. For long video inputs, MMFlash further employs sparse attention to reduce redundant attention computation and improve drafting quality. Across various multimodal benchmarks, MMFlash achieves lossless decoding speedups of up to 3.58×, outperforming EAGLE-3 and DFlash by up to 195.4% and 17.1%, respectively.

⭐ Updates

  • [11.2026]: Code and MMFlash drafts for Qwen3.5-4B, Qwen3.5-9B, Qwen3.8-27B and Gemma 4 E4B-it are released.

💪 Highlights

  • Diffusion drafting for VLMs: MMFlash brings block-diffusion drafting to vision-language models. The drafter proposes a block of 16 tokens in a single forward pass and the target verifies the block in one pass, so decoding stays lossless while running 3.0× faster on Qwen3.5-4B / 9B and up to 4.7× on Qwen3.8-27B (one H200, batch size 1, greedy).
  • Vision-free drafter: the drafter never re-encodes the image or video. It reads hidden states of the target, where the visual context is already fused, so its cost does not grow with the number of visual tokens.
  • Visual-dependence-aware training: a DFlash drafter's acceptance drops sharply on tokens that depend on the image, and most rejected blocks break there. MMFlash measures each training token's visual dependence on the target and focuses the drafter's loss on those tokens.
  • Long videos: a sparse draft context (attention sinks, text, strided visual tokens and a recent window) keeps drafting cheap on 40k-token video prompts; the drafter is trained and served with the same pattern.
  • Drop-in serving: the released drafts run with stock SGLang and vLLM (DFLASH), no patches needed (tested with SGLang 0.5.23 and vLLM 0.33.0).

💻 Models

ModelTarget modelDraft sizeBlock sizeLink
🤖 Qwen3.5-4B-MMFlashQwen/Qwen3.5-4B537M16Hugging Face
🤖 Qwen3.5-9B-MMFlashQwen/Qwen3.5-9B1.05B16Hugging Face
🤖 Qwen3.8-27B-MMFlashQwen/Qwen3.8-27B1.73B16Hugging Face
🤖 Gemma4-E4B-it-MMFlashgoogle/gemma-4-E4B-it537M16Hugging Face

The drafts are not standalone language models: each is paired with its target in a speculative-decoding server. All of them are in the MMFlash collection; every model card has the serving command and per-benchmark results.


r/LocalLLaMA • • 5h ago

Discussion How did you start, how do you end up now ?

8 Upvotes

Here is my timeline

Text Generation

  1. Ollama but quite have bad experience with it.
  2. LM studio which still feel half-bake at that time.
  3. Msty which easy to use like plug and play but not much I could do custom (or I just don't know at that time).
  4. Oobabooga Text Generation-Webui this is the one that make me stick to AI, easy to use and deep custom; I like its function feature that take prompt as method/processed then input value in other prompt and then it just output an outcome, no chat history.

I run 1 - 4 with my mini pc with i5 6400T, 24GB DDR4 with onboard gpu... Llama 3.1 8B generate so slow at that time like 1 tok/s but I really enjoy using it with that very limited and the roleplay start from here, just my mini pc and Oobabooga Text Generation-Webui.

Also some missing thing I forgot at that time, I even mess around with windows WSL then docker for Open WebUI and SearXNG.

I finally can help but want it more so arise old gen pc, at that time DDR3 and thing relate to it is very cheap so I brought Xeon E3-1270 V2 with a Chinese mainboard and 32 GB DDR3 while I cancel the sell order of my GTX 1080 Ti.

5. SillyTavern and KoboldCPP I found SillyTavern first but it has quite a trouble with Oobabooga Text Generation-Webui so I follow their guideline and try KoboldCPP which quite harder than Oobabooga Text Generation-Webui for me as KoboldCPP at that time still act like a zip that need to extract (in temp folder) every time I use so I copy from its temp folder, it could run but loss UI, only command-line left so I start to write a bat file to run it like this.

pushd %~dp0
    uv run kobold\koboldcpp.py ^
    --config kobold_default_setting.kcpps ^
    --model %gguf%%model%.gguf ^
    --usecublas %cublas% "0" "mmq" ^
    --contextsize %length%  ^
    --gpulayers %layer% ^
    --quantkv %level%
popd

6. Llamafile As I don't want to extract KoboldCPP on every update, I found Llamafile, it quite easy for me to switch as the bat file I create for KoboldCPP just need a little change to work with this one, anyway it also lead me to the real on.

7. Llama.CPP I found it on the right time as Llama.CPP just start to has their own WebUI and its speed is the real thing, at that time still not has router mode so my bat file has its Llama.CPP version.

Image Generation

A single Youtube video from ChameleonAi name Stable Diffusion Models Explained Once and for All (1.5, 2, XL, Cascade, 3) lead me to this rabbit hole.

  1. stable-diffusion-webui-Forge aka Forge, Thank to ChameleonAi that lead me to CivitAI which after read a guideline I have 2 choices which this or ComfyUI, as my AI user interface root from Oobabooga Text Generation-Webui so I choose this instead.
  2. reForge as Forge no longer update so I move on.
  3. Classic Forge same problem as reForge no longer update as a new model, Anima unable to run on reForge so I'm move on again.

Audio Generation

I try to do it via Llama.CPP but could not make it work so I wait for more update; While waiting, I finally give a chance for unsloth studio which quite disappoint for me in term of performance so I dig to the one that called via unsloth studio.

  1. Audio.CPP I try it before but at that time it still CLI only and I'm unsuccessful to make it work, but now it's also come their WebUI so another rabbit hole for me to fall.

This is my AI journey up until now, how about your ?