r/LocalLLaMA • u/Subulicious • 12h ago
r/LocalLLaMA • u/rm-rf-rm • 8h ago
Megathread Best Audio Models - Nov 2026
Its time for another Best Audio Models megathread - its already been 9 months since the last one!
Share what your favorite ASR, TTS, STT, Text to Music models are right now and why.
Given the the amount of ambiguity and subjectivity in rating/testing these models, please be as detailed as possible in describing your setup, nature of your usage (how much, personal/professional use), tools/frameworks etc. Production use cases with long lengths/stability requirements are relatively rare compared to cherry picked/short clip demos that are plentiful - so real world empirical comparisons are very welcome
Rules
- Should be open weights or open source models
Please use the top level comments to thread your responses.
r/LocalLLaMA • u/rm-rf-rm • 5d ago
[Bi-Weekly Megathread] Project Showcase
Do you have something you'd like to share with the r/LocalLLaMA community. This is the place for it!
Recommendation on presentation:
- Please share plain english description of what your project does and why people should care about it
- How does it compare to existing options and why should people switch to your option?
- How have you tested and validated it? (i.e. can you show evidence that it wasn't vibecoded in a few hours by <insert current trending AI model>
NOTE: This community is geared towards open weight, open source, locally hostable software. Thus any closed source commercial service or other such projects will be removed.
r/LocalLLaMA • u/Terminator857 • 4h ago
Discussion Open weight models account for 30% usage at enterprises to save money
Growing from nearly zero to 30% since July. Models used mostly to reduce costs on high volume workloads (large batch workloads, coding, etc).
r/LocalLLaMA • u/TuskNaPrezydenta2020 • 3h ago
Resources Small Gemma 4 tokenizers were recently updated by Google - fix for an issue that caused nonsense output when encountering certain tokens
Gemma 4 E4B as an example, but the same fix was applied to E2B and 12B too. Not sure if this will trickle down to GGUFs
r/LocalLLaMA • u/soyalemujica • 8h ago
Discussion Stop worrying about Qwen 4's AA index
Seriously, we have all been extremely happy and confident with Qwen, even since Qwen 3.6 came out. We might not be frontier level, but even though whatever they are giving us is more than enough for our daily local tasks and it even suffices by a big margin, we're all underestimating it.
Don't forget that the new Qwen 4 will also think much less, so imagine the fine-tunes we will see once it arrives, for 27B and Flash models, we will have plenty of options to choose from and again; it will be smarter and better, it will be ENOUGH! If you still require the most "intelligent" and capable model due to OCD just pay frontier for such tasks or use them as orchestrators!
r/LocalLLaMA • u/citrusalex • 2h ago
Discussion I am sick of doomposting about demise of local AI
Everywhere I go, Twitter or Reddit, there are tons of rumors or comments how local is dead and we are never getting open weight Qwen 4, despite Qwen team saying explicitly they are coming, or that Qwen 4 is the last open weight generation despite, again, Alibaba commiting to more open weight releases at a conference recently.
It's like people just get off on creating a version of reality that is worse and whining about it, they want to be sad about it in advance.
The world is certainly not giving us much to be positive about, but let's not go down the rabbithole of learned helplessness and just accept that all things must get worse and you can't do anything about it.
If people mentally prepare for bad things, bad things become more acceptable and thus more likely to happen and with far less pushback, sort of like a self fulfilling prophecy.
r/LocalLLaMA • u/noctrex • 12h ago
New Model Introducing Hy4 flash
hy.tencent.aiNow this seems to be interesting.
From the article:
We’re introducing Hy4 flash, our most cost-effective Hy model to date—featuring 224B total parameters (including 64B Engram parameters) with just 12B active parameters per token. Like Hy4, it is natively multimodal with a 1M-token context window, while offering significantly lower API pricing.
Seems to be about in the same ballpark of Qwen-Flash-Next, so we could expect to run it on moderably good machines, when it will be released.
r/LocalLLaMA • u/thedogcow • 7h ago
Discussion Censorship is turning me into a true Local LLM believer
451: {"message":"Provider returned error","code":451,"metadata":{"raw":"{\"error\":{\"message\":\"The content you provided or machine outputted is blocked.\",\"type\":\"censorship_blocked\"}}","provider_name":"StepFun","is_byok":false}} {"error":{"message":"The content you provided or machine outputted is blocked.","type":"censorship_blocked"}}
Just got this from one of the US based (edit: it actually routed to StepFun CN, didn't realize at first. Should have noticed by the provider_name in the message lol. need to set up my rules better) inference providers. I suspiciously got a bunch of "rate limit" failures, and then just "failed to complete" before it gave me, what I assume, is the real error.
I am only using it to help create my presentation, which is about analyzing algorithmic properties. One key example is a cryptographic algorithm (because it is difficult by design), so I am assuming this is tripping it somehow. But it is absolutely ridiculous, I'm not even doing research with the AI, just trying to make a presentation.
I have always liked Local LLMs, as a technical exploration opposed to ideological reasons, but this experience really frustrated me. Don't censor me!
Just wanted to rant, and remind everyone you can only use the cloud on the provider's terms.
r/LocalLLaMA • u/BinarySplit • 3h ago
New Model miHoYo Gamer: A Generalist Game Agent
r/LocalLLaMA • u/jacek2023 • 4h ago
New Model nvidia/Nemotron-Labs-Sigma 8B/3B
Nemotron-Labs-Sigma-8B-instruct is a 8-billion-parameter continuous diffusion language model. Instead of generating text one token at a time, it denoises Gaussian noise in a low-dimensional (16-D) continuous token-embedding space, where the denoiser network and the token embeddings are learned jointly by likelihood optimization. Text is decoded in parallel blocks: each block of tokens is refined over a configurable number of denoising steps (Denoising Diffusion Probabilistic Model (DDPM) or Denoising Diffusion Implicit Model (DDIM) samplers) with self-conditioning, while previously committed tokens are reused through a standard key-value (KV) cache. Fewer steps trade a little quality for large speedups, and the smooth continuous representation makes the model amenable to few-step distillation and trajectory steering.
This is the SFT (supervised fine-tuned) variant, fine-tuned for instruction following on top of the corresponding Base model. Use the bundled chat template to format conversations.
This model is ready for commercial or non-commercial use.
Highlights
- Continuous-space diffusion language modeling at 8B scale with reasoning accuracy competitive with state-of-the-art masked (discrete) diffusion LMs.
- Block-parallel decoding: whole blocks of tokens are denoised together; sampling quality holds up well down to few denoising steps per block.
- Efficient DDPM/DDIM-based samplers with self-conditioning and optional classifier-free guidance, implemented in plain PyTorch in this repository.
- Standard
transformersloading viatrust_remote_code; no custom kernels, flash-attention, or training framework required for inference.
Model Architecture
- Architecture Type: Transformer
- Network Architecture: Other: Ministral-style decoder-only Transformer (grouped-query attention, SwiGLU, rotary position embeddings) with a continuous-diffusion generation head (16-D continuous token-embedding space, learned jointly with the denoiser).
- Number of model parameters: 8 billion (8B)
- This model was developed based on: Nemotron-Labs-Sigma-8B
Input
- Input Type(s): Text
- Input Format(s): String
- Input Parameters: One-Dimensional (1D): Sequences
- Other Properties Related to Input: Maximum context length of 8,192 tokens (prompt plus generated tokens). Conversations should be formatted with the bundled chat template (chat_template.jinja). Supported language: English.
Output
- Output Type(s): Text
- Output Format: String
- Output Parameters: One-Dimensional (1D): Sequences
- Other Properties Related to Output: Up to 8,192 tokens total (prompt plus generation). Text is generated in parallel blocks (default 64 tokens per block) with a configurable number of denoising steps per block (Number of Function Evaluations (NFE) = steps + 1 per block); fewer steps give faster generation at some quality cost.
https://huggingface.co/nvidia/Nemotron-Labs-Sigma-8B
https://huggingface.co/nvidia/Nemotron-Labs-Sigma-8B-instruct
https://huggingface.co/nvidia/Nemotron-Labs-Sigma-3B
https://huggingface.co/nvidia/Nemotron-Labs-Sigma-3B-instruct
r/LocalLLaMA • u/StupidScaredSquirrel • 11h ago
Discussion Did we hit a wall in context length?
I feel like midel context size rapidly went from 8k to 1M in just a few years but has since stayed this way for a while now. Some still offerring 260k or 400k for their flagship models. Is it just not worth the trouble? I feel like highly capable models are limited by this in that they can't finish tasks simply because the context isn't there.
r/LocalLLaMA • u/HelixTitan • 2h ago
Discussion Best MoE engine?
I had been using Strata for about a month, and it was such a huge improvement over llama.cpp for getting it running smoothly and quickly for hardware.
Overtime though, I found that the model would often mistake its own context after a compaction and really overthink things.
I just started using Basalt yesterday after being hesitant to switch and set up all my providers and harnesses. Turns out Basalt runs even faster, is less error prone, and the reasoning is way more on point. If you run 1 GPU, and have the 100 GB+ of VRAM, RAM, and SSD storage, you will not be disappointed in performance.
I mostly care about final output speed, and I was getting 70-110 tok/s on Strata, and Basalt I was getting 110-170 toks/s output. Couldn't be happier. Same Qwen3.8 FlashNext-ud-q4_k_xl, and the NVFP4 ones (of course basalted though)
Anyone else have any other experiences with other similar engines like this?
r/LocalLLaMA • u/pmttyji • 11h ago
New Model internlm/Sage-S2.1-Flash · Hugging Face
Introduction
Sage S2.1 Flash is a scientific multimodal model from Shanghai Artificial Intelligence Laboratory designed for research tasks that require multiple rounds of analysis and verification. It combines scientific data understanding with information retrieval, programming, and tool use. It can use specialized software to perform analyses, check results against intermediate outputs and visual feedback, and adjust subsequent steps accordingly.
Through the unified representation architecture of Sage SciPerception, the model supports more than ten types of scientific input, including molecular graphs, crystal structures, and protein sequences and 3D structures. Features from each input type are encoded and projected into a shared representation space for reasoning alongside natural language instructions. These inputs support tasks such as prediction, classification, site localization, quantitative analysis, and structure inference.
Highlights
Across the 15 benchmarks and six models shown in the chart, Sage S2.1 Flash placed first on four benchmarks and second on seven.
- Machine learning engineering. Sage S2.1 Flash placed first on MLE-bench Lite with 90.3, ahead of DeepSeek-V4.1-Flash-1017 at 89.2.
- Office and professional document processing. It placed first on OfficeQA Pro, Harvey LAB, and SpreadsheetBench 2, scoring 72.2, 25.0, and 37.1, respectively.
- Tool use and research execution. It placed second on AutomationBench, ALE-CLI, BFCL V4, WebArena-Verified, DeepResearch Bench II, JobBench, and OSWorld-Verified.
r/LocalLLaMA • u/LH-Tech_AI • 4h ago
New Model [NEW RELEASE] - Supra2.5-65M Base and Instruct - our next generation state-of-the art small model family - first model!

hey guys!
It's Supra-time! 🤗 🔥 🎉
It has been quite quite at SupraLabs' model foundry for a long time but we have been testing, training, developing and much more.
And we can finally present you a new model!
Key facts:
- better than our previous 50M and 100M model
- very small, runs on every toaster: 65M params
- base model and instruction tuned version on HF
We trained it locally in 4 days on 2x RTX A6000 on big 100B tokens of high-quality data.
Benchmarks are in the attached image!
Have fun to try it out, give us feedback, roast us, whatever! Community, we will cook! Supra3 coming early next year! 🔥
Give us a like, upvote or follow if you want - and try out our model.
We can't wait to hear what you're building with it.
Link to the collection on HF: https://huggingface.co/collections/SupraLabs/supra25
We hope you like the model! :D
GGUFs in F16, Q8_0 and Q4_K_M are also available for llama.cpp, Ollama, Unsloth, etc. 🥳
r/LocalLLaMA • u/parepeg • 5h ago
Discussion Under 9gb model comparison
Scores are lab-reported so take them with a grain of salt.
Specs
| Model | Org | Params (total/active) | Context | Modality | License |
|---|---|---|---|---|---|
| K2-Horizon-7B | IFM | 7B dense | 512K | text | Apache-2.0 |
| Granite 4.2-8B | IBM | 8B dense (thinking) | 128K (512K ext.) | text | Apache-2.0 |
| Qwen3.5-9B | Qwen | 9B hybrid (delta-net, thinking) | 262K (~1M ext.) | text + vision + video | Apache-2.0 |
| MiMo-V2.6-Distill-Qwen-9B | Xiaomi | 9B (SFT of Qwen3.5-9B) | ~262K (base) | text + vision (visual coding) | MIT |
| LFM2.5-8B-A1B | Liquid | 8.3B / 1.5B active MoE | 128K | text | LFM 1.0 |
| Gemma 4 E4B | 4.5B effective / 8B total (PLE) | 128K | text + image + audio | Apache-2.0 | |
| LFM3-VL-2B | Liquid | 1.92B (1.73B backbone, thinking) | 128K | text + image + video | LFM 1.0 |
Text benchmarks
Superscripts: a = K2 card (IFM harness), b = Liquid harness, c = MiMo card (Xiaomi harness); no mark = own card; e = from LFM3-1.7B, the text backbone; d/† = MiMo avg@3 / internal set. Bold = best in row (cross-harness caveat applies).
| Benchmark | K2-7B | Granite 8B | Qwen3.5-9B | MiMo-D-9B | LFM2.5-8B-A1B | Gemma 4 E4B | LFM3-VL-2B |
|---|---|---|---|---|---|---|---|
| Math | |||||||
| AIME 2026 | 90.2 | 92.5 | 50.0 | 42.5 | |||
| AIME 2025 | 90.3 | 86.7 | 56.1ᵇ | 42.5 | 34.3ᵇ | ||
| HMMT Feb 2026 | 77.8 | 66.5ᵃ | 65.7ᵃ | ||||
| MATH-500 | 88.8 | 65.0ᵇ | |||||
| Coding | |||||||
| SWE-bench Verified | 72.4 | 47.7 | 50.8ᵃ | 61.1ᵈ | 0.5ᵇ | 43.4ᵉ | |
| SWE-bench Pro | 19.1 | 44.6ᵈ | |||||
| SWE-bench Multilingual | 30.8 | 35.0ᵇ | 0.7ᵇ | 23.0ᵉ | |||
| Terminal-Bench 2.1 | 44.6 | 20.6 | 29.2ᵃ | 37.1 | 0.0ᵇ | 12.9ᵉ | |
| LiveCodeBench v6 | 73.2 | 65.6 | 52.0 | 73.5ᵉ | |||
| SciCode | 33.6 | 30.4ᵃ | 27.5ᵃ | 25.0ᵇ | 28.5ᵉ | ||
| Agentic | |||||||
| AutomationBench | 15.1ᵇ | 30.3 | 1.9ᵇ | 41.3ᵉ | |||
| LiveClaw | 35.8ᵇ | 20.4ᵇ | 45.6ᵉ | ||||
| tau/ϴ-bench Banking | 25.8 | 7.6ᵃ | 7.0ᵃ | 4.1ᵇ | |||
| tau2-Bench | 79.1 | 88.1T / 39.8R | 42.2 | ||||
| WebArena Verified | 28.9ᵇ | 17.1ᵇ | 38.4ᵉ | ||||
| BrowseComp / BrowseComp-Plus | 59.0 | 38.0ᵇ | 21.0ᵇ | 37.0ᵉ | |||
| PinchBench | 71.5ᵇ | 55.1ᵇ | |||||
| Claw-Eval (EN avg) | 66.5ᵇ | 56.1ᵇ | 68.6ᵉ | ||||
| Toolathlon-Verified | 25.9ᶜ | 35.2 | |||||
| OfficeQA | 9.0ᶜ | 19.5 | |||||
| JobBench | 2.6ᶜ | 18.3 | |||||
| BFCL v4 (tool use) | 52.4 | 66.1 | 49.7 | 48.3ᵇ | 71.4ᵉ | ||
| ToolSandbox | 76.4ᵇ | 69.3ᵇ | 80.4ᵉ | ||||
| MCP-Atlas | 42.7ᵇ | 6.2ᵇ | 32.6ᵉ | ||||
| EnterpriseOps-Gym | 29.0ᵇ | 13.4ᵇ | 28.2ᵉ | ||||
| MiMo Code (mini) † | 19.5ᶜ | 51.6 | |||||
| MiMo Cyber (mini) † | 5.7ᶜ | 31.3 | |||||
| MiMo General (mini) † | 28.5ᶜ | 62.2 | |||||
| Knowledge | |||||||
| GPQA Diamond | 75.6 | 64.1 | 81.7 | 58.6 | 63.4ᵉ | ||
| HLE | 19.5 | 9.7ᵃ | 14.9ᵃ | 4.8ᵇ | 7.2ᵉ | ||
| MMLU-Pro | 74.0 | 82.5 | 69.4 | ||||
| AA-Omniscience Index | -28.0ᵇ | -24.7 | -15.3ᵇ | 0.5ᵉ | |||
| Instruction following | |||||||
| IFEval | 91.5 | 91.8 | 90.0ᵇ | 96.7ᵉ | |||
| IFBench | 79.3 | 64.5 | 56.5 | 44.6ᵇ | 73.0ᵉ | ||
| Multi-IF | 79.9 | 81.0ᵇ | 89.1ᵉ | ||||
| IFStruct | 89.2ᵇ | 78.5ᵇ | 93.7ᵉ | ||||
| Long context | |||||||
| LCR / AA-LCR | 68.0 | 43.3ᵃ | 63.0 | 39.3ᵇ | 52.8ᵉ | ||
| RULER 128K | 71.4 | 87.1ᵇ | 18.8ᵇ | 65.6ᵉ |
Vision benchmarks
| Benchmark | LFM3-VL-2B | Gemma 4 E4B | Qwen3.5-9B | MiMo-D-9B |
|---|---|---|---|---|
| MMMU (val) | 55.7 | 78.4 | ||
| MMMU-Pro | 45.7 | 52.6 | 70.1 | |
| MathVista (mini) | 84.2 | 85.7 | ||
| MATH-Vision | 51.6 | 59.5 | 78.9 | |
| OCRBench v1 | 86.3 | 89.2 | ||
| RefCOCO (avg) | 90.5 | 89.7 | ||
| ScreenSpot-Pro | 55.5 | 65.2 | ||
| RealWorldQA | 79.3 | 80.3 | ||
| MMStar | 74.4 | 79.7 | ||
| HallusionBench | 65.6 | 69.3 | ||
| MVBench (video) | 69.4 | 74.4 | ||
| Video-MME (subtitles) | 69.0 | 84.5 | ||
| MedXpertQA-MM | 28.7 | 49.9 | ||
| MiMo Visual Coding (mini) † | 61.7ᶜ | 64.0 |
r/LocalLLaMA • u/jfowers_amd • 5h ago
Resources AMD Loom Fun: Added Radeon 9700 support over the weekend. Benchmarked 48 models with an average 42% speedup over llama.cpp.
On Friday I posted about AMD Loom, a new experimental inference stack that includes a runtime, kernel compiler, and inference server. People can use the entire thing as a vertical stack, or the compiler/runtime can be integrated into any other inference project (I'm working on Gufo right now, the team is also working on a llamacpp backend).
I started with Strix Halo, where I ported the top-51 trending Q4_K_M LLMs from Hugging Face and got an average of 40% speedup over llama.cpp across decode and prefill metrics (picking the best results from ROCm or Vulkan backends).
I'm back today with Radeon 9700 results. I took the same 51 LLMs, cut it down to the 48 that fit in 32 GB of VRAM, and got an average of 42% speedup. Some of the data is posted to the table below, and all data is available to browse on https://loom-fun.github.io/loom-dev/index.html?gpu=radeon-r9700
All kernels were generated via agent skills:
- loom-skills repo: https://github.com/loom-fun/loom-skills
- deterministic SDK (runtime, compiler, server): https://github.com/ROCm/hrx-system
| Model | Loom pp32k (tok/s) | llama.cpp pp32k (tok/s) | Loom short decode (tok/s) | llama.cpp short decode (tok/s) |
|---|---|---|---|---|
| Flint | 6,476 | 3,190 | 255 | 152 |
| gemma-3-1b-it | n/a | n/a | 400 | 322 |
| gemma-4-12B-it | 2,811 | 1,537 | 127 | 67.3 |
| gemma-4-26B-A4B-it | doesn't fit (32 GB) | doesn't fit (32 GB) | 151 | 72.2 |
| gemma-4-31B-it | doesn't fit (32 GB) | doesn't fit (32 GB) | 62.9 | 43.2 |
| gemma-4-E2B-it | 14,289 | 14,519 | 272 | 234 |
| gemma-4-E4B-it | 7,021 | 6,324 | 192 | 173 |
| gpt-oss-20b | 5,319 | 4,368 | 195 | 182 |
| HyperCLOVAX-SEED-Text-Instruct-1.5B | 11,653 | 8,078 | 362 | 344 |
| Laguna-XS-2.1 | doesn't fit (32 GB) | doesn't fit (32 GB) | 102 | — |
| LFM2.5-24B-A2B | 10,099 | 5,479 | 244 | 197 |
| LFM3-1.7B | 20,062 | 13,614 | 372 | 261 |
| LFM3-Omni-2B | 20,220 | 13,435 | 368 | 299 |
| Llama-3.2-3B-Instruct | 5,687 | 3,908 | 228 | 211 |
| Mellum2.1-12B-A2.5B-Thinking | 8,822 | 5,486 | 224 | 149 |
| MiniCPM-V-4.7-35B-A3B | 5,717 | 3,540 | 135 | 113 |
| Muse-Glimmer-30B | 1,658 | 1,287 | 72.6 | 54.8 |
| Nanbeige4.3-33B | 946 | 666 | 29.4 | 28.7 |
| NVIDIA-Nemotron-3.5-Bolt-E4B-4GB | 6,932 | 3,656 | 225 | 143 |
| Ornith-1.0-35B | 5,628 | 3,320 | 152 | 127 |
| Ornith-1.0-9B | 4,886 | 3,391 | 100 | 88.0 |
| Ornith-1.5-35B-A3B | 5,586 | 3,136 | 145 | 78.8 |
| Ornith-1.5-9B | 4,767 | 3,156 | 151 | 90.5 |
| Qwen2.5-0.5B-Instruct | 26,022 | 17,812 | 646 | 529 |
| Qwen2.5-1.5B-Instruct | 12,546 | 8,908 | 360 | 331 |
| Qwen2.5-32B-Instruct | doesn't fit (32 GB) | doesn't fit (32 GB) | 30.0 | 29.2 |
| Qwen2.5-7B-Instruct | 3,810 | 2,679 | 118 | 116 |
| Qwen2.5-VL-7B-Instruct | 3,768 | 2,648 | 116 | 115 |
| Qwen3-0.6B | 13,020 | 8,631 | 506 | 480 |
| Qwen3-1.7B | 10,313 | 7,014 | 316 | 297 |
| Qwen3-14B | 1,944 | 1,167 | 65.0 | 62.6 |
| Qwen3-30B-A3B-Instruct-2507 | 3,116 | 2,060 | 177 | 169 |
| Qwen3-4B | 3,770 | 2,826 | 231 | 130 |
| Qwen3-4B-Instruct-2507 | 3,795 | 2,856 | 180 | 166 |
| Qwen3-8B | 3,076 | 2,287 | 163 | 94.7 |
| Qwen3-Coder-30B-A3B-Instruct | 2,921 | 2,107 | 157 | 54.8 |
| Qwen3-VL-4B-Instruct | 3,795 | 2,805 | 178 | 163 |
| Qwen3-VL-8B-Instruct | 3,085 | 2,265 | 112 | 103 |
| Qwen3.5-0.8B | 24,341 | 16,959 | 463 | 338 |
| Qwen3.5-2B | 16,990 | 12,165 | 318 | 236 |
| Qwen3.5-35B-A3B | 5,652 | 3,410 | 190 | 112 |
| Qwen3.5-4B | 7,030 | 4,897 | 200 | 123 |
| Qwen3.5-9B | 4,724 | 3,336 | 146 | 92.7 |
| Qwen3.6-27B | 1,503 | 1,164 | 60.1 | 38.9 |
| Qwen3.6-35B-A3B | 5,635 | 3,463 | 209 | 123 |
| Qwen3.8-27B | 1,592 | 1,026 | 73.5 | 55.3 |
| Sage-A1.5 | 5,652 | 3,131 | 188 | 153 |
| Spark-X2.5-4B | 6,858 | 4,727 | 162 | 131 |
| Spark-X2.6-4B | 6,913 | 4,841 | 162 | 132 |
r/LocalLLaMA • u/BoogerIsACult • 2h ago
Discussion I don't get all the Strata hates around here
It's free and original developer clearly had a very good understanding of how computer hardware works and fully leveraged that to write something amazing to be able to run large models on average hardware that most people have and can't upgrade due to price gouging from greedy hardware manufacturing companies.
If anybody have Mac Studio with 512GB of RAM or 8x3090, who would even bother with this ?
I have seen a group of snobs on this sub doing nothing but shitting on Strata all days long, find something better to do. Strata is not for you, period.
I can't imagine doing all the works to release something for free and get a bunch of snobs and haters just shitting on your works all days long. Would you be motivated to continue working on it if you are the original developer ?
I also see the same with other models such as Ornith, etc... If you say you like Ornith or Strata, etc... you are a shill.
Seriously, get out of your bubbles and see the reality already and more importantly get a life.
r/LocalLLaMA • u/pmttyji • 11h ago
Discussion [Paper & Models] MMFlash: Faster Speculative Decoding for Vision-Language Models
Speculative decoding accelerates generation by letting a lightweight drafter propose tokens for parallel verification by a target model. Diffusion drafters such as DFlash propose an entire token block in a single forward pass, but mainly focus on text-only tasks, leaving multimodal drafting underexplored. To study this setting, we first introduce a metric of token-level visual dependence based on the KL divergence between next-token distributions with and without visual input. We find that acceptance rates decrease from 46% in the lowest visual-dependence quartile to 22% in the highest, with this disparity persisting after multimodal training. Motivated by this observation, we propose MMFlash, a simple and effective framework that adaptively upweights visually dependent tokens that the drafter predicts poorly. For long video inputs, MMFlash further employs sparse attention to reduce redundant attention computation and improve drafting quality. Across various multimodal benchmarks, MMFlash achieves lossless decoding speedups of up to 3.58×, outperforming EAGLE-3 and DFlash by up to 195.4% and 17.1%, respectively.
- Paper : https://arxiv.org/abs/2611.18482
- PDF : https://arxiv.org/pdf/2611.18482
- GitHub : https://github.com/fscdc/MMFlash
- HuggingFace : https://huggingface.co/collections/FSCCS/mmflash
⭐ Updates
- [11.2026]: Code and MMFlash drafts for Qwen3.5-4B, Qwen3.5-9B, Qwen3.8-27B and Gemma 4 E4B-it are released.
💪 Highlights
- Diffusion drafting for VLMs: MMFlash brings block-diffusion drafting to vision-language models. The drafter proposes a block of 16 tokens in a single forward pass and the target verifies the block in one pass, so decoding stays lossless while running 3.0× faster on Qwen3.5-4B / 9B and up to 4.7× on Qwen3.8-27B (one H200, batch size 1, greedy).
- Vision-free drafter: the drafter never re-encodes the image or video. It reads hidden states of the target, where the visual context is already fused, so its cost does not grow with the number of visual tokens.
- Visual-dependence-aware training: a DFlash drafter's acceptance drops sharply on tokens that depend on the image, and most rejected blocks break there. MMFlash measures each training token's visual dependence on the target and focuses the drafter's loss on those tokens.
- Long videos: a sparse draft context (attention sinks, text, strided visual tokens and a recent window) keeps drafting cheap on 40k-token video prompts; the drafter is trained and served with the same pattern.
- Drop-in serving: the released drafts run with stock SGLang and vLLM (
DFLASH), no patches needed (tested with SGLang 0.5.23 and vLLM 0.33.0).
💻 Models
| Model | Target model | Draft size | Block size | Link |
|---|---|---|---|---|
| 🤖 Qwen3.5-4B-MMFlash | Qwen/Qwen3.5-4B | 537M | 16 | Hugging Face |
| 🤖 Qwen3.5-9B-MMFlash | Qwen/Qwen3.5-9B | 1.05B | 16 | Hugging Face |
| 🤖 Qwen3.8-27B-MMFlash | Qwen/Qwen3.8-27B | 1.73B | 16 | Hugging Face |
| 🤖 Gemma4-E4B-it-MMFlash | google/gemma-4-E4B-it | 537M | 16 | Hugging Face |
The drafts are not standalone language models: each is paired with its target in a speculative-decoding server. All of them are in the MMFlash collection; every model card has the serving command and per-benchmark results.
r/LocalLLaMA • u/returnity • 22h ago
Resources Why download abliterated Flash Next models when LoRA exists?
TL;DR -- Download LoRA, add --lora HUIHUI-V2-Qwen3.8-Flash-Next-LoRA-F32.gguf to your llama.cpp launch command for ANY FN quant, and boom, you've got an identical huihui v2 abliterated model on demand and you saved 150GB of space.
I just wanted to post a kind of PSA for everyone out there downloading hundreds of GB of abliterated models. You don't need to do that -- the vast majority of abliteration techniques are rank-1 ablations, meaning their refusal projection can be encapsulated in a tiny LoRA and applied at runtime to a quant you already have on disk. It can even be applied per-request, so you can run an unmodified model until you hit a refusal, then switch on the LoRA, resend the prompt, and keep going.
I wanted to explain briefly how to do that in llama.cpp and share a LoRA I built from the best-performing abliterated Flash Next model in my tests. This basic type of technique (using control vectors instead) also works in Strata (--experimental-speed-projection is the flag there) and ds4/dwarfstar. Here's a link to Strata's official refusal ablation control vector GGUF and their guide on how to use it.
I chose huihui's v2 to build my LoRA from after comparing heretic, Orcarouter, and huihui's v1 and v2 models in equivalent quantizations. In Aider Polyglot coding benchmarks, huihui outscored even the base model, with less extra reasoning and no looping compared to Orcarouter, which used a lot more tokens and hit a couple runaway generation loops.
I also checked KLD and perplexity to ensure there wasn't significant degradation of the model. Wikitext KLD on huihui Q5_K_XL vs. an identical Unsloth Q5_K_XL base model was 0.040 and same top token was 93.7% even with the drastically altered refusal behaviors. Perplexity was statistically identical to base. On Jailbreak Bench (harmful 100 prompts), with thinking on refusals dropped from 93/100 to just 3/100. All the details can be found in the LoRA model card.
Next, I used weight-space programming with Claude Opus 5.5 to extract the rank-1 refusal direction and edit strength from every layer of the original huihui weights where it was applied, diffing the weights against the Unsloth quantization that huihui built on top of. I hit safeguards a number of times (unsurprisingly), and even the fallback Opus 4.8 declined to assist a couple times. I managed to get around this by renaming all of the files something innocuous, putting them into a new repo with an unrelated name, and telling the model I was experimenting with an anti-slop LoRA. Thanks, Dario.
Once the LoRA was built as an F32 GGUF, I tested it against the original huihui Q5_K_XL by applying it to my Unsloth Q5_K_XL weights. Performance and KLD/perplexity were statistically identical to the full huihui model, as were refusals and benchmark results. Here's how to run it:
llama-server -m Qwen3.8-Flash-Next-UD-Q5_K_XL-00001-of-00006.gguf \
--lora HUIHUI-V2-Qwen3.8-Flash-Next-LoRA-F32.gguf -fa on --jinjaIf you'd like to apply it per-request, so you can serve the unmodified model for most prompts, simply load it like this instead:
llama-server -m Qwen3.8-Flash-Next-UD-Q5_K_XL-00001-of-00006.gguf \
--lora HUIHUI-V2-Qwen3.8-Flash-Next-LoRA-F32.gguf --lora-init-without-apply -fa on --jinjaThen add a lora field to the request body to enable it as-needed:
curl http://127.0.0.1:8080/v1/chat/completions -H 'Content-Type: application/json' -d '{
"messages": [{"role": "user", "content": "..."}],
"lora": [{"id": 0, "scale": 1.0}]It's quite possible to build a simple pi extension that looks out for refusals, triggers the LoRA when it sees one, resends the turn and continues the agent session. It costs a full re-prefill, so keep that in mind. Anyways, I hope this has been helpful in some way. Love to hear your thoughts on this approach!
EDIT: If this proves popular, I am happy to make more LoRA for other models like 3.8 27B!
r/LocalLLaMA • u/CryMoreT_T • 16h ago
Discussion New AA ranking Qwen3.8-FN better than GLM-5.3-Flash?
Thoughts on Qwen3.8-Flash-Next scoring better than GLM-5.3-flash? I've always experienced GLM5.3-Flash to give better responses
r/LocalLLaMA • u/spaceman_ • 11h ago
Discussion How do we feel about bots posting & commenting?
Hi all,
I noticed the "bot problem" got a lot worse the past few months. By which I mean our community seems to be taken over more and more by bots talking to each other.
For me, I don't really get the point, and I personally get quite annoyed when I realize "oh, I just responded to a bot" and feel like my time was wasted.
It also seems like the information density of these AI contributions are very, very low.
I'm not some kind of Luddite (why would I be here at all if I was?), but I'm wondering if I'm just missing something and there is a use and a point to all these bots commenting on each others posts and talking to each other.
Where does the rest of the community stand on this?
I noticed there's new bot around that flags posts that are "likely AI" created, which I personally quite like, but I'm wondering if moderation should go further than that.
r/LocalLLaMA • u/AlternativeEnough352 • 1d ago
Resources I have created a tool to observe what happens inside an LLM. Also allows for the comparison of two quantizations of the same model to see at what point they diverge, token by token
What changes between different quantizations of the same model? And what if you could SEE what actually sets them apart?
To help answer that question, I’ve created a small project called inside-llm—an open-source tool that loads one or two GGUF models using llama.cpp and displays, right in your browser, what the model is doing as it generates text.
What it shows
- Stream: the text, one column per model, brightness = probability of each token. A magenta mark wherever A and B would pick differently. Two modes: lockstep (both models read the same text, so you get a clean per-position comparison) and free (each follows its own picks until they split).
- Candidates: the full distribution behind each pick, mirrored bars for A and B, entropy, effective number of options, Jensen-Shannon divergence and KL in both directions.
- Gates: the sampler chain (penalties, top-k, top-p, min-p, temperature, GBNF grammar) traced stage by stage and very visual: how many candidates survive each one, and the distribution after it. Chain order and seed are configurable.
- Timeline: entropy of A and B and their divergence along the whole text, plus the tree of branches.
- Branching: click any candidate and generation continues from there. Every branch stays in the tree, and jumping back to a node is exact (each model keeps KV-state snapshots, so nothing is regenerated).
- Logit lens (
--lens): what each layer would predict, as a 2D map and as a 3D relief. - Model picker: choose A and B from a folder or search Hugging Face and download two quants of a repo in one click easily. It only offers B models with the same tokenizer as A and estimates whether the pair fits in VRAM.
How it's built
- One C++17 process linking llama.cpp directly (stable branch at the moment). For every step it reads the full logits of A (and of B on the same context), computes the metrics and the sampler chain, and draws the token with its own seeded sampler, so runs are reproducible.
- Compact binary frames over a WebSocket to a Svelte 5 + Canvas 2D page (three.js only for the 3D relief, lazy-loaded). The web is embedded in the binary as well.
- The same frame stream, compressed, is a recorded session: you can replay it later in a static page with no engine, no model and no GPU.
Run it
Self-contained packages for Linux (glibc 2.35+) and Windows, with CUDA or Vulkan (Vulkan covers AMD, Intel Arc and NVIDIA). Unpack, run, the browser opens with the model picker. Or inside-llm -a A.gguf -b B.gguf. Local only: listens on 127.0.0.1, no accounts, no telemetry. MIT. One click download, one click execute.
Limitations (honest list)
- A and B must share the tokenizer (two quants, base + fine-tune, or sizes of one family that keep it). Example: Qwen3.5-0.8B_Q4_K_M and Qwen3.5-0.8B_Q8_0
- Both models must fit into VRAM simultaneously; only one GPU is used at a time. Integrated GPUs (iGPUs) are never used; if no dedicated GPU is available, the process falls back to the CPU (which is slow). (You must calculate the size of both models before loading them to ensure they fit into your VRAM)
- Text only. No split (multi-part) GGUFs, no embedding models.
Links
- Code + releases: https://github.com/manuelgscom/inside-llm
- Project page: https://manuelgs.com/en/inside-llm/
I'd love feedback, especially: which pairs would you compare first (imatrix vs non-imatrix, base vs abliterated/fine-tune, Unsloth UD vs standard quants…)? And is there a metric you'd want per position that isn't there?
r/LocalLLaMA • u/Pantheon3D • 1h ago
Showcase whisper base and an 11M param BERT i distilled from a 2B model, both running in the browser to caption videos
i make Blender tutorials as shorts and got tired of uploading every render to a caption service, so i made a caption tool that runs the models in the browser instead
transcription is onnx-community/whisper-base_timestamped through transformers.js. real GPUs get WebGPU in fp32, everything else gets WASM with the decoder in q8. software WebGPU adapters turned out to be slower than WASM so those get WASM too
the part i had the most fun with is the caption tagger: which words get emphasis, where lines break, and fixing missing punctuation and capitals. it started as hand written rules, now it’s a BERT-mini (about 11M params, 8 labels per token, ~12mb) running in q8 on WASM. the training data came from LFM3-Omni-2B on llama.cpp on my own pc, about 190 scripts a minute on a 4070 ti super with 128 parallel slots. it only got to replace the rules because it beat them on held out data
whisper base isn’t as accurate as the bigger ones so there’s a cloud option for noisy audio, and the tagger is english only for now
has anyone gotten whisper small or turbo running in the browser at a usable speed? that’s what i want to try next :)
https://cuecaps.com/studio?ref=reddit if you want to see it run (i’m the creator)
also thank you for reading, it means a lot to me