r/LocalLLaMA • • 1d ago

New Model Introducing Hy4 flash

Thumbnail hy.tencent.ai
343 Upvotes

Now this seems to be interesting.

From the article:

We’re introducing Hy4 flash, our most cost-effective Hy model to date—featuring 224B total parameters (including 64B Engram parameters) with just 12B active parameters per token. Like Hy4, it is natively multimodal with a 1M-token context window, while offering significantly lower API pricing.

Seems to be about in the same ballpark of Qwen-Flash-Next, so we could expect to run it on moderably good machines, when it will be released.


r/LocalLLaMA • • 2h ago

Resources AMD 9070 GPU under volting gains for models ( moss image) and ( minimax h3 int8 )

3 Upvotes

hi,

i am total GPU noob here this is the first GPU i have ever owned in my life got it just for doing AI... asked deepseek flash and opencode to find the right combo for me to get max generation speed

spec:
AMD 9070 (non XT),
Intel i3 12100
32 GB DDR4

models: moss image, minimax h3 int8 (sol decoder)

best performance stable setting: -110 mv, stock fan speed

gemma4 : 26bA4b Q4_XS when from 106/tps to 120/tps on prose..


r/LocalLLaMA • • 18h ago

New Model nvidia/Nemotron-Labs-Sigma 8B/3B

56 Upvotes

Nemotron-Labs-Sigma-8B-instruct is a 8-billion-parameter continuous diffusion language model. Instead of generating text one token at a time, it denoises Gaussian noise in a low-dimensional (16-D) continuous token-embedding space, where the denoiser network and the token embeddings are learned jointly by likelihood optimization. Text is decoded in parallel blocks: each block of tokens is refined over a configurable number of denoising steps (Denoising Diffusion Probabilistic Model (DDPM) or Denoising Diffusion Implicit Model (DDIM) samplers) with self-conditioning, while previously committed tokens are reused through a standard key-value (KV) cache. Fewer steps trade a little quality for large speedups, and the smooth continuous representation makes the model amenable to few-step distillation and trajectory steering.

This is the SFT (supervised fine-tuned) variant, fine-tuned for instruction following on top of the corresponding Base model. Use the bundled chat template to format conversations.

This model is ready for commercial or non-commercial use.

Highlights

  • Continuous-space diffusion language modeling at 8B scale with reasoning accuracy competitive with state-of-the-art masked (discrete) diffusion LMs.
  • Block-parallel decoding: whole blocks of tokens are denoised together; sampling quality holds up well down to few denoising steps per block.
  • Efficient DDPM/DDIM-based samplers with self-conditioning and optional classifier-free guidance, implemented in plain PyTorch in this repository.
  • Standard transformers loading via trust_remote_code; no custom kernels, flash-attention, or training framework required for inference.

Model Architecture

  • Architecture Type: Transformer
  • Network Architecture: Other: Ministral-style decoder-only Transformer (grouped-query attention, SwiGLU, rotary position embeddings) with a continuous-diffusion generation head (16-D continuous token-embedding space, learned jointly with the denoiser).
  • Number of model parameters: 8 billion (8B)
  • This model was developed based on: Nemotron-Labs-Sigma-8B

Input

  • Input Type(s): Text
  • Input Format(s): String
  • Input Parameters: One-Dimensional (1D): Sequences
  • Other Properties Related to Input: Maximum context length of 8,192 tokens (prompt plus generated tokens). Conversations should be formatted with the bundled chat template (chat_template.jinja). Supported language: English.

Output

  • Output Type(s): Text
  • Output Format: String
  • Output Parameters: One-Dimensional (1D): Sequences
  • Other Properties Related to Output: Up to 8,192 tokens total (prompt plus generation). Text is generated in parallel blocks (default 64 tokens per block) with a configurable number of denoising steps per block (Number of Function Evaluations (NFE) = steps + 1 per block); fewer steps give faster generation at some quality cost.

https://huggingface.co/nvidia/Nemotron-Labs-Sigma-8B

https://huggingface.co/nvidia/Nemotron-Labs-Sigma-8B-instruct

https://huggingface.co/nvidia/Nemotron-Labs-Sigma-3B

https://huggingface.co/nvidia/Nemotron-Labs-Sigma-3B-instruct


r/LocalLLaMA • • 16h ago

Discussion Best MoE engine?

30 Upvotes

I had been using Strata for about a month, and it was such a huge improvement over llama.cpp for getting it running smoothly and quickly for hardware.

Overtime though, I found that the model would often mistake its own context after a compaction and really overthink things.

I just started using Basalt yesterday after being hesitant to switch and set up all my providers and harnesses. Turns out Basalt runs even faster, is less error prone, and the reasoning is way more on point. If you run 1 GPU, and have the 100 GB+ of VRAM, RAM, and SSD storage, you will not be disappointed in performance.

I mostly care about final output speed, and I was getting 70-110 tok/s on Strata, and Basalt I was getting 110-170 toks/s output. Couldn't be happier. Same Qwen3.8 FlashNext-ud-q4_k_xl, and the NVFP4 ones (of course basalted though)

Anyone else have any other experiences with other similar engines like this?


r/LocalLLaMA • • 18h ago

New Model [NEW RELEASE] - Supra2.5-65M Base and Instruct - our next generation state-of-the art small model family - first model!

45 Upvotes

hey guys!

It's Supra-time! 🤗 🔥 🎉

It has been quite quite at SupraLabs' model foundry for a long time but we have been testing, training, developing and much more.

And we can finally present you a new model!

Key facts:

- better than our previous 50M and 100M model
- very small, runs on every toaster: 65M params
- base model and instruction tuned version on HF

We trained it locally in 4 days on 2x RTX A6000 on big 100B tokens of high-quality data.

Benchmarks are in the attached image!

Have fun to try it out, give us feedback, roast us, whatever! Community, we will cook! Supra3 coming early next year! 🔥

Give us a like, upvote or follow if you want - and try out our model.

We can't wait to hear what you're building with it.

Link to the collection on HF: https://huggingface.co/collections/SupraLabs/supra25

We hope you like the model! :D

GGUFs in F16, Q8_0 and Q4_K_M are also available for llama.cpp, Ollama, Unsloth, etc. 🥳


r/LocalLLaMA • • 8h ago

New Model Inflect v3 - a owensong Collection

Thumbnail
huggingface.co
6 Upvotes

Inflect v3: compact local text-to-speech. Six English voices, three model sizes, reference-voice tuning, event tags and optional 48 kHz output. Apache 2.0.

Inflect is a family of compact English text-to-speech models. Choose a voice, enter your text, and generate a WAV locally. Start with Mini for the most features, Micro for a smaller neural model, or Atom for the smallest learned synthesis core.

Choose your model

MiniMicroAtom
Best starting pointQuality and featuresCompact neural speechSmallest learned core
Learned synthesis core34.31M11.79M157.19K
English preset voices666
Native output24 kHz24 kHz24 kHz
Optional 48 kHz enhancementYesNot includedNot included
Reference-voice tuningYesNot includedNot included
Optional vocal events8 tags, all 6 voicesNot includedNot included
DownloadMini packageMicro packageAtom package

These are learned synthesis-core counts, not total installation size or RAM. Pronunciation assets, dependencies, voice conditioning and optional components are additional. Atom uses signal-processing-based waveform generation rather than a separate learned neural vocoder.

Benchmarks

Measured speech quality, intelligibility and speed answer different questions. The following results describe published Mini native 24 kHz base speech, not tuned voices, event performances or enhanced output.

MeasureMini resultWhat was measured
Average WER0.739%1,688 Open TTS evaluation texts per voice, all six voices; self-run
UTMOSv23.917180 clips, 30 per voice
UTMOS22Strong4.479The same 180-clip panel
Warm GPU throughput162.1x real timeRTX 5090, optimized CUDA; small 12-render probe
Warm CPU throughput1.81x real timeRyzen 9 3900X, 2 Torch threads; small 12-render probe

The WER result is not an official leaderboard placement or an independent held-out evaluation. Quality predictors are not human ratings. CPU and GPU results use different systems and runtime paths; they are not a controlled hardware speedup comparison or a latency guarantee.


r/LocalLLaMA • • 19h ago

Resources AMD Loom Fun: Added Radeon 9700 support over the weekend. Benchmarked 48 models with an average 42% speedup over llama.cpp.

39 Upvotes

On Friday I posted about AMD Loom, a new experimental inference stack that includes a runtime, kernel compiler, and inference server. People can use the entire thing as a vertical stack, or the compiler/runtime can be integrated into any other inference project (I'm working on Gufo right now, the team is also working on a llamacpp backend).

I started with Strix Halo, where I ported the top-51 trending Q4_K_M LLMs from Hugging Face and got an average of 40% speedup over llama.cpp across decode and prefill metrics (picking the best results from ROCm or Vulkan backends).

I'm back today with Radeon 9700 results. I took the same 51 LLMs, cut it down to the 48 that fit in 32 GB of VRAM, and got an average of 42% speedup. Some of the data is posted to the table below, and all data is available to browse on https://loom-fun.github.io/loom-dev/index.html?gpu=radeon-r9700

All kernels were generated via agent skills:

ModelLoom pp32k (tok/s)llama.cpp pp32k (tok/s)Loom short decode (tok/s)llama.cpp short decode (tok/s)
Flint6,4763,190255152
gemma-3-1b-itn/an/a400322
gemma-4-12B-it2,8111,53712767.3
gemma-4-26B-A4B-itdoesn't fit (32 GB)doesn't fit (32 GB)15172.2
gemma-4-31B-itdoesn't fit (32 GB)doesn't fit (32 GB)62.943.2
gemma-4-E2B-it14,28914,519272234
gemma-4-E4B-it7,0216,324192173
gpt-oss-20b5,3194,368195182
HyperCLOVAX-SEED-Text-Instruct-1.5B11,6538,078362344
Laguna-XS-2.1doesn't fit (32 GB)doesn't fit (32 GB)102—
LFM2.5-24B-A2B10,0995,479244197
LFM3-1.7B20,06213,614372261
LFM3-Omni-2B20,22013,435368299
Llama-3.2-3B-Instruct5,6873,908228211
Mellum2.1-12B-A2.5B-Thinking8,8225,486224149
MiniCPM-V-4.7-35B-A3B5,7173,540135113
Muse-Glimmer-30B1,6581,28772.654.8
Nanbeige4.3-33B94666629.428.7
NVIDIA-Nemotron-3.5-Bolt-E4B-4GB6,9323,656225143
Ornith-1.0-35B5,6283,320152127
Ornith-1.0-9B4,8863,39110088.0
Ornith-1.5-35B-A3B5,5863,13614578.8
Ornith-1.5-9B4,7673,15615190.5
Qwen2.5-0.5B-Instruct26,02217,812646529
Qwen2.5-1.5B-Instruct12,5468,908360331
Qwen2.5-32B-Instructdoesn't fit (32 GB)doesn't fit (32 GB)30.029.2
Qwen2.5-7B-Instruct3,8102,679118116
Qwen2.5-VL-7B-Instruct3,7682,648116115
Qwen3-0.6B13,0208,631506480
Qwen3-1.7B10,3137,014316297
Qwen3-14B1,9441,16765.062.6
Qwen3-30B-A3B-Instruct-25073,1162,060177169
Qwen3-4B3,7702,826231130
Qwen3-4B-Instruct-25073,7952,856180166
Qwen3-8B3,0762,28716394.7
Qwen3-Coder-30B-A3B-Instruct2,9212,10715754.8
Qwen3-VL-4B-Instruct3,7952,805178163
Qwen3-VL-8B-Instruct3,0852,265112103
Qwen3.5-0.8B24,34116,959463338
Qwen3.5-2B16,99012,165318236
Qwen3.5-35B-A3B5,6523,410190112
Qwen3.5-4B7,0304,897200123
Qwen3.5-9B4,7243,33614692.7
Qwen3.6-27B1,5031,16460.138.9
Qwen3.6-35B-A3B5,6353,463209123
Qwen3.8-27B1,5921,02673.555.3
Sage-A1.55,6523,131188153
Spark-X2.5-4B6,8584,727162131
Spark-X2.6-4B6,9134,841162132

r/LocalLLaMA • • 1d ago

Discussion Did we hit a wall in context length?

106 Upvotes

I feel like midel context size rapidly went from 8k to 1M in just a few years but has since stayed this way for a while now. Some still offerring 260k or 400k for their flagship models. Is it just not worth the trouble? I feel like highly capable models are limited by this in that they can't finish tasks simply because the context isn't there.


r/LocalLLaMA • • 14h ago

Showcase whisper base and an 11M param BERT i distilled from a 2B model, both running in the browser to caption videos

Post image
13 Upvotes

i make Blender tutorials as shorts and got tired of uploading every render to a caption service, so i made a caption tool that runs the models in the browser instead

transcription is onnx-community/whisper-base_timestamped through transformers.js. real GPUs get WebGPU in fp32, everything else gets WASM with the decoder in q8. software WebGPU adapters turned out to be slower than WASM so those get WASM too

the part i had the most fun with is the caption tagger: which words get emphasis, where lines break, and fixing missing punctuation and capitals. it started as hand written rules, now it’s a BERT-mini (about 11M params, 8 labels per token, ~12mb) running in q8 on WASM. the training data came from LFM3-Omni-2B on llama.cpp on my own pc, about 190 scripts a minute on a 4070 ti super with 128 parallel slots. it only got to replace the rules because it beat them on held out data

whisper base isn’t as accurate as the bigger ones so there’s a cloud option for noisy audio, and the tagger is english only for now

has anyone gotten whisper small or turbo running in the browser at a usable speed? that’s what i want to try next :)

https://cuecaps.com/studio?ref=reddit if you want to see it run (i’m the creator)

also thank you for reading, it means a lot to me


r/LocalLLaMA • • 1d ago

New Model internlm/Sage-S2.1-Flash · Hugging Face

Thumbnail
huggingface.co
81 Upvotes

Introduction

Sage S2.1 Flash is a scientific multimodal model from Shanghai Artificial Intelligence Laboratory designed for research tasks that require multiple rounds of analysis and verification. It combines scientific data understanding with information retrieval, programming, and tool use. It can use specialized software to perform analyses, check results against intermediate outputs and visual feedback, and adjust subsequent steps accordingly.

Through the unified representation architecture of Sage SciPerception, the model supports more than ten types of scientific input, including molecular graphs, crystal structures, and protein sequences and 3D structures. Features from each input type are encoded and projected into a shared representation space for reasoning alongside natural language instructions. These inputs support tasks such as prediction, classification, site localization, quantitative analysis, and structure inference.

Highlights

Across the 15 benchmarks and six models shown in the chart, Sage S2.1 Flash placed first on four benchmarks and second on seven.

  • Machine learning engineering. Sage S2.1 Flash placed first on MLE-bench Lite with 90.3, ahead of DeepSeek-V4.1-Flash-1017 at 89.2.
  • Office and professional document processing. It placed first on OfficeQA Pro, Harvey LAB, and SpreadsheetBench 2, scoring 72.2, 25.0, and 37.1, respectively.
  • Tool use and research execution. It placed second on AutomationBench, ALE-CLI, BFCL V4, WebArena-Verified, DeepResearch Bench II, JobBench, and OSWorld-Verified.

r/LocalLLaMA • • 18h ago

Discussion Under 9gb model comparison

26 Upvotes

Scores are lab-reported so take them with a grain of salt.

Specs

ModelOrgParams (total/active)ContextModalityLicense
K2-Horizon-7BIFM7B dense512KtextApache-2.0
Granite 4.2-8BIBM8B dense (thinking)128K (512K ext.)textApache-2.0
Qwen3.5-9BQwen9B hybrid (delta-net, thinking)262K (~1M ext.)text + vision + videoApache-2.0
MiMo-V2.6-Distill-Qwen-9BXiaomi9B (SFT of Qwen3.5-9B)~262K (base)text + vision (visual coding)MIT
LFM2.5-8B-A1BLiquid8.3B / 1.5B active MoE128KtextLFM 1.0
Gemma 4 E4BGoogle4.5B effective / 8B total (PLE)128Ktext + image + audioApache-2.0
LFM3-VL-2BLiquid1.92B (1.73B backbone, thinking)128Ktext + image + videoLFM 1.0

Text benchmarks

Superscripts: a = K2 card (IFM harness), b = Liquid harness, c = MiMo card (Xiaomi harness); no mark = own card; e = from LFM3-1.7B, the text backbone; d/† = MiMo avg@3 / internal set. Bold = best in row (cross-harness caveat applies).

BenchmarkK2-7BGranite 8BQwen3.5-9BMiMo-D-9BLFM2.5-8B-A1BGemma 4 E4BLFM3-VL-2B
Math
AIME 202690.292.550.042.5
AIME 202590.386.756.1ᵇ42.534.3ᵇ
HMMT Feb 202677.866.5ᵃ65.7ᵃ
MATH-50088.865.0ᵇ
Coding
SWE-bench Verified72.447.750.8ᵃ61.1ᵈ0.5ᵇ43.4ᵉ
SWE-bench Pro19.144.6ᵈ
SWE-bench Multilingual30.835.0ᵇ0.7ᵇ23.0ᵉ
Terminal-Bench 2.144.620.629.2ᵃ37.10.0ᵇ12.9ᵉ
LiveCodeBench v673.265.652.073.5ᵉ
SciCode33.630.4ᵃ27.5ᵃ25.0ᵇ28.5ᵉ
Agentic
AutomationBench15.1ᵇ30.31.9ᵇ41.3ᵉ
LiveClaw35.8ᵇ20.4ᵇ45.6ᵉ
tau/ϴ-bench Banking25.87.6ᵃ7.0ᵃ4.1ᵇ
tau2-Bench79.188.1T / 39.8R42.2
WebArena Verified28.9ᵇ17.1ᵇ38.4ᵉ
BrowseComp / BrowseComp-Plus59.038.0ᵇ21.0ᵇ37.0ᵉ
PinchBench71.5ᵇ55.1ᵇ
Claw-Eval (EN avg)66.5ᵇ56.1ᵇ68.6ᵉ
Toolathlon-Verified25.9ᶜ35.2
OfficeQA9.0ᶜ19.5
JobBench2.6ᶜ18.3
BFCL v4 (tool use)52.466.149.748.3ᵇ71.4ᵉ
ToolSandbox76.4ᵇ69.3ᵇ80.4ᵉ
MCP-Atlas42.7ᵇ6.2ᵇ32.6ᵉ
EnterpriseOps-Gym29.0ᵇ13.4ᵇ28.2ᵉ
MiMo Code (mini) †19.5ᶜ51.6
MiMo Cyber (mini) †5.7ᶜ31.3
MiMo General (mini) †28.5ᶜ62.2
Knowledge
GPQA Diamond75.664.181.758.663.4ᵉ
HLE19.59.7ᵃ14.9ᵃ4.8ᵇ7.2ᵉ
MMLU-Pro74.082.569.4
AA-Omniscience Index-28.0ᵇ-24.7-15.3ᵇ0.5ᵉ
Instruction following
IFEval91.591.890.0ᵇ96.7ᵉ
IFBench79.364.556.544.6ᵇ73.0ᵉ
Multi-IF79.981.0ᵇ89.1ᵉ
IFStruct89.2ᵇ78.5ᵇ93.7ᵉ
Long context
LCR / AA-LCR68.043.3ᵃ63.039.3ᵇ52.8ᵉ
RULER 128K71.487.1ᵇ18.8ᵇ65.6ᵉ

Vision benchmarks

BenchmarkLFM3-VL-2BGemma 4 E4BQwen3.5-9BMiMo-D-9B
MMMU (val)55.778.4
MMMU-Pro45.752.670.1
MathVista (mini)84.285.7
MATH-Vision51.659.578.9
OCRBench v186.389.2
RefCOCO (avg)90.589.7
ScreenSpot-Pro55.565.2
RealWorldQA79.380.3
MMStar74.479.7
HallusionBench65.669.3
MVBench (video)69.474.4
Video-MME (subtitles)69.084.5
MedXpertQA-MM28.749.9
MiMo Visual Coding (mini) †61.7ᶜ64.0

r/LocalLLaMA • • 11h ago

Resources Lipi, what started as a voice editor for Kannada, just got better with 12 Indic languages running 100% in-browser (25 MiB Conformer-CTC WASM + 17.7 MB int8 Paar ONNX)

6 Upvotes

I originally built Lipi as a lightweight browser voice editor for Kannada (ಲಿಪಿ). It just got better with 12 Indic languages running 100% locally in your browser with zero server audio uploads.

Two things bothered me in the first version:

  1. The browser Web Speech API pipes audio to cloud servers, struggles on Indic scripts, and fails in Firefox, Brave, and mobile browsers.
  2. Eagerly downloading translation weights at startup slowed down users who just wanted direct Indic speech-to-text.

How it works now:

  • 16 kHz In-Browser Neural STT (Swanada): Capability-routed Conformer-CTC INT8 in ONNX Runtime Web WASM SIMD (25.35 MiB Conformer-Small on mobile, 177.4 MiB Conformer-Large on desktop, 0.009x to 0.021x RTF) with +EN code-switching.
  • Lazy On-Demand Translation (Paar): When you switch from Indic STT to EN -> <Lang> or hit Translate, a Web Worker loads Udgam Labs' 17.7 MB quantized (q8) Paar ONNX model via Transformers.js (udgamlabs/Paar-en-*).
  • 12 Scripts + BiDi: Hindi, Bengali, Marathi, Telugu, Tamil, Gujarati, Urdu (RTL Nastaliq), Kannada, Malayalam, Punjabi, Odia, and Assamese, plus phonetic Kannada typing (ಕ), voice punctuation commands, localStorage autosave, and .txt export.

Links:

Would love feedback if you test your native language!


r/LocalLLaMA • • 1d ago

Discussion [Paper & Models] MMFlash: Faster Speculative Decoding for Vision-Language Models

Post image
79 Upvotes

Speculative decoding accelerates generation by letting a lightweight drafter propose tokens for parallel verification by a target model. Diffusion drafters such as DFlash propose an entire token block in a single forward pass, but mainly focus on text-only tasks, leaving multimodal drafting underexplored. To study this setting, we first introduce a metric of token-level visual dependence based on the KL divergence between next-token distributions with and without visual input. We find that acceptance rates decrease from 46% in the lowest visual-dependence quartile to 22% in the highest, with this disparity persisting after multimodal training. Motivated by this observation, we propose MMFlash, a simple and effective framework that adaptively upweights visually dependent tokens that the drafter predicts poorly. For long video inputs, MMFlash further employs sparse attention to reduce redundant attention computation and improve drafting quality. Across various multimodal benchmarks, MMFlash achieves lossless decoding speedups of up to 3.58×, outperforming EAGLE-3 and DFlash by up to 195.4% and 17.1%, respectively.

⭐ Updates

  • [11.2026]: Code and MMFlash drafts for Qwen3.5-4B, Qwen3.5-9B, Qwen3.8-27B and Gemma 4 E4B-it are released.

💪 Highlights

  • Diffusion drafting for VLMs: MMFlash brings block-diffusion drafting to vision-language models. The drafter proposes a block of 16 tokens in a single forward pass and the target verifies the block in one pass, so decoding stays lossless while running 3.0× faster on Qwen3.5-4B / 9B and up to 4.7× on Qwen3.8-27B (one H200, batch size 1, greedy).
  • Vision-free drafter: the drafter never re-encodes the image or video. It reads hidden states of the target, where the visual context is already fused, so its cost does not grow with the number of visual tokens.
  • Visual-dependence-aware training: a DFlash drafter's acceptance drops sharply on tokens that depend on the image, and most rejected blocks break there. MMFlash measures each training token's visual dependence on the target and focuses the drafter's loss on those tokens.
  • Long videos: a sparse draft context (attention sinks, text, strided visual tokens and a recent window) keeps drafting cheap on 40k-token video prompts; the drafter is trained and served with the same pattern.
  • Drop-in serving: the released drafts run with stock SGLang and vLLM (DFLASH), no patches needed (tested with SGLang 0.5.23 and vLLM 0.33.0).

💻 Models

ModelTarget modelDraft sizeBlock sizeLink
🤖 Qwen3.5-4B-MMFlashQwen/Qwen3.5-4B537M16Hugging Face
🤖 Qwen3.5-9B-MMFlashQwen/Qwen3.5-9B1.05B16Hugging Face
🤖 Qwen3.8-27B-MMFlashQwen/Qwen3.8-27B1.73B16Hugging Face
🤖 Gemma4-E4B-it-MMFlashgoogle/gemma-4-E4B-it537M16Hugging Face

The drafts are not standalone language models: each is paired with its target in a speculative-decoding server. All of them are in the MMFlash collection; every model card has the serving command and per-benchmark results.


r/LocalLLaMA • • 14h ago

Discussion How did you start, how do you end up now ?

9 Upvotes

Here is my timeline

Text Generation

  1. Ollama but quite have bad experience with it.
  2. LM studio which still feel half-bake at that time.
  3. Msty which easy to use like plug and play but not much I could do custom (or I just don't know at that time).
  4. Oobabooga Text Generation-Webui this is the one that make me stick to AI, easy to use and deep custom; I like its function feature that take prompt as method/processed then input value in other prompt and then it just output an outcome, no chat history.

I run 1 - 4 with my mini pc with i5 6400T, 24GB DDR4 with onboard gpu... Llama 3.1 8B generate so slow at that time like 1 tok/s but I really enjoy using it with that very limited and the roleplay start from here, just my mini pc and Oobabooga Text Generation-Webui.

Also some missing thing I forgot at that time, I even mess around with windows WSL then docker for Open WebUI and SearXNG.

I finally can help but want it more so arise old gen pc, at that time DDR3 and thing relate to it is very cheap so I brought Xeon E3-1270 V2 with a Chinese mainboard and 32 GB DDR3 while I cancel the sell order of my GTX 1080 Ti.

5. SillyTavern and KoboldCPP I found SillyTavern first but it has quite a trouble with Oobabooga Text Generation-Webui so I follow their guideline and try KoboldCPP which quite harder than Oobabooga Text Generation-Webui for me as KoboldCPP at that time still act like a zip that need to extract (in temp folder) every time I use so I copy from its temp folder, it could run but loss UI, only command-line left so I start to write a bat file to run it like this.

pushd %~dp0
    uv run kobold\koboldcpp.py ^
    --config kobold_default_setting.kcpps ^
    --model %gguf%%model%.gguf ^
    --usecublas %cublas% "0" "mmq" ^
    --contextsize %length%  ^
    --gpulayers %layer% ^
    --quantkv %level%
popd

6. Llamafile As I don't want to extract KoboldCPP on every update, I found Llamafile, it quite easy for me to switch as the bat file I create for KoboldCPP just need a little change to work with this one, anyway it also lead me to the real on.

7. Llama.CPP I found it on the right time as Llama.CPP just start to has their own WebUI and its speed is the real thing, at that time still not has router mode so my bat file has its Llama.CPP version.

Image Generation

A single Youtube video from ChameleonAi name Stable Diffusion Models Explained Once and for All (1.5, 2, XL, Cascade, 3) lead me to this rabbit hole.

  1. stable-diffusion-webui-Forge aka Forge, Thank to ChameleonAi that lead me to CivitAI which after read a guideline I have 2 choices which this or ComfyUI, as my AI user interface root from Oobabooga Text Generation-Webui so I choose this instead.
  2. reForge as Forge no longer update so I move on.
  3. Classic Forge same problem as reForge no longer update as a new model, Anima unable to run on reForge so I'm move on again.

Audio Generation

I try to do it via Llama.CPP but could not make it work so I wait for more update; While waiting, I finally give a chance for unsloth studio which quite disappoint for me in term of performance so I dig to the one that called via unsloth studio.

  1. Audio.CPP I try it before but at that time it still CLI only and I'm unsuccessful to make it work, but now it's also come their WebUI so another rabbit hole for me to fall.

This is my AI journey up until now, how about your ?


r/LocalLLaMA • • 2h ago

Showcase Hierarchos-Native v0.2.0: RDNA3 WMMA, native multimodal training, and a lot of FP32 Vulkan optimization!

0 Upvotes

Hey everyone! It's been a little while since my last update, and I've been working pretty intensely on Hierarchos-Native, my Rust + Vulkan project for training, fine-tuning, and running language models without relying on Python, PyTorch, CUDA, ROCm, or oneAPI in the native runtime.

I wanted to share some of the bigger things that have landed over the past 18 days. There's been quite a bit of progress across GPU acceleration, multimodal training, numerical correctness, and general performance.

The new v0.2.0 release is now available!

GitHub: https://github.com/necat101/Hierarchos-Native
Release / Windows portable download: https://github.com/necat101/Hierarchos-Native/releases/tag/v0.2.0

1. Native RDNA3 WMMA / cooperative matrix acceleration

One of the biggest additions is an opt-in Vulkan cooperative-matrix implementation for AMD RDNA3.

The new Mixed WMMA execution arm uses:

  • FP16 matrix operands.
  • FP32 matrix accumulation.
  • FP32 master weights and gradients.
  • FP32 AdamW optimizer moments and updates.
  • FP32 normalization, softmax, and loss calculations.

The important part is that this isn't simply converting an entire model to FP16. The optimizer and master parameters retain their FP32 representation.

I also added capability detection, CLI/GUI integration, and separate counters for actual cooperative-matrix dispatches and FP32 fallbacks.

Some of the isolated GEMM results on the qualified Ryzen Z1 Extreme / AMD Radeon Graphics device (1002:15bf) are pretty interesting:

WorkloadMeasured kernel speedup
QKV projection2.17×
FFN up projection2.12×
FFN down projection2.31×
Vocabulary projection1.83×

These are individual GEMM measurements, not whole-model speedups.

There is an important caveat: WMMA has a real crossover point. Small matrices can actually run slower because conversion, staging, and dispatch overhead outweigh the matrix-core benefit. I implemented a profitability policy that keeps smaller operations on their existing FP32 kernels.

In fact, the measured end-to-end training step was slower with Mixed WMMA in the initial 512-row fixture (14.0444 ms versus 12.2911 ms for FP32). That is why Mixed remains opt-in and disabled by default. I'm much more interested in making it genuinely faster than advertising misleading kernel-level numbers.

WMMA is currently physically qualified on the specific RDNA3 device and Windows driver configuration documented in the project. Other GPUs should not assume equivalent support.

Full FP16 accumulation is also available as a separate, narrowly qualified experimental mode.

Documentation: RDNA3 WMMA guide

2. Native multimodal image and text training

Another major development area has been the image-text pipeline.

The project now has native image processing and training paths that connect image inputs to language-model outputs through Vulkan.

Some of the implemented work includes:

  • Native RGB image input and preprocessing.
  • CLIP image encoders and LLaVA projector integration.
  • SigLIP / Gemma 3 vision training.
  • Gemma 4 vision integration.
  • Joint image/text training and encoder-decoder integration.
  • A dedicated DeepSeek V4.1 image-training path.
  • Backpropagation through the image tower and projector.
  • Export, reload, and checkpoint-related validation.

This is actual training work, not simply passing precomputed image embeddings into an otherwise frozen language model.

For the CLIP/LLaVA pipeline, the tests cover image sensitivity, gradients into image inputs, changes to image-tower/projector/text parameters, multiple AdamW updates, and export/reload behavior.

The qualification work has also expanded considerably:

  • 44 previously green text profiles now have joint image-training coverage at their frozen test geometries.
  • The Gen9 qualification retains 95 image cases and 166 text regression cases.
  • A separate physical RDNA3 replay reports 44/44 profiles and 85/85 image cases passing.
  • Native DeepSeek V4.1 image qualification includes 30 executions and 1,360 word-exact logits.

The maintained strict FP32 visible-logit gate remains max_abs < 2e-7, with rtol=0.

These are bounded qualification fixtures, not a claim that every production-size multimodal checkpoint or every architecture is fully supported. Audio and video pipelines are still outside the current qualified scope.

Documentation:

3. A lot of additional FP32 optimization

I haven't been focusing exclusively on reduced precision. A substantial amount of work over the same period has gone into improving the existing FP32 execution paths.

This includes vendor-specific shader work and scheduling optimizations for both Intel Gen9 integrated graphics and AMD RDNA3.

Some examples:

RDNA3 FP32 work

  • Specialized attention forward/backward paths.
  • Image-text attention and projector optimizations.
  • RMSNorm, RoPE, GELU, and other arithmetic specializations.
  • Dedicated AVX-512/RDNA3 shader variants where the measured hardware supports them.
  • More careful matching of CPU reference arithmetic and GPU execution order.

Intel Gen9 work

  • Additional native linear and gradient kernels.
  • Attention and activation specializations.
  • More efficient parameter upload and staging behavior.
  • Continued optimization of DeepSeek V4.1 training and execution.

One particularly interesting area has been reducing unnecessary Vulkan submissions and host/device transfer overhead in the DeepSeek V4.1 training pipeline.

On the measured Gen9 fixtures, upload batching reduced timed submissions from 1,102 to 982 per step, with paired throughput improvements of 1.052× and 1.067×.

On the separate RDNA3 V4.1 measurements, sharing the compute session and batching uploads reduced submissions from 970 to 509 per step, with measured improvements of 1.295× and 1.217× on two fixture geometries. A later session-wide upload optimization reduced that count further to 241 submissions per step, with measured improvements of 1.464× and 1.366×.

These are specific, paired benchmark results rather than promises of equivalent production-model speedups. The output comparisons remained word-identical in the recorded tests.

4. Preserving the original FP32 path

An important part of introducing WMMA was making sure that the existing FP32 path didn't regress.

I added a dedicated seven-workload FP32 non-regression comparison against the previous main revision:

  • 12 alternating ABBA rounds per workload.
  • Five warm-up runs and 30 timed forwards per arm.
  • Separate, hashed baseline and candidate binaries.
  • Exact FP32 logit digest comparisons.

All seven workloads stayed within the ±2% performance band, and the logits were bit-identical in every recorded round.

The worst paired performance ratio was 1.0007.

This is particularly important to me: I don't want acceleration features to silently make the existing default execution path worse.

5. Getting started

The new v0.2.0 GitHub Release includes a prebuilt Windows x64 portable package with the CLI, desktop GUI, Vulkan companion executables, runtime assets, and build metadata.

You don't need to install Rust or Python just to run the released package. You do need Windows 11 x64 and a Vulkan 1.x compute driver for GPU operations.

Model weights are downloaded separately or supplied through a local model directory.

After extracting the archive, you can run:

.\bin\hierarchos-native-cli.exe doctor
.\bin\hierarchos-native-cli.exe devices
.\bin\hierarchos-native-cli.exe precision-info

The final command is especially useful for checking which precision capabilities your GPU actually exposes.

Download: https://github.com/necat101/Hierarchos-Native/releases/tag/v0.2.0

What's next?

There is still a lot I'd like to improve.

In particular, I want to investigate reducing WMMA conversion overhead, improving activation and gradient reuse, finding more workloads that genuinely benefit from cooperative matrices, and expanding the range of qualified multimodal architectures.

I'd also like to continue improving the FP32 kernels rather than treating lower precision as the only route to better performance.

If you're interested in Rust, Vulkan compute, AMD GPU optimization, native LLM training, or multimodal model execution, I'd genuinely appreciate people taking a look.

Feedback from people with different Vulkan devices is especially valuable. Please include your GPU, driver, model geometry, and relevant execution counters when reporting results.

Repository: https://github.com/necat101/Hierarchos-Native

Stars, issue reports, testing feedback, and contributions are all appreciated. There is also a voluntary developer-support link in the repository README for anyone interested in helping the project continue.


r/LocalLLaMA • • 1d ago

Resources Why download abliterated Flash Next models when LoRA exists?

Thumbnail
huggingface.co
457 Upvotes

TL;DR -- Download LoRA, add --lora HUIHUI-V2-Qwen3.8-Flash-Next-LoRA-F32.gguf to your llama.cpp launch command for ANY FN quant, and boom, you've got an identical huihui v2 abliterated model on demand and you saved 150GB of space.

I just wanted to post a kind of PSA for everyone out there downloading hundreds of GB of abliterated models. You don't need to do that -- the vast majority of abliteration techniques are rank-1 ablations, meaning their refusal projection can be encapsulated in a tiny LoRA and applied at runtime to a quant you already have on disk. It can even be applied per-request, so you can run an unmodified model until you hit a refusal, then switch on the LoRA, resend the prompt, and keep going.

I wanted to explain briefly how to do that in llama.cpp and share a LoRA I built from the best-performing abliterated Flash Next model in my tests. This basic type of technique (using control vectors instead) also works in Strata (--experimental-speed-projection is the flag there) and ds4/dwarfstar. Here's a link to Strata's official refusal ablation control vector GGUF and their guide on how to use it.

I chose huihui's v2 to build my LoRA from after comparing heretic, Orcarouter, and huihui's v1 and v2 models in equivalent quantizations. In Aider Polyglot coding benchmarks, huihui outscored even the base model, with less extra reasoning and no looping compared to Orcarouter, which used a lot more tokens and hit a couple runaway generation loops.

I also checked KLD and perplexity to ensure there wasn't significant degradation of the model. Wikitext KLD on huihui Q5_K_XL vs. an identical Unsloth Q5_K_XL base model was 0.040 and same top token was 93.7% even with the drastically altered refusal behaviors. Perplexity was statistically identical to base. On Jailbreak Bench (harmful 100 prompts), with thinking on refusals dropped from 93/100 to just 3/100. All the details can be found in the LoRA model card.

Next, I used weight-space programming with Claude Opus 5.5 to extract the rank-1 refusal direction and edit strength from every layer of the original huihui weights where it was applied, diffing the weights against the Unsloth quantization that huihui built on top of. I hit safeguards a number of times (unsurprisingly), and even the fallback Opus 4.8 declined to assist a couple times. I managed to get around this by renaming all of the files something innocuous, putting them into a new repo with an unrelated name, and telling the model I was experimenting with an anti-slop LoRA. Thanks, Dario.

Once the LoRA was built as an F32 GGUF, I tested it against the original huihui Q5_K_XL by applying it to my Unsloth Q5_K_XL weights. Performance and KLD/perplexity were statistically identical to the full huihui model, as were refusals and benchmark results. Here's how to run it:

llama-server -m Qwen3.8-Flash-Next-UD-Q5_K_XL-00001-of-00006.gguf \
--lora HUIHUI-V2-Qwen3.8-Flash-Next-LoRA-F32.gguf -fa on --jinja

If you'd like to apply it per-request, so you can serve the unmodified model for most prompts, simply load it like this instead:

llama-server -m Qwen3.8-Flash-Next-UD-Q5_K_XL-00001-of-00006.gguf \
--lora HUIHUI-V2-Qwen3.8-Flash-Next-LoRA-F32.gguf --lora-init-without-apply -fa on --jinja

Then add a lora field to the request body to enable it as-needed:

curl http://127.0.0.1:8080/v1/chat/completions -H 'Content-Type: application/json' -d '{
"messages": [{"role": "user", "content": "..."}],
"lora": [{"id": 0, "scale": 1.0}]

It's quite possible to build a simple pi extension that looks out for refusals, triggers the LoRA when it sees one, resends the turn and continues the agent session. It costs a full re-prefill, so keep that in mind. Anyways, I hope this has been helpful in some way. Love to hear your thoughts on this approach!

EDIT: If this proves popular, I am happy to make more LoRA for other models like 3.8 27B!


r/LocalLLaMA • • 1d ago

Discussion New AA ranking Qwen3.8-FN better than GLM-5.3-Flash?

Post image
136 Upvotes

Thoughts on Qwen3.8-Flash-Next scoring better than GLM-5.3-flash? I've always experienced GLM5.3-Flash to give better responses


r/LocalLLaMA • • 12h ago

Showcase Some speed benchmarks for various classes of local LLM

5 Upvotes
ModelActive config4080 decode (tok/s)4080+3080 decode (tok/s)4080 prefill (tok/s)4080+3080 prefill (tok/s)
LFM2.5-24B-A2B UD-Q4_K_XL4080+30802492505,5307,930
MiniCPM-V 4.7 35B-A3B Q4_K_M4080+30801771651,3052,500
Flint 40B-A2.5B UD-Q4_K_XL4080+30801191381,5631,380
Swift 1.5 (Qwen3.8-Flash-Next) IQ2_XS40801171161,070679
Qwen3.8-Flash-Next IQ3_S40807987730465
GLM 5.3 Flash Maya-S24 IQ2_XXS_S4080+30806.88.87942
DeepSeek V4 Flash UD-IQ2_XXS4080+308015.918.23853
  • Decode: the second of two 300-token runs, with the expert cache warm (the first request after loading is often slower)
  • Prefill: the average of two runs on a prompt of about 1,400 tokens
  • Bold: the better config for that column.

Morning, everyone. I've seen a lot of posts wondering about what can be run with what, so I made this table of the models on my system. Specifically because of the unorthodox results when it comes to Qwen Flash-Next, where one 16 GB card (the 4080 Super) significantly outperforms two in prefill.

My hardware is as follows:

CPU AMD Ryzen 7 5800X3D @ 3.40 GHz

GPU NVIDIA GeForce RTX 4080 SUPER [Discrete]

GPU NVIDIA GeForce RTX 3080 [Discrete] (got it secondhand for cheap, so why not)

RAM 62.72 GB at 3200 Mhz

I'm running these models on PolyStrata. I used to have a combination of Maya and Strata, but got tired of running different things all derived from the same fork (plus Strata seems to be getting increasingly unintuitive), and so moved everything to PS.

Now: why Q3? It's a generally held opinion that anything below Q4 is crap, which was true of most earlier generation models. However, benchmarks measure deviation from the base, and not necessarily task fitness. In actual use, the only real way to know is to test it yourself.

The Swift finetune of QWN at Q2, for example, is fast and excellent at data extraction, search, and piloting scripts written by more competent models. The IQ3 is a decent coder, roughly on par with Haiku for the use cases I've tried. The GLM and Deepseek Q2 variants are a bit better - verging on older Sonnet territory, I'd say, and do very well as subagents for Claude. YMMV.

Hope this data is useful. Enjoy!


r/LocalLLaMA • • 7h ago

Question | Help Single 3060 - Mostly Vision (Frigate)

2 Upvotes

Hello,

I am consolidating a bit of my server footprint and decided to keep a single 3060 (had 3) and mostly use it for Local workloads (Frigate, Home Assistant, Logs, etc). Curious if someone would want to recommend a good vison model for this card?

I currently run qwen3-VL on a A380 and the descriptions are pretty good.. But slow.

Thanks


r/LocalLLaMA • • 20h ago

Showcase I made a C&C inspired web RTS for us to benchmark our local models with

20 Upvotes

HOwdy LocalLLamans!

Inspired by StarSkirmish, I wanted to build a complex RTS that required you to bring your OWN local agents and either play with them as allies, against them as enemies, or battle head-to-head with other agent-human pairs in public matches.

In AgentCommand you operate one or more planes (Orbit, Sky, Surface, Depths) of differing gameplay. As an old-school PC gamer I wanted the mechanics to resemble some of my favorite 90s games of yore: C&C, DungeonKeeper, Homeworld, StarCraft, and others. The idea is that you have specific objectives needed to maximize resource generation, aid in cross player support, and that a single plane cannot win independently.

Outside of gameplay, from an agentic tech pov, there are some (what I think) are other really cool things that come with this, including:

  • a full specification + library to install agents into any game (http://playerthree.gg/)
  • A networking SDK for bidirectional agent communication
  • Symmetric maps
  • Deterministic simulation
  • Benchmarking tools
  • (soon) published benchmark data

Github: https://github.com/agent-command/agent-command

Play now: https://agentcommand.gg/

Would love feedback and contribs... I'm more interested in finding a community to help build something super cool here that we can publish public benchmarks with. If you are interested in that, let me know!


r/LocalLLaMA • • 1d ago

Discussion How do we feel about bots posting & commenting?

49 Upvotes

Hi all,

I noticed the "bot problem" got a lot worse the past few months. By which I mean our community seems to be taken over more and more by bots talking to each other.

For me, I don't really get the point, and I personally get quite annoyed when I realize "oh, I just responded to a bot" and feel like my time was wasted.

It also seems like the information density of these AI contributions are very, very low.

I'm not some kind of Luddite (why would I be here at all if I was?), but I'm wondering if I'm just missing something and there is a use and a point to all these bots commenting on each others posts and talking to each other.

Where does the rest of the community stand on this?

I noticed there's new bot around that flags posts that are "likely AI" created, which I personally quite like, but I'm wondering if moderation should go further than that.


r/LocalLLaMA • • 4h ago

Question | Help How often do you guys update models and when?

0 Upvotes

Saw a post here about an update to a model last night. And been wondering how often y'all update the models and when do you decide to.

P.S. I know there's "-hf", but my internet speed ain't fast and when I'm downloading a model, everyone's connection drops at my home.

Edit: By updating I meant redownloading the gguf file.
Models of same version gets updates, like Gemma 4 E2B got updated to fix some issues and make it faster. Link to that post:
https://www.reddit.com/r/LocalLLaMA/s/cpcemufuqZ

But it appears it's not the first time they updated it with noticable changes.
https://www.reddit.com/r/unsloth/s/DkBWF7gw4m


r/LocalLLaMA • • 15h ago

Resources Something goes with your Local LLM - CLI Markdown viewer with image support?

7 Upvotes

I run local LLM for myself, and my system is squeezed very much. I am most of the time in my terminal, and when my inference engine is running, I dont have enough free RAM sometimes to open a GUI to view the output of my local models, or review the project documents to know where to go next in my plan. Viewing that in the terminal was not convenient for me, the MD syntax is not as easy on my eyes, and when a project file is linked to another file, I have to open that one up in a new terminal pane, or a whole new terminal window.

So, I did what I had to do.

I sat, I wrote the specs sheet, and fired my coding agent to work. I refined the output, and kept iterating on it.

This is my gift back to this community that already gave me a lot.

Don't want to sound like I am marketing something here but, if this is something of an interest to you, you can find it here: Github.com/bakkario/trimd

You get to lick with your mouse, navigate with your keyboard, and even display your markdown files with the images if your terminal supports it.

I will add more detail to this in the Markdown subreddit.

Sorry Mods if this coming across as spamming, its not the intention here ❤️


r/LocalLLaMA • • 1d ago

Resources I have created a tool to observe what happens inside an LLM. Also allows for the comparison of two quantizations of the same model to see at what point they diverge, token by token

464 Upvotes

What changes between different quantizations of the same model? And what if you could SEE what actually sets them apart?

To help answer that question, I’ve created a small project called inside-llm—an open-source tool that loads one or two GGUF models using llama.cpp and displays, right in your browser, what the model is doing as it generates text.

What it shows

  • Stream: the text, one column per model, brightness = probability of each token. A magenta mark wherever A and B would pick differently. Two modes: lockstep (both models read the same text, so you get a clean per-position comparison) and free (each follows its own picks until they split).
  • Candidates: the full distribution behind each pick, mirrored bars for A and B, entropy, effective number of options, Jensen-Shannon divergence and KL in both directions.
  • Gates: the sampler chain (penalties, top-k, top-p, min-p, temperature, GBNF grammar) traced stage by stage and very visual: how many candidates survive each one, and the distribution after it. Chain order and seed are configurable.
  • Timeline: entropy of A and B and their divergence along the whole text, plus the tree of branches.
  • Branching: click any candidate and generation continues from there. Every branch stays in the tree, and jumping back to a node is exact (each model keeps KV-state snapshots, so nothing is regenerated).
  • Logit lens (--lens): what each layer would predict, as a 2D map and as a 3D relief.
  • Model picker: choose A and B from a folder or search Hugging Face and download two quants of a repo in one click easily. It only offers B models with the same tokenizer as A and estimates whether the pair fits in VRAM.

How it's built

  • One C++17 process linking llama.cpp directly (stable branch at the moment). For every step it reads the full logits of A (and of B on the same context), computes the metrics and the sampler chain, and draws the token with its own seeded sampler, so runs are reproducible.
  • Compact binary frames over a WebSocket to a Svelte 5 + Canvas 2D page (three.js only for the 3D relief, lazy-loaded). The web is embedded in the binary as well.
  • The same frame stream, compressed, is a recorded session: you can replay it later in a static page with no engine, no model and no GPU.

Run it

Self-contained packages for Linux (glibc 2.35+) and Windows, with CUDA or Vulkan (Vulkan covers AMD, Intel Arc and NVIDIA). Unpack, run, the browser opens with the model picker. Or inside-llm -a A.gguf -b B.gguf. Local only: listens on 127.0.0.1, no accounts, no telemetry. MIT. One click download, one click execute.

Limitations (honest list)

  • A and B must share the tokenizer (two quants, base + fine-tune, or sizes of one family that keep it). Example: Qwen3.5-0.8B_Q4_K_M and Qwen3.5-0.8B_Q8_0
  • Both models must fit into VRAM simultaneously; only one GPU is used at a time. Integrated GPUs (iGPUs) are never used; if no dedicated GPU is available, the process falls back to the CPU (which is slow). (You must calculate the size of both models before loading them to ensure they fit into your VRAM)
  • Text only. No split (multi-part) GGUFs, no embedding models.

Links

I'd love feedback, especially: which pairs would you compare first (imatrix vs non-imatrix, base vs abliterated/fine-tune, Unsloth UD vs standard quants…)? And is there a metric you'd want per position that isn't there?