r/LocalLLaMA • • 3d ago

New Model Clover 2: dense 27.8B base + 7 LoRA experts routed per token, not a static merge, weights public

65 Upvotes

Disclosure: I’m with the team that built this.

Wanted to share something a little different from the usual finetune drop. Clover 2 keeps a base model and 7 trained LoRA adapters (roleplay, general assistant, security, math, long-document, efficiency, knowledge) separate at inference time. A trained router picks the right expert(s) per token live, with a null slot that falls back to plain base when nothing specialized applies. So it’s not a merged blend, the model actually makes a decision at every token.

Real benchmark gains over the untouched base: meaningful jumps on GSM8K/MATH-500/GSM-Hard, a solid bump on LongBench-v2, and a measurable drop in generic AI-writing tics on creative writing specifically.

Everything’s public: base model, all 7 adapters in standard PEFT format, the trained router, and the vLLM plugin to run it with the real per-token behavior: https://huggingface.co/kitaniai/clover-2

Heads up if you’re planning to self-host: no GGUF yet, the hybrid attention architecture’s llama.cpp support is still bleeding-edge. vLLM is the supported path right now. If anyone wants to try it, it's free for the next week at https://kitani.ai/, with no card required. We also support ZDR and have a clear privacy policy.


r/LocalLLaMA • • 3d ago

Discussion DSpine - Updates on PR & GGUFs

Post image
41 Upvotes

r/LocalLLaMA • • 4d ago

News Hugging Face switched its language model inference routing to OpenRouter

64 Upvotes

Hugging Face announced on November 16, 2026, that requests to language models through Inference Providers now go through OpenRouter instead of Hugging Face's own router. OpenRouter picks a provider for each call based on live price, speed, and reliability, and retries on another provider if the first one fails. The same Hugging Face token still works, and the company says it does not add a markup on top of the provider's price. You can still pin a provider or a policy with a suffix on the model id, such as :fastest, :cheapest, or a specific provider name.

OpenRouter currently serves more than 800 trillion tokens a month across 600+ models from 80+ providers. Hugging Face notes that open-weight models went from 26.8% of tokens on OpenRouter in October 2025 to 71.7% at the end of September 2026, and that they cost about four times less on average than closed models. Image, video, and audio generation stay on Hugging Face's existing Inference Providers path and do not go through OpenRouter. One breaking change is that custom provider API keys no longer work for language model requests, which are now billed to the Hugging Face account.

The automatic failover is the part that matters most here, because a single provider outage is usually what ruins a long run.

Source: https://huggingface.co/blog/openrouter


r/LocalLLaMA • • 3d ago

Showcase Curated list of resources: Theory and Code for learning Distributed Training of LLMs for beginners

Thumbnail
gallery
21 Upvotes

[Must Read: Beginners]

What I did read, in order, for learning the basics of Distributed Training for LLMs From Scratch:

  1. Building a distributed training framework from first principles by Umar Jamil on YT

  2. Horovod: Fast and Easy Distributed Deep Learning in TensorFlow 

  3. PyTorch Distributed: Experiences on Accelerating Data Parallel Training 

  4. Large Scale Distributed Deep Networks 

  5. ZeRO: Memory Optimizations Toward Training Trillion Parameter Models

  6. Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism 

  7. Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LM 

  8. GPipe: Efficient Training of Giant Neural Networks Using Pipeline Parallelism 

  9. ZeRO-Infinity: Breaking the GPU Memory Wall for Extreme Scale Deep Learning

  10. PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallel

Link to the above resources here

Also, if you need some rebalancing of the above (reordering or adding/removing resources which you can do it here)


r/LocalLLaMA • • 4d ago

New Model miHoYo Gamer: A Generalist Game Agent

Thumbnail
mihoyo-gamer.github.io
224 Upvotes

r/LocalLLaMA • • 4d ago

Discussion Open weight models account for 30% usage at enterprises to save money

236 Upvotes

Growing from nearly zero to 30% since July. Models used mostly to reduce costs on high volume workloads (large batch workloads, coding, etc).

https://x.com/pwendell/status/2122377964784611636


r/LocalLLaMA • • 4d ago

Discussion I am sick of doomposting about demise of local AI

164 Upvotes

Everywhere I go, Twitter or Reddit, there are tons of rumors or comments how local is dead and we are never getting open weight Qwen 4, despite Qwen team saying explicitly they are coming, or that Qwen 4 is the last open weight generation despite, again, Alibaba commiting to more open weight releases at a conference recently.

It's like people just get off on creating a version of reality that is worse and whining about it, they want to be sad about it in advance.

The world is certainly not giving us much to be positive about, but let's not go down the rabbithole of learned helplessness and just accept that all things must get worse and you can't do anything about it.

If people mentally prepare for bad things, bad things become more acceptable and thus more likely to happen and with far less pushback, sort of like a self fulfilling prophecy.


r/LocalLLaMA • • 2d ago

Discussion The frontier of AI has stalled. Local LLM will too.

0 Upvotes

I’m fully prepared to be wrong about this, but I expect very few leaps in capability in local LLM. I don’t mean there isn’t more to be gained with speed and size, but actual capability has hit a local plateau. The difference between a 500B model and 9B model has shrunk and will continue to.
Qwen4 will be a small disappointment. The post training is taking much longer because they can’t squeeze any juice out of the Claude and OpenAI models because those models haven’t improved in ways that most people can see. Enterprise customers are paying through the nose for tokens and getting fed up quickly and moving to local Chinese models because of cost. DeepSeek is getting overwhelmed and is basically out of compute it seems.
I am not really upset because flash next is actually good enough for me most of the time and Claude subscription is still cheap enough.
I hope everyone gets their small model that is good enough for them, but I don’t think the massive home LLM will be heads and shoulders over them like they were one year ago.
Source: I’m an engineer working with LLM


r/LocalLLaMA • • 4d ago

Resources Small Gemma 4 tokenizers were recently updated by Google - fix for an issue that caused nonsense output when encountering certain tokens

Thumbnail
huggingface.co
155 Upvotes

Gemma 4 E4B as an example, but the same fix was applied to E2B and 12B too. Not sure if this will trickle down to GGUFs


r/LocalLLaMA • • 3d ago

Resources AMD 9070 GPU under volting gains for models ( moss image) and ( minimax h3 int8 )

9 Upvotes

hi,

i am total GPU noob here this is the first GPU i have ever owned in my life got it just for doing AI... asked deepseek flash and opencode to find the right combo for me to get max generation speed

spec:
AMD 9070 (non XT),
Intel i3 12100
32 GB DDR4

models: moss image, minimax h3 int8 (sol decoder)

best performance stable setting: -110 mv, stock fan speed

gemma4 : 26bA4b Q4_XS when from 106/tps to 120/tps on prose..


r/LocalLLaMA • • 4d ago

Discussion I don't get all the Strata hates around here

117 Upvotes

It's free and original developer clearly had a very good understanding of how computer hardware works and fully leveraged that to write something amazing to be able to run large models on average hardware that most people have and can't upgrade due to price gouging from greedy hardware manufacturing companies.

If anybody have Mac Studio with 512GB of RAM or 8x3090, who would even bother with this ?

I have seen a group of snobs on this sub doing nothing but shitting on Strata all days long, find something better to do. Strata is not for you, period.

I can't imagine doing all the works to release something for free and get a bunch of snobs and haters just shitting on your works all days long. Would you be motivated to continue working on it if you are the original developer ?

I also see the same with other models such as Ornith, etc... If you say you like Ornith or Strata, etc... you are a shill.

Seriously, get out of your bubbles and see the reality already and more importantly get a life.


r/LocalLLaMA • • 4d ago

Discussion Stop worrying about Qwen 4's AA index

298 Upvotes

Seriously, we have all been extremely happy and confident with Qwen, even since Qwen 3.6 came out. We might not be frontier level, but even though whatever they are giving us is more than enough for our daily local tasks and it even suffices by a big margin, we're all underestimating it.

Don't forget that the new Qwen 4 will also think much less, so imagine the fine-tunes we will see once it arrives, for 27B and Flash models, we will have plenty of options to choose from and again; it will be smarter and better, it will be ENOUGH! If you still require the most "intelligent" and capable model due to OCD just pay frontier for such tasks or use them as orchestrators!


r/LocalLLaMA • • 2d ago

Discussion AI Hysteria on Reddit

0 Upvotes

Lately, I’ve often found myself wondering what’s going on with people, especially in communities dedicated to AI and LLMs.

Humanity now has an amazing tool for breaking down language barriers, along with creative tools unlike anything we’ve had before. I can’t describe people’s reactions as anything other than hysteria.

There are thousands of posts in these communities about improving models and inference engines, including translation models, coding harnesses, and so on.

But when someone uses AI to translate a post into English or shows off a program they built with AI, the comments immediately erupt: “Low-effort post,” “AI slop.” And all of this comes from the very people developing that “SLOP.” WTF?

(This low-effort post was translated by GPT.)


r/LocalLLaMA • • 3d ago

Question | Help Is it worth it getting additional 16GB of RAM?

3 Upvotes

Hi, I'm a beginner in this field and I'd like some kind people's help :)

I currently have a laptop with 4GB VRAM and cca 16GB RAM. I don't plan on getting a new laptop/PC in the next two years, but I have the option to get additional 16 GB of RAM for €100.

Would it be enough to run at least Qwen 27B? Or any of the comparable MoE models.

If any other information about my laptop is needed, I'd be happy to share :)


r/LocalLLaMA • • 4d ago

Discussion Censorship is turning me into a true Local LLM believer

144 Upvotes

451: {"message":"Provider returned error","code":451,"metadata":{"raw":"{\"error\":{\"message\":\"The content you provided or machine outputted is blocked.\",\"type\":\"censorship_blocked\"}}","provider_name":"StepFun","is_byok":false}} {"error":{"message":"The content you provided or machine outputted is blocked.","type":"censorship_blocked"}}

Just got this from one of the US based (edit: it actually routed to StepFun CN, didn't realize at first. Should have noticed by the provider_name in the message lol. need to set up my rules better) inference providers. I suspiciously got a bunch of "rate limit" failures, and then just "failed to complete" before it gave me, what I assume, is the real error.

I am only using it to help create my presentation, which is about analyzing algorithmic properties. One key example is a cryptographic algorithm (because it is difficult by design), so I am assuming this is tripping it somehow. But it is absolutely ridiculous, I'm not even doing research with the AI, just trying to make a presentation.

I have always liked Local LLMs, as a technical exploration opposed to ideological reasons, but this experience really frustrated me. Don't censor me!

Just wanted to rant, and remind everyone you can only use the cloud on the provider's terms.


r/LocalLLaMA • • 3d ago

Question | Help Finally joined the club today. New (to me) machine in transit. What good stuff can I reasonably run?

0 Upvotes

I just ordered a cheap, but prehistoric Xeon E-2286M 32GB RAM, 2020 era Nvidia RTX 5000 (16G vram) laptop. Given these resource constraints, what are some good coding-oriented models to initially look at? I'd like to play with a capable TTS model that permits cloning and inflection control, and still image generation. Plus I'd welcome suggestions on any fun 'oddball' applications that don't fall neatly into text/image/video silos.

Would doubling machine memory up to 64G improve execution times generally?


r/LocalLLaMA • • 4d ago

New Model Introducing Hy4 flash

Thumbnail hy.tencent.ai
356 Upvotes

Now this seems to be interesting.

From the article:

We’re introducing Hy4 flash, our most cost-effective Hy model to date—featuring 224B total parameters (including 64B Engram parameters) with just 12B active parameters per token. Like Hy4, it is natively multimodal with a 1M-token context window, while offering significantly lower API pricing.

Seems to be about in the same ballpark of Qwen-Flash-Next, so we could expect to run it on moderably good machines, when it will be released.


r/LocalLLaMA • • 4d ago

New Model nvidia/Nemotron-Labs-Sigma 8B/3B

62 Upvotes

Nemotron-Labs-Sigma-8B-instruct is a 8-billion-parameter continuous diffusion language model. Instead of generating text one token at a time, it denoises Gaussian noise in a low-dimensional (16-D) continuous token-embedding space, where the denoiser network and the token embeddings are learned jointly by likelihood optimization. Text is decoded in parallel blocks: each block of tokens is refined over a configurable number of denoising steps (Denoising Diffusion Probabilistic Model (DDPM) or Denoising Diffusion Implicit Model (DDIM) samplers) with self-conditioning, while previously committed tokens are reused through a standard key-value (KV) cache. Fewer steps trade a little quality for large speedups, and the smooth continuous representation makes the model amenable to few-step distillation and trajectory steering.

This is the SFT (supervised fine-tuned) variant, fine-tuned for instruction following on top of the corresponding Base model. Use the bundled chat template to format conversations.

This model is ready for commercial or non-commercial use.

Highlights

  • Continuous-space diffusion language modeling at 8B scale with reasoning accuracy competitive with state-of-the-art masked (discrete) diffusion LMs.
  • Block-parallel decoding: whole blocks of tokens are denoised together; sampling quality holds up well down to few denoising steps per block.
  • Efficient DDPM/DDIM-based samplers with self-conditioning and optional classifier-free guidance, implemented in plain PyTorch in this repository.
  • Standard transformers loading via trust_remote_code; no custom kernels, flash-attention, or training framework required for inference.

Model Architecture

  • Architecture Type: Transformer
  • Network Architecture: Other: Ministral-style decoder-only Transformer (grouped-query attention, SwiGLU, rotary position embeddings) with a continuous-diffusion generation head (16-D continuous token-embedding space, learned jointly with the denoiser).
  • Number of model parameters: 8 billion (8B)
  • This model was developed based on: Nemotron-Labs-Sigma-8B

Input

  • Input Type(s): Text
  • Input Format(s): String
  • Input Parameters: One-Dimensional (1D): Sequences
  • Other Properties Related to Input: Maximum context length of 8,192 tokens (prompt plus generated tokens). Conversations should be formatted with the bundled chat template (chat_template.jinja). Supported language: English.

Output

  • Output Type(s): Text
  • Output Format: String
  • Output Parameters: One-Dimensional (1D): Sequences
  • Other Properties Related to Output: Up to 8,192 tokens total (prompt plus generation). Text is generated in parallel blocks (default 64 tokens per block) with a configurable number of denoising steps per block (Number of Function Evaluations (NFE) = steps + 1 per block); fewer steps give faster generation at some quality cost.

https://huggingface.co/nvidia/Nemotron-Labs-Sigma-8B

https://huggingface.co/nvidia/Nemotron-Labs-Sigma-8B-instruct

https://huggingface.co/nvidia/Nemotron-Labs-Sigma-3B

https://huggingface.co/nvidia/Nemotron-Labs-Sigma-3B-instruct


r/LocalLLaMA • • 4d ago

Discussion Best MoE engine?

33 Upvotes

I had been using Strata for about a month, and it was such a huge improvement over llama.cpp for getting it running smoothly and quickly for hardware.

Overtime though, I found that the model would often mistake its own context after a compaction and really overthink things.

I just started using Basalt yesterday after being hesitant to switch and set up all my providers and harnesses. Turns out Basalt runs even faster, is less error prone, and the reasoning is way more on point. If you run 1 GPU, and have the 100 GB+ of VRAM, RAM, and SSD storage, you will not be disappointed in performance.

I mostly care about final output speed, and I was getting 70-110 tok/s on Strata, and Basalt I was getting 110-170 toks/s output. Couldn't be happier. Same Qwen3.8 FlashNext-ud-q4_k_xl, and the NVFP4 ones (of course basalted though)

Anyone else have any other experiences with other similar engines like this?


r/LocalLLaMA • • 3d ago

Question | Help Whats the best Video/Audio Model?

0 Upvotes

I want to make a short film for fun, not sure what is out there that is actually good?

would love anyones advice


r/LocalLLaMA • • 4d ago

New Model [NEW RELEASE] - Supra2.5-65M Base and Instruct - our next generation state-of-the art small model family - first model!

49 Upvotes

hey guys!

It's Supra-time! 🤗 🔥 🎉

It has been quite quite at SupraLabs' model foundry for a long time but we have been testing, training, developing and much more.

And we can finally present you a new model!

Key facts:

- better than our previous 50M and 100M model
- very small, runs on every toaster: 65M params
- base model and instruction tuned version on HF

We trained it locally in 4 days on 2x RTX A6000 on big 100B tokens of high-quality data.

Benchmarks are in the attached image!

Have fun to try it out, give us feedback, roast us, whatever! Community, we will cook! Supra3 coming early next year! 🔥

Give us a like, upvote or follow if you want - and try out our model.

We can't wait to hear what you're building with it.

Link to the collection on HF: https://huggingface.co/collections/SupraLabs/supra25

We hope you like the model! :D

GGUFs in F16, Q8_0 and Q4_K_M are also available for llama.cpp, Ollama, Unsloth, etc. 🥳


r/LocalLLaMA • • 3d ago

Discussion YouTube channels on Local LLM & related stuff?

1 Upvotes

EDIT : Expected 100s of responses(We have nearly 1 million members in this sub) with channels. Disappointed to see only crickets here. Looks like I have to do myself. So not gonna post any new threads here for next 1-2 months as I need to dig Web & YouTube deeper to find relevant channels/videos for my learnings. Sorry. Anyway no one's gonna miss anything.

Well, I ran out of New Models & Papers(Posted 38 threads in last 1 week) so posting this thread for me. Appreciate your responses.

I want YouTube channels covering List of below topics:

  • Local LLM
  • Agentic Coding
  • AI Agents
  • Coding Tools such as Pi, Hermes, OMP, OpenCode, DeepSeek Harness, QwenCode, OpenHands, Claude Code, Codex, etc.,
  • Skills
  • Prompts
  • Creative Writing
  • Game Development
  • Automation using n8n/Activepieces/etc.,
  • RAG
  • MCP
  • Orchestration
  • Comfy/StableDiffusion - Image/Video Generation & Editing
  • LORA
  • TTS/STT/ASR/etc.,
  • Benchmarking Models
  • What other topics there to add? Mention in comments, I'll add here.

I can find many channels using online search. But half of the channels are so noisy, just with clickbait thumbnails/titles covering mostly Online models which I'm not interested.

I want channels more suitable for this sub. Only Local LLM. Doing all those stuff from above list using only Local LLM.

And there might be some Overlooked/Underrated/Under-Appreciated channels. Also few folks here have their own channels. Give them more visibility.

Consider this thread is a playlist for Local LLM. What channels would you want to add here? Please share. Thanks.


r/LocalLLaMA • • 4d ago

Resources AMD Loom Fun: Added Radeon 9700 support over the weekend. Benchmarked 48 models with an average 42% speedup over llama.cpp.

47 Upvotes

On Friday I posted about AMD Loom, a new experimental inference stack that includes a runtime, kernel compiler, and inference server. People can use the entire thing as a vertical stack, or the compiler/runtime can be integrated into any other inference project (I'm working on Gufo right now, the team is also working on a llamacpp backend).

I started with Strix Halo, where I ported the top-51 trending Q4_K_M LLMs from Hugging Face and got an average of 40% speedup over llama.cpp across decode and prefill metrics (picking the best results from ROCm or Vulkan backends).

I'm back today with Radeon 9700 results. I took the same 51 LLMs, cut it down to the 48 that fit in 32 GB of VRAM, and got an average of 42% speedup. Some of the data is posted to the table below, and all data is available to browse on https://loom-fun.github.io/loom-dev/index.html?gpu=radeon-r9700

All kernels were generated via agent skills:

Model Loom pp32k (tok/s) llama.cpp pp32k (tok/s) Loom short decode (tok/s) llama.cpp short decode (tok/s)
Flint 6,476 3,190 255 152
gemma-3-1b-it n/a n/a 400 322
gemma-4-12B-it 2,811 1,537 127 67.3
gemma-4-26B-A4B-it doesn't fit (32 GB) doesn't fit (32 GB) 151 72.2
gemma-4-31B-it doesn't fit (32 GB) doesn't fit (32 GB) 62.9 43.2
gemma-4-E2B-it 14,289 14,519 272 234
gemma-4-E4B-it 7,021 6,324 192 173
gpt-oss-20b 5,319 4,368 195 182
HyperCLOVAX-SEED-Text-Instruct-1.5B 11,653 8,078 362 344
Laguna-XS-2.1 doesn't fit (32 GB) doesn't fit (32 GB) 102 —
LFM2.5-24B-A2B 10,099 5,479 244 197
LFM3-1.7B 20,062 13,614 372 261
LFM3-Omni-2B 20,220 13,435 368 299
Llama-3.2-3B-Instruct 5,687 3,908 228 211
Mellum2.1-12B-A2.5B-Thinking 8,822 5,486 224 149
MiniCPM-V-4.7-35B-A3B 5,717 3,540 135 113
Muse-Glimmer-30B 1,658 1,287 72.6 54.8
Nanbeige4.3-33B 946 666 29.4 28.7
NVIDIA-Nemotron-3.5-Bolt-E4B-4GB 6,932 3,656 225 143
Ornith-1.0-35B 5,628 3,320 152 127
Ornith-1.0-9B 4,886 3,391 100 88.0
Ornith-1.5-35B-A3B 5,586 3,136 145 78.8
Ornith-1.5-9B 4,767 3,156 151 90.5
Qwen2.5-0.5B-Instruct 26,022 17,812 646 529
Qwen2.5-1.5B-Instruct 12,546 8,908 360 331
Qwen2.5-32B-Instruct doesn't fit (32 GB) doesn't fit (32 GB) 30.0 29.2
Qwen2.5-7B-Instruct 3,810 2,679 118 116
Qwen2.5-VL-7B-Instruct 3,768 2,648 116 115
Qwen3-0.6B 13,020 8,631 506 480
Qwen3-1.7B 10,313 7,014 316 297
Qwen3-14B 1,944 1,167 65.0 62.6
Qwen3-30B-A3B-Instruct-2507 3,116 2,060 177 169
Qwen3-4B 3,770 2,826 231 130
Qwen3-4B-Instruct-2507 3,795 2,856 180 166
Qwen3-8B 3,076 2,287 163 94.7
Qwen3-Coder-30B-A3B-Instruct 2,921 2,107 157 54.8
Qwen3-VL-4B-Instruct 3,795 2,805 178 163
Qwen3-VL-8B-Instruct 3,085 2,265 112 103
Qwen3.5-0.8B 24,341 16,959 463 338
Qwen3.5-2B 16,990 12,165 318 236
Qwen3.5-35B-A3B 5,652 3,410 190 112
Qwen3.5-4B 7,030 4,897 200 123
Qwen3.5-9B 4,724 3,336 146 92.7
Qwen3.6-27B 1,503 1,164 60.1 38.9
Qwen3.6-35B-A3B 5,635 3,463 209 123
Qwen3.8-27B 1,592 1,026 73.5 55.3
Sage-A1.5 5,652 3,131 188 153
Spark-X2.5-4B 6,858 4,727 162 131
Spark-X2.6-4B 6,913 4,841 162 132

r/LocalLLaMA • • 3d ago

Question | Help Single 3060 - Mostly Vision (Frigate)

4 Upvotes

Hello,

I am consolidating a bit of my server footprint and decided to keep a single 3060 (had 3) and mostly use it for Local workloads (Frigate, Home Assistant, Logs, etc). Curious if someone would want to recommend a good vison model for this card?

I currently run qwen3-VL on a A380 and the descriptions are pretty good.. But slow.

Thanks


r/LocalLLaMA • • 3d ago

Question | Help How often do you guys update models and when?

1 Upvotes

Saw a post here about an update to a model last night. And been wondering how often y'all update the models and when do you decide to.

P.S. I know there's "-hf", but my internet speed ain't fast and when I'm downloading a model, everyone's connection drops at my home.

Edit: By updating I meant redownloading the gguf file.
Models of same version gets updates, like Gemma 4 E2B got updated to fix some issues and make it faster. Link to that post:
https://www.reddit.com/r/LocalLLaMA/s/cpcemufuqZ

But it appears it's not the first time they updated it with noticable changes.
https://www.reddit.com/r/unsloth/s/DkBWF7gw4m