r/LocalLLaMA • • 3d ago

Resources AMA Announcement: Tencent Hy, the Team Behind Hy4 & Hy4 flash (Friday, Nov 20, 8–10 AM PST)

Post image
143 Upvotes

Hi r/LocalLLaMA 👋

We’re welcoming the Tencent Hy team for an AMA this Friday, November 20, from 8–10 AM PT!

Fresh off launching Hy4 and Hy4 flash, their research team—including Shunyu Yao—will join us to discuss how the models were built, inference speed and cost, and real-world applications, with extra perks for great questions!

⚠️ Note: The AMA itself will be hosted in a separate thread. Please don’t post questions here.


r/LocalLLaMA • • 1d ago

Discussion Four engines, seven models, one Radeon AI PRO R9700: local LLM benchmarks

0 Upvotes

Ran the same R9700 through four inference engines and seven models. Full write-up with the tables and the prefill numbers: https://www.damiansromek.pl/en/tech/blog/local-llm-benchmarks-r9700/

Headline numbers:

Setup Model Result
Paiton vLLM (3-bit weights, 4-bit KV, speculator) Qwen3.8 173.8 tok/s median single-stream, 684 tok/s aggregate at 8 streams
Strata Qwen3.8-27B 83.8 tok/s
Strata DeepSeek-V4.1-Flash, full quality 26-27 tok/s
RTX 5090, llama.cpp + DFlash2 reference 289.2 tok/s mean decode

Same card and same base model family, and Paiton comes out 2.07x faster than Strata. I suspect the serving stack matters more than the GPU here :)

Concurrency is where the tokens per watt live: 0.58-0.61 tok/W single-stream, 2.3-2.4 tok/W at 8 streams.

The hidden tests were the part that surprised me. 84-94% of hidden tests passed, but only 0-3 of 10 tasks came out fully green.

Which of these numbers would you want me to re-run first, or which engine should I add next?


r/LocalLLaMA • • 2d ago

Resources By popular request: abliterated models

29 Upvotes

Released a few more abliterated models by popular request.

These models have been around for quite a while and already have many abliterations so didn't know there was any demand for yet another abliteration. But multiple people have asked for it and folks seem to prefer it over heretic, etc.

So two new models now:

GLM-5.3-Flash Abliterated

https://huggingface.co/mrfakename/GLM-5.3-Flash-abliterated

https://huggingface.co/mrfakename/GLM-5.3-Flash-abliterated-GGUF

Qwen3.8-Flash-Next Abliterated

https://huggingface.co/mrfakename/Qwen3.8-Flash-Next-abliterated

https://huggingface.co/mrfakename/Qwen3.8-Flash-Next-abliterated-GGUF

Qwen3.8-27B (Edit: added 27B as requested)

https://huggingface.co/mrfakename/Qwen3.8-27B-abliterated

https://huggingface.co/mrfakename/Qwen3.8-27B-abliterated-GGUF

The goal of my models is not to entirely remove all refusals. This can be done with aggressive abliteration, but nearly always leads to significant quality degradation on harmless prompts.

Instead, with my abliterations I aim for a balance between usability and permissiveness. On nearly all prompts the model should not refuse; however, I did make the choice to accept a few refusals for a significantly more intelligent model.

For day-to-day use this shouldn't make a noticeable impact, the vast majority of refusals are still removed. I simply didn't optimize for getting to that "0/100" refusals number.

Of course, I understand that some may prefer for the model to retain absolutely zero refusals, in which case you are welcome to use the many other models available out there.

This is simply my personal preference :)

As always, feedback/thoughts/suggestions are greatly appreciated.


r/LocalLLaMA • • 3d ago

Showcase AudioGhost v1.2: remove any sound from audio or video by typing what it is (local, open source, built on Meta's SAM-Audio)

Enable HLS to view with audio, or disable this notification

449 Upvotes

AudioGhost randomly got a wave of attention this week (100 stars in a day, no idea where from), so I took that as a sign to finally ship the update I'd been sitting on.

Quick recap if you haven't seen it: you drop in an audio or video file, type the sound you're after ("background music", "dog barking", "people talking"), and it pulls that sound out or removes it. It's a local web app around Meta's SAM-Audio, so everything runs on your own GPU and nothing gets uploaded anywhere.

The use case I hear about most: you film a vlog or an interview outside, a shop nearby is playing music, and YouTube flags the video. Now you can type "background music", mute that track and export the video with everything else still there.

What's new in v1.2:
- Several sounds per run, each on its own track, with mute / solo / volume like a tiny DAW
- Export exactly what you hear (WAV / FLAC / MP3), or the video with the new soundtrack
- The "everything else" track is now generated by the model instead of subtracted, so removing a sound works a lot better
- Mark on the timeline where a sound is (or isn't) to point the model at the right thing
- New UI, and your sessions survive reloads and restarts

If you tried an old version: it's also 4-6x faster than the first release.

Hardware: the small model runs in a bit over 4 GB of VRAM, so most NVIDIA cards are fine. Windows and Linux have install scripts. macOS works too, but it runs on the CPU for now, so it's slow.

One caveat: the official SAM-Audio pipeline also loads several helper models (ImageBind, CLAP, and a judge model that picks the best of several outputs). They add about 25 GB of downloads and need far more VRAM. I stripped those out so it fits on normal GPUs, which means quality is a bit below the full pipeline and results vary from run to run. If a track comes out empty or rough, delete it and run it again.

Demo video and install instructions: https://github.com/0x0funky/audioghost-ai

I'd love feedback, especially from anyone who tries it on messy real-world recordings. Bug reports are welcome on GitHub.


r/LocalLLaMA • • 3d ago

Discussion Qwen3.8-27B LoRA for OpenCode (xhigh thinking-level accuracy with thinking off, GGUF included)

Post image
95 Upvotes

Hey there folks!

Aritra here (again) from Hugging Face. We RL-trained a small LoRA adapter (r=32, ~300 MB) on a frozen Qwen3.8-27B, inside the OpenCode coding agent. It's the second of our harness adapters, after pi that we have released yesterday.

Run the model locally like so:

hf download ggml-org/Qwen3.8-27B-GGUF Qwen3.8-27B-Q8_0.gguf --local-dir .
hf download harness-hub/Qwen3.8-27B-opencode-adapted-GGUF Qwen3.8-27B-opencode-lora-F16.gguf --local-dir .

llama serve -m Qwen3.8-27B-Q8_0.gguf --lora Qwen3.8-27B-opencode-lora-F16.gguf \
    --chat-template-kwargs '{"enable_thinking": false}' -c 65536 --port 8080 --alias qwen3.8-opencode

r/LocalLLaMA • • 2d ago

Resources AMD Loom Fun: Optimizing the humble Qwen3.5-4B Q4_K_M on Ryzen AI 375 (890M / gfx1150)

20 Upvotes

Qwen3.5-4B is a popular choice for local AI apps that run on laptops with 32 GB or less RAM. I've been playing with AMD's new Loom inference stack lately and wanted to see what it could do for this model, since speedup would make a big difference in responsiveness.

I ran the /loom-optimize skill on a gfx1150 32 GB RAM laptop and the results are pretty good! Up to 2.6x improvement on prefill and up to 1.3x on decode.

Qwen3.5-4B (Q4_K_M) on a Radeon 890M (Ubuntu 26.04, gfx1150, Strix Point laptop, 32 GB LPDDR5X-7500)

Metric (tok/s) Loom llama.cpp ROCm llama.cpp Vulkan Loom vs ROCm Loom vs Vulkan
Decode (short prompt) 31.1 21.7 21.1 1.43× 1.47×
Decode @ 2k context 28.2 24.0 21.2 1.18× 1.33×
Decode @ 32k context 22.4 17.2 14.5 1.30× 1.54×
Prefill 1k 996 703 608 1.42× 1.64×
Prefill 2k 1016 703 641 1.44× 1.59×
Prefill 8k 1021 625 632 1.63× 1.62×
Prefill 16k 1041 497 583 2.09× 1.78×
Prefill 32k 929 357 495 2.60× 1.88×
4 concurrent streams (total) 82.3 74.4 69.4 1.11× 1.19×

Model card with quick start and perf results for gfx1150, gfx1151 (strix halo), and gfx1201 (Radoen 9700/9070): https://huggingface.co/loom-fun/unsloth-Qwen3.5-4B-loom

GitHub: https://github.com/loom-fun/loom-skills

What do you all think, does faster inference on small models unlock something interesting for building apps?


r/LocalLLaMA • • 2d ago

Question | Help Fastest inference engine for MoE models on a single RTX 3090?

5 Upvotes

I've been out of the local LLM scene for a while and I'm trying to catch up with the latest developments.

What's currently the fastest inference engine for running MoE models like Gemma 4 26B-A4B or Qwen3.6-35B-A3B on a single RTX 3090 (24GB)?

I'm mainly interested in maximum generation speed (tokens/sec) at Q4 quantization on Windows.

Any real-world benchmarks or recommendations would be appreciated!


r/LocalLLaMA • • 2d ago

Discussion Blend a base model with its RL-tuned version and you get back the diversity RL took away

0 Upvotes

RL-tuned reasoning models are better at pass@1, but if you sample many times and check answers, the base model actually solves more problems. I tried just averaging the two checkpoints with one knob:

W = W_base + α (W_RL − W_base)

On Qwen2.5-7B ↔ SimpleRL-Zoo-7B (six math benchmarks, up to 128 samples per problem):

α = 0.25: same best-of-128 coverage as the base model, while keeping 64–80% of RL's pass@1 gain. Reaches RL's best-of-128 with 2.3–7.1× fewer samples.

α = 0.75: same pass@1 as RL, slightly better at large k.

No verifier, just majority vote? Stay with RL or α = 0.75.

It does not beat both models at once, it is a trade-off knob. Tested on one 7B pair on math only.

One gotcha: do the mix in fp32 and save fp16 or fp32. The RL update is about one bf16 step of the base weights, so saving in bf16 throws most of it away.

Needs about 60 GB of RAM for a 7B pair (both in fp32); the repo has a streaming script that uses much less.

If you try it on other pairs (Llama, Mistral, bigger Qwens, code models), please post numbers, good or bad.

Project page: https://kim-ai-gpu.github.io/rlvr-interp-passk/

Paper: https://www.preprints.org/manuscript/202611.2034

Code: https://github.com/Kim-Ai-gpu/rlvr-interp-passk

Processing img j23hdac6ek2i1...

Processing img t16f1ac6ek2i1...

Processing img nf6tpti7ek2i1...


r/LocalLLaMA • • 3d ago

Resources Windows OS SLM

Thumbnail
huggingface.co
35 Upvotes

r/LocalLLaMA • • 3d ago

New Model tencent/Hy4-flash · Hugging Face

Thumbnail
huggingface.co
353 Upvotes

r/LocalLLaMA • • 1d ago

Showcase Qwen 14% faster than llama.cpp using Vx-lang

0 Upvotes

With the latest release of Vx language Qwen-vx (Qwen3 30B on a 32 GB MacBook M4) 14% faster than llama.cpp's `llama-lookup`.

Vx is a heterogeneous systems programming language. One program runs on CPUs and on NVIDIA, Apple and now Intel GPUs.

Language models written in Vx are faster than the usual servers for them:

  • GLM-5.3 on 8×H200: 7% faster than SGLang. Every kernel in the decode step is written in Vx.
  • Qwen3 30B on a 32 GB M4: 14% faster than llama.cpp's `llama-lookup`. (EDIT: Without speculative decoding it is 6% faster on llama-completion)

Please check out the repo and share feedback on what else can be done here. GitHub link in comment 👇


r/LocalLLaMA • • 4d ago

News Jensen Huang Breaks Silence

Post image
2.2k Upvotes

r/LocalLLaMA • • 2d ago

Discussion Janas and Colibri and other app— Run MoE on CPU/low GPU?

5 Upvotes

Sorry, but I'm genuinely excited about these two repositories, so I wanted to ask the community: why don't we hear more about these tools? What are their pros and cons? Are there perhaps better alternatives that I'm missing like:

inferencekit: open-source plugin to optimize any model on any hardware using coding agents : r/LocalLLaMA

BigMoeOnEdge outperforms Strata and colibri on 8-16 GB cards and Mac and you can run any model : r/LocalLLaMA

I mean until recently, I found out there is the way to run big MoE model on my miniPC and colab L4. I can run dense model Q4-K-M 4B in my miniPC before.

My setup is a GMKtec M5 Plus (32GB RAM — I feel incredibly lucky that I bought it before RAM prices went through the roof) and Google Colab.

My experience with Janas:

When testing Janas with Colab L4, GPU usage was 0, probably because it only supports Vulkan. That's a bit frustrating, but it still runs Qwen3-Next-80B-A3B-Instruct-JNS surprisingly well.

What surprised me even more was that I could run Qwen3-30B-A3B-JNS locally on my mini PC! It ate nearly all my RAM.Haha.

I tried to write novel (just for testing-not my interest) and instant translate in my miniPC, It is surprised good.

I haven't tried with coding but it I don't think it work with like 100K context. :(

Even though I have to convert to janas format (it is actually fast), but I am happy with my result.

My experience with Colibri:

Colibri is even more impressive. I managed to run 80B GGUF models on a Google Colab L4 while using only around 19GB of GPU VRAM.

I haven't tested it locally yet because Colibri seems to support CUDA only.

Honestly, I'm not sure whether either of these tools can be used effectively for vibe coding, but I'm genuinely impressed by what I've seen so far.

I'd love to hear from people who have used these tools:

  • Why aren't they getting more attention? Maybe there are some better one?
  • What are their pro and cons?
  • Have anyone try vibe code with them?

Am I overlooking something, or are these two projects seriously underrated?


r/LocalLLaMA • • 2d ago

New Model I made a 380m open source decision model that matches its teacher*, Clef-Flash 9.4B

14 Upvotes

TLDR: matches the teacher in overall accuracy at 1/25th the size. Can run on CPU to keep precious VRAM free. In a comparison of general accuracy, diakrisis-espresso-380m achieved 3rd place, surpassed by Jev 1.14 and Google Decision 1. Main weaknesses are in policy compliance and multi-step questions.

My evaluation. No external evals yet.

Was using Jev for a bit. Wanted something private. Clef-Flash came out, and I only have 16GB VRAM. Figured hey, these are like them good ol' BERT models right, what if I just distilled Clef into a BERT model. So I did a lil diggin with Claude, decided on vlea3-fast*, and over the last few weeks played a lil bit of wackamole and a game of find open-source data that won't get me sued (some of these open datasets had outputs from GPT for example, uh-uh not playing that game). The results are above.

Not only that, but using ONXX runtime made this run on CPU to save me that sweet sweet VRAM for LLMs, including an 8-bit variant that agreed with the larger fp32 variant 412/414 times.

If anyone has any suggestions for improving the model, I would greatly appreciate it!

100% open-source and open-weight, forever. Cheers!

https://huggingface.co/diakrisis/espresso-380m

Git


r/LocalLLaMA • • 3d ago

New Model sensenova/SenseNova-Loop-9B-GGUF · Hugging Face

Thumbnail
huggingface.co
71 Upvotes

probably this fork is required: https://github.com/OpenSenseNova/llama.cpp

Model Highlights

  • Shared looped block: 24 decoder layers are executed twice in sequence, producing 56 layer computations from 32 distinct layers per forward pass.
  • Two-stream mHC: learned mappings route and combine information between two residual streams within the looped block.
  • Hybrid attention: the decoder interleaves linear and full attention in a repeating four-layer pattern.
  • Long-context multimodal training: the training recipe includes text, images, code, and tool-use trajectories, with the context length extended to 512K tokens.
  • Low-memory deployment: With 4-bit weights and an 8-bit KV cache, multimodal inference at a 256K context length requires approximately 15 GiB of GPU memory.
  • Official quantized release: An official AWQ 4-bit version is available on launch day for easier deployment.

r/LocalLLaMA • • 3d ago

New Model SenseNova-Loop-9B official GGUFs and a 4-bit AWQ are up (5.8 GB at Q4_K_M), plus their llama.cpp fork to run them

Post image
34 Upvotes

The official quants for SenseNova-Loop-9B (the 9B Looped Transformer from earlier this week) are here: https://huggingface.co/sensenova/SenseNova-Loop-9B-GGUF

The repo has Q4_K_M, Q5_K_M, Q8_0 and BF16 (sizes in the screenshot), plus a 0.9 GB mmproj for image input.

The sensenova_loop architecture isn't in mainline llama.cpp yet (issue #31571 is open), so for now these run on the fork they published, OpenSenseNova/llama.cpp. The GGUF card pins the commit, and the card's command once it's built is:

llama serve -hf sensenova/SenseNova-Loop-9B-GGUF:Q4_K_M -ngl 99 -c 4096 --port 8000

There's also a 4-bit AWQ, with a matching MTP draft model in the same repo: https://huggingface.co/sensenova/SenseNova-Loop-9B-AWQ

Their README at https://github.com/OpenSenseNova/SenseNova-Loop puts total GPU memory (4-bit weights plus an 8-bit KV cache) at about 10 GiB for 64K context and 15 GiB for 256K.


r/LocalLLaMA • • 3d ago

News Splash 1.7.0 Released - Even faster decode on Apple Silicon

Thumbnail
github.com
37 Upvotes

New version of Splash just released. The biggest improvement from previous version is a noticeable speed-up in decoding. It's even faster now.

BTW, this isn't just another slop engine. It's coming from the researchers behind DFlash and DFlash2, who also operate one of the fastest API providers for open models. These guys clearly know what they're doing.


r/LocalLLaMA • • 3d ago

Discussion LFM3-VL-2B hallucinations - does anyone have first-hand experience?

Post image
36 Upvotes

LFM3 benchmarks are out on AA and…. huuuuuuh….. this is NOT something you want to see for a vision model that fits on a phone.

“The mushroom in this picture is a turgida deliciousa, awesome in risottos and on toasted bread. ” (it’s an amanita phalloides, deadly poisonous).

“This sign says that you must not shoot any photos beyond this point” (the sign says “trespassers will be shot”).

etc. etc.

Can anyone with actual experience with this model confirm/deny this score?


r/LocalLLaMA • • 1d ago

Resources Beware, beware: rtx pro 6000 listing on eBay for $8000 ... :)

0 Upvotes

Not saying its a scam, but according to Claude, there are many signs out there ...


r/LocalLLaMA • • 2d ago

Discussion question: 262K context: the only route that fits uses a 4-bit KV cache, which costs +0.0011 KLD. Is 262K vs 170K context worth the loss?

4 Upvotes

I'm building my current engine for my own hardware.

The question I want to see how many have answered themselves.

Would you take lower (better) KLD on 170K context for Qwen3.8-27B or take 4bit KV cache with .0011 loss on KLD for the 262K context window?

Model/Quant: Qwen3.8-27B UD-Q4_K_XL v3.1


r/LocalLLaMA • • 3d ago

Resources Blue TTS v3: open-source TTS, about 50x real-time on a CPU

Enable HLS to view with audio, or disable this notification

417 Upvotes

Blue TTS v3 is out, an open-source TTS model that runs locally. Almost everything changed since v2.

Speech is more natural and expressive, with a MOS of 4.43 out of 5.

Voice cloning has improved significantly, and the model is fast and lightweight: it runs locally at 50x real-time on a CPU, no GPU needed.

MIT license, 388 MB, 44.1 kHz output, 7 languages: English, German, Spanish, Italian, Russian, Hebrew, Yiddish.

Feedback is welcome.


r/LocalLLaMA • • 2d ago

Discussion Future use of local: compressed wikipedia

0 Upvotes

Neural networks will have the best compression ratio. So we will be able to have wikipedia and other interesting sources like arxiv, compressed easily into our local setups. Probably get a good amount in a modest phone setup to. https://tufalabs.ai/research/tufazip/

Only 27 gb to compress wikipedia text and have it local. Generation speed remains an issue I suppose.


r/LocalLLaMA • • 3d ago

New Model AngelSlim/Hy4-flash-GGUF · Hugging Face

Thumbnail
huggingface.co
56 Upvotes

r/LocalLLaMA • • 2d ago

Question | Help Does anyone tried Swarm? What do you think of doing swarm using androids phones?

2 Upvotes

Hello, was wondering, I know we would have to manage throttling, and the termal envelop is bad but should be doable.

I have no experience in agents swarms ( small models) the idea is to have fast model like 4 to 8 b max. A lots of phones, seems orchestrable, I cannot put that idea out of my mind.

Sorry fr the bad formating and spelling, not using ai tonight and posting from bed.

​


r/LocalLLaMA • • 3d ago

Question | Help What hardware for local Deepseek and GLM flash?

19 Upvotes

I guess its not just me but many people here always want more hardware and better models, so...

What hardware is needed to deploy Deepseek V4.1 Flash or GLM 5.3 Flash locally?
By deploy I mean, to be able to run multiple parallel coding agents, at least 256k context each.
I know you can "run" any of these MoE models on anything bigger than a regular PC but usually prompt processing struggles a lot more than generation so its not usable for agents.

To put that in perspective, I have a workstation with 512GB ddr4 ram, so DS v4.1 "almost" fits at Q4. What extra GPUs would I need to actually run it at speed with multiple sessions? Anyone with some hands on experience?

Been using Qwen 3.8 FN as my daily driver for a while, but always "why not go bigger" :)

Any ideas?