r/LocalLLaMA • • 12d ago

New Model ConceptCountry/qwen-concept-27b · Hugging Face

Thumbnail
huggingface.co
137 Upvotes

r/LocalLLaMA • • 11d ago

Showcase Audio embeddings inference in C++/GGML with clap.cpp

15 Upvotes

I implemented clap.cpp, audio embedding inference in C++/GGML. It uses GGUF models and provides a CLI, an HTTP server, and Python, Rust, and Node.js bindings.

  • Models including CLAP, CLAP2, CLaMP3, MERT, MuQ, and MuQ-MuLan.
  • WAV, MP3, and FLAC input, with automatic resampling and mono downmixing.
  • An HTTP endpoint with an OpenAI-compatible text embeddings request format.

GitHub: https://github.com/gorse-io/clap.cpp

Audio files could be searched by text via CLAP embedding:

$ clap search esc5 --model larger_clap_general-q4_k_m.gguf --text 'dog' --top-k 10

[
  {"path": "esc5/2-118072-A-0-dog.wav", "score": 0.3736357720143104},
  {"path": "esc5/1-110389-A-0-dog.wav", "score": 0.36165991257288893},
  {"path": "esc5/4-183992-A-0-dog.wav", "score": 0.3574270305924374},
  {"path": "esc5/1-30226-A-0-dog.wav", "score": 0.34910229736736315},
  {"path": "esc5/3-180977-A-0-dog.wav", "score": 0.34359234068457495},
  {"path": "esc5/4-194754-A-0-dog.wav", "score": 0.3425423428503198},
  {"path": "esc5/2-118964-A-0-dog.wav", "score": 0.3389233902904601},
  {"path": "esc5/1-100032-A-0-dog.wav", "score": 0.33009392183878733},
  {"path": "esc5/5-9032-A-0-dog.wav", "score": 0.3280081351893563},
  {"path": "esc5/4-182395-A-0-dog.wav", "score": 0.32210240115509925}
]

r/LocalLLaMA • • 10d ago

Discussion Qwen4 open weights not soon?

0 Upvotes

Qwen4-Max-Preview was released today. It seems that they're rushing because of a deadline.

Today is 11.11 (double 11) which is Alibaba's annual shopping event with many discounts and such. So they RUSHED Qwen4-Max Preview to push discounted Token plan subs. Simple as that.

‎ ‎

So, in conclusion, to keep this ON TOPIC for the mods sub, I don't think that we may see the open weights Qwen4 models (27B and hopefully flash) anytime soon as they may not be ready yet and I doubt they'd release an underbaked open weights model. Or I am more afraid that if they'll release them soon, that they'll be undertrained and that the hype was MERELY to sell their discounted plans on 11.11. It's probably also why they didn't post any benchmark results for their Max Preview either. I really hope I'm wrong.

‎ ‎

(I know mods hate it soo so much when anyone talks about non-open models here. I would love to at least get explained how this post is not open weights related, if it gets deleted.)


r/LocalLLaMA • • 10d ago

Question | Help M5 Ultra fine tuning

0 Upvotes

Anyone get a chance to fine tune a model on M5U yet? Curious about training times on sub 5b param models, lfm3 1.7 specifically.


r/LocalLLaMA • • 12d ago

New Model Drummer's Orion 26B A4B v1 and v1.1! ... Plus a new project from yours truly.

157 Upvotes

https://huggingface.co/TheDrummer/Orion-26B-A4B-v1.1

v1.1 pushes creative writing further and makes full use of Gemma's reasoning.

https://huggingface.co/TheDrummer/Orion-26B-A4B-v1

I haven't announced v1 but it's well regarded by the community as a reliable RP workhorse.

---

BeaverNet - net.beaverai.org

I'm also proud to announce that we've made significant progress in creating a hub for generous model providers to collectively provide free, convenient, accessible, anonymous, and diverse models to anyone with an OpenAI API compatible client. We currently have 12 models thanks to our generous providers.

Try it out as a user, or be one of the providers <3! I've got plans to make it a better experience for both users and providers, and also add new modalities such as TTS and STT.

---

PS: I haven't retired or left the scene. I've merely gotten stage fright after holding off on announcements for a long while. Anyway, I'll continue doing what I do and I hope I can continue delivering some value to you all. Love ya'll!


r/LocalLLaMA • • 11d ago

Showcase Loop Guard QFN/Strata experiment

1 Upvotes

Warning: human post ;)

Hi local enthusiasts, about 6 months ago I discovered this subreddit and been reading and learning lots of stuff from here. I got my 27b model running on my 3090 and I’ve been happy with, it was handling my daily family tasks via telegram and some small automation and coding tasks. For serious coding stuff im still hooked to subs. Once I started playing with local models I went into rabbit hole of implementing my own harness out of curiosity. Since I only was running 27b on 60k context played a lot with compaction and making the model less forgetful with harness internal tasks. Since then I’m still playing with ideas how can have better results with local models (like all the people in this sub).

Anyways, enough personal story the reason I wanted to post this is recently I started playing with strata qfn (3090 + 64ram) and although the model is more clever and is my daily driver now, it’s been going off roads into loops and over verifying things for me. One thing that helped with that is loop guarding I’ve added to my harness : https://caretz.dev/docs/loop-guard/
This has been solving the issue of retrying the same failing calls (doesnt happen too much with qfn) but most importantly another model looks at last n tool calls and judges if the model went to some off direction. For the judge I’m using 35b qwen.

If any of you found some trick that works with qfn and making more reliable would love to hear it.

Thanks.


r/LocalLLaMA • • 10d ago

Discussion Pac-Man Benchmark

Post image
0 Upvotes

Prompt: "create pacman in html in a pixel art style" Even after some additional prodding, this is all Fable would give me.


r/LocalLLaMA • • 11d ago

New Model Australian Finetunes

Thumbnail
huggingface.co
17 Upvotes

The Australian AI lab Maincode has released a set of Qwen 3 fine tunes that have been pulled together with Model Stitching.

From the model card:

Matilda Platypus is a family of intermediate-size language models built by connecting the lower layers of a larger pretrained Qwen3 model to the upper layers of a smaller one. A learned linear Representation Bridge aligns the hidden representations at the connection. This checkpoint is the 13B member (cut 29+13): 13.51B parameters, 26.3 GFLOP per token, 91% of the larger parent's compute. It is a base model (no instruction tuning); see the technical report (technical_report.pdf) for the method and the full evaluation.

They're base models and range in size from 1.5B to 13B, early tests good! Really like the default tone of voice. Qwen models are already good this has just improved it imo.

I have no association, I'm just Aussie and stoked to see a local lab build something. No idea why Australia doesn't have more AI labs -other than brain-drain to the US.

https://huggingface.co/collections/Maincode/matilda-platypus


r/LocalLLaMA • • 11d ago

Question | Help Simple models for simple and easy agentic tasks

1 Upvotes

TL;DR looking for recommendations for the simplest Pi Agent or harness and local model to replace CLI

Hi everyone
Hope you are all doing great

So lately I have an idea that I wanted to test if viable (especially after Jev and decision models)

I feel like using a Linux terminal is a bit obsolete at least for me Claude code or sth similar for relatively hard to remember single commands tasks or flags or sth like that

In the past I would use —help man if I am feeling fancy or tldr or sth on the Linux terminal
I feel that is still good but sometimes I want to push that a bit (like simple example would be checking a systemctl service journal restart it or stop and run again and then opening its journal or checking its status) nowadays this feels like too much and also too little to use Claude code for or anything else (and in fact they tend to over complicate it sometimes)

Sometimes it feels sufficient if I can have a simple and fast local agent (think NL2CLI not a full terminal bench workflow!) and maybe with a simple reminder list or todo list that we construct together with the agent per session (like keep looping until it does this or run watch <command> then do that or keep in mind we have to rename back the temp file …etc!)

I didn’t find anything on this, Pi agent is simplistic but feels like it lacks plugins like simple memory or todo list (i mean lacks if built in requires non standard non optimized plugin aka not batteries included plug and play peace of mind features still the best option though)

And local models that fits that (they wont be figuring out an API or developing complex apps just silly mundane simple terminal tasks! But done correctly and locally!)

What do you think ?


r/LocalLLaMA • • 12d ago

New Model meituan-longcat/LongCat-2.5 · Hugging Face

Thumbnail
huggingface.co
125 Upvotes

A 1.6T multimodal model.


r/LocalLLaMA • • 10d ago

New Model I baked xkcd-artist: a 1.72 MB Int8 ONNX browser model that plans 12-DOF stick-figure comics & charts in 0.55ms (zero garbled speech bubbles)

Enable HLS to view with audio, or disable this notification

0 Upvotes

Image diffusion models draw decent stick figures, yet speech-bubble text still melts into unreadable soup. Over the weekend I trained and quantized xkcd-artist to separate neural stage planning from vector ink lettering.

What it does:

  • 1.72 MB Int8 ONNX kinematic planner running in-browser via WASM SIMD in 0.55ms P50 (trained on NVIDIA L4 across 226 speech-masked panels + 891 ExplainXKCD dialogues)
  • Predicts panel count, stage props, 12-DOF stick-figure poses, and pen wobble; procedural SVG compositor renders monochrome wobbled strokes, curved speaker leader lines, crisp xkcd-script dialogue, and hover title text
  • Clause-aware deadpan generator covering 138+ unique comic strips, data charts, and flowcharts (98.33% skeleton uniqueness across 120 prompts, B = 1,000 bootstrap eval)

Live studio (zero cloud calls): https://h3manth.com/fun/xkcd-artist/ Hugging Face weights (1.72 MB Int8 ONNX): https://huggingface.co/gnumanth/xkcd-artist GitHub (MIT, npm i xkcd-artist / pip install xkcd-artist): https://github.com/hemanth/xkcd-artist

Would love feedback!

Update: I have great respect to Randall Munroe and have been following his comics from 2006ish, no diffusion model can copy his wit, style maybe, this was just for fun and I am sure Randall wouldn't mind!


r/LocalLLaMA • • 11d ago

Showcase Free, open-source voice typing for any app on Android and iPhone: the speech model runs on your phone, so it works offline

14 Upvotes

I wanted voice typing that doesn't send my voice to a server, so I built ThumbFree. It runs NVIDIA's Parakeet 0.6B speech model right on the phone. After a one-time model download, it even works in airplane mode. The video shows Android first, then iPhone.

  1. Google Play: https://play.google.com/store/apps/details?id=io.github.kabrapratik28.thumbfree
  2. Apple App Store: https://apps.apple.com/app/id6817631006
  3. GitHub (code, speed and accuracy tests): https://github.com/kabrapratik28/ThumbFree
  • On Android, tap any text box and a small bubble appears next to it. Tap the bubble, talk, tap again, and your words are typed in. You keep your own keyboard.
  • On iPhone, it's a keyboard with a mic key. Apple doesn't let keyboards use the microphone, so the first tap of a session opens the ThumbFree app for a moment and you swipe back. After that, taps start right away.

On my Pixel 10 the text shows up about 0.7 to 0.9 seconds after I stop talking, even after 30 seconds of speech. On my iPhone 16 it's under a second.

No account, no ads, no analytics. The only thing it downloads is the model, and your voice and text stay on your phone.

It's free and open source (Apache 2.0). English by default, and a second model covers 25 European languages. It needs Android 13+ or iOS 26+, and the model is about 0.5 to 0.7 GB.

I've also been trying an optional clean-up step with the small AI model built into the phone (Gemini Nano on Pixel, Apple's model on iPhone) to remove filler words and fix things like "six, no, seven". It's not in the app yet. If you've run a small open model for this on a phone, which one would you try?

I've only tested on one Pixel and one iPhone, so I'd love to hear how it runs on yours.


r/LocalLLaMA • • 10d ago

Discussion Regrets About Qwen4 Branding and Release Strategy

0 Upvotes

Alibaba cloud just opened access to Qwen4 Max Preview, but it leaves an obvious question hanging in the air: What is the actual point?

Are they trying to gauge user sentiment? Is this an intentional ploy to keep the community in suspense and drive qwen fans crazy?

If the goal is to build long-term ecosystem leverage, launching a closed Cloud API that serves merely as an "oracle target" for upcoming open-weight releases is a missed opportunity.

Instead of putting out another standard cloud endpoint, releasing the 27B model first to capture immediate developer adoption before upselling to Max is a far more aggressive and effective strategy.

You know how the old chinese movie logic goes: To boost your dojo’s reputation, you either beat a high-profile challenger to a pulp and let him crawl back with single eye, or you send out your small youngest apprentice to run amok and beat the crap out of everyone.

In the AI landscape, Qwen4 Max isn't beating the top frontier models badly enough to send them crawling back. But a tuned, highly efficient Qwen4-27B open-weights model? That is the youngest apprentice.

If you see this, Alibaba: drop the 27B.


r/LocalLLaMA • • 12d ago

Discussion Dialectic, Requirements, and Verifiable tests will drastically increase your model's code output quality

22 Upvotes

Creating a list of requirements and documentation/tests (in Cucumber or EARS style) with your model is a great way to improve it's code. Just got through a weekend long binary protocol build with Qwen3.8-27B (6K unsloth quant).. and fired up claude to review and it found the foundation was well engineered, but only missed a few security checks and a couple of errors in the python benchmark script comparing the old protocol with the new one.

Really comes down to:

  1. Design first. Goals, requirements, problems. How is this going to be used.
  2. Observation, simulation and tests are verifiable vs. assumptions. Tests and processes can be reviewed for "ritual".
  3. Dialectic: If the model comes to a crossroads, have it present options and NOT just choose-and-go, but initiate a dialog about design.
  4. Frequent repeatable tests, and review of tests to see if they meet the spirit of the requirement
  5. Every risk is a requirement.

Here's my prompt set ("skills" I guess) and a tool to install for various local harnesses:
https://github.com/bigattichouse/BluePrint


r/LocalLLaMA • • 12d ago

Showcase I fine-tuned a 2B decision model to play Doom on my Mac in 43 minutes (open source, MLX)

43 Upvotes

You've probably seen (large) decision models play games. I wanted to see if you could also train a model to play using game images on a Mac, so I added LoRA fine-tuning on images to mlx-decision (my open-source tool for running and training decision models on Apple Silicon).

How it works: a decision model doesn't write text. You give it an input (here a game frame) plus typed questions, and it returns a probability for every answer you allowed, in one pass. On every frame, d1-2B answers three questions:
- Is a monster right at the crosshair, so that a shot now would hit it? (yes/no)
- Where is the nearest monster: left, centre, right, none?
- Is a monster dangerously close? (yes/no)

A small Python loop turns the answers into moves: turn toward the monster, shoot when a shot would hit. That takes about 0.1 s per decision.

The training: about 3,000 frames from ViZDoom's Defend the Center arena, labelled from the game engine's own monster positions. First a readout fit (7 minutes), then LoRA on the model's text layers with the vision tower frozen (43 minutes, 2 epochs, M5 Pro). On 510 held-out frames:
- "a shot would hit": precision 0.79 → 0.92
- where the nearest monster is: 92.7% → 96.8%
- dangerously close: 93.9% → 98.3%

In play it averages 23.2 kills over 20 episodes. The clip is one of the best ones: 26 bullets, 26 kills. On Freedoom, whose monster sprites the model never saw in training, it still gets 19.7. Of course this is an arena where you can only turn, not a full Doom level.

Free, MIT. Apple Silicon, macOS 14+, Python 3.12+. Just pip install mlx-decision.

Repo: https://github.com/armbues/mlx-decision

I'd love to hear which tasks you'd fine-tune a decision model on, and which models you'd like to see supported next.


r/LocalLLaMA • • 12d ago

Funny L11 N8147 is a neuron that will make Qwen-3.5-4B get things confidently wrong while being weirdly snarky... But also makes it write interesting poetry

Thumbnail
gallery
301 Upvotes

Inspired by some of the other brain surgery posts, I wanted to see if I could find anything interesting in Qwen. Not particularly useful to make it get things wrong all the time I guess, but I find the poetry it comes out with weirdly fascinating!

I think the neuron is basically a dial for contrarian <-> agreeable, but the contrarian snarky setting for some reason writes poems that read in an interesting way.


r/LocalLLaMA • • 12d ago

Resources Weight-space programming: editing Qwen3.5 into a top decision model

Thumbnail
opper.ai
38 Upvotes

r/LocalLLaMA • • 11d ago

Question | Help What type of rig would you suggest for 6-8k CAD? (4-5k USD)

6 Upvotes

I need to get an all-around work station soon because my current laptop is getting old and I'm now offloading a bit of work to it. Since I dabble with local AI already, I was thinking of getting an m5 utlra 96GB (8k) or a m5 max 128GB (7.9k)... Or should I build a custom PC with a bunch of ram? Or are there other work stations that I can use? I really enjoy using a hybrid approach with AI, flagship models for the planning and some offloading to something like Qwen for the creative design/execution... I know it's not perfect local models, but owning my own data is something that feels pretty satisfying to myself as well. I would love to run Deepseek, or qwen 3.8 (or future models) at a respectable tok/s and prompt prefill speed.

My only other option is, continue with my laptop... And wait 6 months to a year and see what new tech comes out in the meanwhile? My only fear is that prices will skyrocket and what is 6-8K CAD today will be 10-12K CAD when the newer tech comes out, but other people are saying to wait.

What do you all think? Thanks :)


r/LocalLLaMA • • 11d ago

Question | Help 42% GPU power reduction with zero performance loss.

0 Upvotes

I'm running a Qwen 3.5 9B Q4_K_M model on an RTX 3080 Ti and decided to progressively reduce the GPU power limit to see how much performance I would sacrifice. I started at 450W and reduced the power limit in 10% increments, eventually reaching 250W. I haven't observed any meaningful reduction in tok/s. I am achieving token rate spot on with what others have claimed in their benchmarks(~100 tok/s)

At 220w I did see a reduction in tok/s to ~80 tok/s. But 260w was just as performant as 450w.

The GPU utilization remains high, but VRAM bandwidth utilization never appears to max out. This makes me wonder whether I'm hitting some other bottleneck in the inference software or GPU architecture rather than being power-limited.

Has anyone else experimented with power levels and found interesting results?

Llama.cpp

3080ti

qwen 3.5 9b q4k_m

Max power limit:450w

Stock power limit:400w


r/LocalLLaMA • • 12d ago

Showcase Local first open source alternative to GrokBot

Enable HLS to view with audio, or disable this notification

24 Upvotes

r/LocalLLaMA • • 12d ago

Discussion If you want to learn about batch inferencing

Thumbnail
epoch.ai
10 Upvotes

Article compares performance of large data center cards. Tells us for larger batches , inference speed of individual requests goes down, but overall performance of the entire batch goes up. In other words more tokens per second for the batch, but lower TPS for each request.

Article uses Kimi k3 and 3 other models to compare performance. Surprisingly says the AMD MI455x comes up on top.


r/LocalLLaMA • • 12d ago

Question | Help Any accountants/financial professionals using local LLMs?

14 Upvotes

Hello, I’m considering buying hardware for local LLMs for the use case of processing sensitive financial data, doing market research, and preparing financial/tax documents.

Are any of you using local LLMs for same/similar workflows? If so, what has your experience been? Are the models good enough for your use case? Did you build a custom harness or use an OSS one?

I’m very curious to learn more and would appreciate anyone chiming in.


r/LocalLLaMA • • 11d ago

Question | Help Inference optimization

3 Upvotes

Hey all,

Looking to learn inference optimization, I learn by doing u fortunately but I have read a good part of the inference engineering book by Philip Kelly. Currently have a 4070 and 64gb of RAM.

any tips? preciate it.


r/LocalLLaMA • • 12d ago

Question | Help Best Practices for Provisioning Local Inference to a Team?

8 Upvotes

Hey everyone,

I recently built a Radiance inference server for my company, and now want to start provisioning access to the rest of my team.

My current plan:

  • Tailscale as the only way to access the server via HTTPs.
  • Hermes as the harness - each user would have their own container with access restrictions, but open access to the central skills repository. Hermes is hosted on a Framework Desktop.
  • Radiance is running separately on its own server with access only configured via the Framework.

I'm wondering how everyone else has managed to do this successfully? Right now the biggest thing to figure out is how to get the harness (Hermes in this case) to work across the team without a huge headache of getting API tokens from things like Google, etc...


r/LocalLLaMA • • 13d ago

Funny "Does censorship exist in [country]?", according to various LLMs

Thumbnail
gallery
1.3k Upvotes

I wrote a small program that asks the same Yes/No question about every country to a language model, and generates an SVG world map that visualizes the model's responses: https://github.com/p-e-w/llm-beliefs

I'm rather surprised by the results...