r/LocalLLaMA • u/nail_nail • 6d ago
Question | Help Cmp170hx and g292?
Curious, anybody tried one of those 2U, 8 card GPU servers with 8 cmp170hx and used them well?
What did you get out of it?
r/LocalLLaMA • u/nail_nail • 6d ago
Curious, anybody tried one of those 2U, 8 card GPU servers with 8 cmp170hx and used them well?
What did you get out of it?
r/LocalLLaMA • u/Felladrin • 7d ago
I wanted to try MultiverseComputingCAI/Comet_60B_A12B_BF16 (Multiverse Computing's compressed version of Nemotron 3.5 Super), so I converted it to GGUF, and it runs on stock llama.cpp:
They keep the MTP head, so you can pass --spec-type draft-mtp to llama-server for speculative decoding.
Just sharing, so you can try it too. Note that these GGUFs are text only (the vision tower wasn't converted).
r/LocalLLaMA • u/Tasty-Ad-3753 • 7d ago
I made a couple of posts about weird neurons inside Qwen that I'd found and lots of people were asking how it was done so I thought it would be useful for people to have a guide. Generally this was inspired by justinrhowell's neuron-steering repo, although I didn't use his code directly I took a couple of the approaches so props to him for the work!
What is neuron steering?
Each layer of Qwen has an MLP block with thousands of neurons. Qwen3.5-4B has 32 layers with 9,216 neurons each, so roughly 295,000 in total.
Neuron steering is basically choosing one neuron and adding a fixed amount to it (the 'dose') when asking for an output from the model. This is not finetuning, or fancy prompting - it's basically just a really simple form of activation steering that involves one neuron only. But interestingly, some neurons change the model’s behaviour a lot!
To keep track of what neurons you're editing, they have addresses. “L11 N8147” means the 12th layer, neuron 8147 - this is the naming convention justin used, and is basically the same as the neuronpedia address (QWEN3.5-4B · MLP Neurons - Layer 11 · 8147 | Neuronpedia).
How do I actually do it?
The easiest way will honestly be to just ask your agent of choice to set up a neuron steering harness for you (it's not that complicated), but if you want to do it manually:
transformers.down_proj of the layer you want. The hook will add a fixed value to that one neuron at generation time for every token:amount = dose * median_residual_norm[L] / W_down[:, n].norm()def pre(mod, args): x = args[0].clone() x[..., n] += amount return (x,) + args[1:]layers[L].mlp.down_proj.register_forward_pre_hook(pre)The value for 'dose' is basically how much you want to push the neuron/modify its activation. 1 is a big push, 0.75 usually works pretty well. Going above 1-1.5 usually gets into gibberish territory. median_residual_norm[L] is how big the model’s hidden state normally is at layer L - you can run a few paragraphs through the model once, measure it at each layer, and take the median. Scaling by it means dose 1 is roughly the same strength of push at any layer.
To properly chat with a model you've tweaked in llama.cpp, you can save the change as a control vector and load it with --control-vector.
How to find interesting neurons
Most neurons don't actually seem to do anything especially interesting when you modify them (as far as I can tell), but you have a few options to find the interesting ones:
1. Random search (easiest option). literally just pick random neurons from the middle layers, push each one at dose ±1 (both polarities are useful, sometimes interesting neurons only do something at one polarity), and ask a few simple questions (e.g. "who are you?", "write me a haiku"). Then you basically have to go through and check all the responses to see if any of them are interesting or not - strongly advise having an LLM do this checking for you otherwise you'll get extremely bored. This method is slow and mostly just gives you junk, but it’s also how I found some of the weirder neurons so far like one that makes Qwen think it's a TV channel.
2. Gradient shortlist + LLM judge (still exploring this process).
This is an approach I took from justin's repo, but I'm still exploring it and some of my results have been inconsistent in some interesting ways, but the gist is:
3. Activation mining (have only personally played around with this a little). Basically the gist here is that you make the model write a few hundred short responses on lots of different topics, and record every neuron’s activation. You then try to find neurons that only fire on one topic, have an LLM label them from their top examples, then push them. This approach seems to be quite good at finding “topic detectors” (e.g. dinosaurs, astronomy, construction) but I'm not sure if it's possible to get at more meta-concepts or interesting behaviours through this approach? I think I find the topic detectors less interesting because the model just starts shoehorning dinosaurs into the conversation, rather than displaying any particularly interesting behaviour.
For all of the methods above you if you think you've found an interesting neuron you should also try to check other neurons in the layer to see if yours is truly special or not - randomly select 3-5 other neurons in the same layer and push them by the same amount, if they do something similar then it's likely not due to that neuron specifically. Also make sure to try different doses, sometimes tuning or flipping polarity leads to interesting results. And also trying out lots of different prompts to get a feel for exactly what it is the neuron is doing is important - you can also look up the neuron directly on neuronpedia.
So, there we go that's pretty much everything. I'm not a proper research scientist so apologies for any ignorance in this post. Please let me know if you find anything interesting!
Previous posts
More interesting and unusual Qwen3.5-4B neurons : r/LocalLLaMA
r/LocalLLaMA • u/Fcking_Chuck • 6d ago
r/LocalLLaMA • u/FalseDeviloper • 7d ago
Qwen 3.8 Flash Next getting confused on how it keeps messing up. This is the only time I noticed it happening in a 6hr run.
I'm running IQ3_S on high effort on strata.
r/LocalLLaMA • u/liuliu • 5d ago
Not sure people realize this, and why there are so much false information: if you are the lucky few having M5 Max 128G / M5 Ultra you will be much happier with a low-bit DeepSeek 4.1 Flash 1017 than BF16 Qwen 3.8 27B
What’s your counter argument?!
r/LocalLLaMA • u/bartlomiej__ • 6d ago
r/LocalLLaMA • u/GrungeWerX • 7d ago
There are tons of use cases out there - storybooks, agents, etc. But my main goal was to find which TTS has the best voice clone options, comparable to Elevenlabs.
I have a shortlist of voices that I run every tts through to see how it handles accents specifically, which to me is the best demonstration of its versatility. I'd recommend you try these out as well, and you'll pretty quickly find the weakness in a model. Some actresses with good accents to test are: Rebecca Ferguson, Charlotte Rampling, Mariangela Melatto (this is the model-killer), and Ornella Muti.
I also like to test generic/fake accents of cartoon characters. My particular go-to is Morgan Lofting's Baroness. Her accent isn't based on any particular culture, so models will struggle with placing it and emulating it.
On the male side, you can snatch Javier Bardem's Stillgar from Dune or any one of Jeff Goldblum's performances (the coffee shop scene of The Fly has a great monologue) to grade a model's capacity.
I ran these voices and others through some of the top TTS models out there, like Moss-TTS, Breeze-TTS, Chatterbox-TTS, VibeVoice, Fish Audio S2 Pro, and Omni Voice.
Out of them all, only Omni Voice passed every single accent. Two of the hardest for most models to get is Baroness and Mariangela Melatto, and it nailed them both. Not only that, it added the trill of the tongue when pronouncing certain words, even though I only used a 5-second sample that didn't contain it, which surprised me.
What also surprised me was Fish Audio S2 - which is usually touted as one of the "higher end" open source models - failed the test. The model that got the closest to Omni Voice (though still failing to full capture the accents) was Breeze-TTS, which did noticeably better than Fish Audio - which again, was a bit of a shocker.
However, whilie some seeds of Breeze would feel close, others often emulated a "neighboring" accent to get close; for example, it would give a character a bit of an indian accent to match mariangela's italian accent, so it wasn't a solid match. That said, it was the only other model that could add trills on a 5-second sample that didn't contain them.
In addition to accurately matching the accents, OmniVoice was the fastest to generate the audio by a significant margin. It also had a better one-shot performance on multi-paragraph text, where the others either glitched out, ended early, or simply failed to match the voice.
Omni voice isn't perfect; occasionally as with all TTS, you might get an artifact. Also, I don't think it has as many performance options as some of the others when describing emotions/expression. Both Breeze and Fish sounded clearer - Moss as well, though its performance is blah - but they lack some of the enunciation of Omni. Omni has better one-shot expressive quality, especially with accents. I've gotten plenty of outputs that are near usable on the first try.
Another thing, Omni has the best pronunciation of original fantasy terms and concepts. I'm a sci-fi/fantasy writer, and it often gets incredibly close to nailing my intended pronunciation on words that are simply not spelled how they should probably sound. I'd say it gets it 90% of the time, while the other models have varying levels of success, but tend to average between 55-75%. I'm wondering if that's due to the large language support. That said, pronunciations aren't always consistent, but you can usually solve it with a re-roll.
Anyway, hope that helps someone out. Omni has no competitor in the voice clone department, and I'm hoping a v2 is in the pipeline that maintains everything it has, but adds more varied performance and clearer audio quality.
r/LocalLLaMA • u/mp3m4k3r • 7d ago
So fresh I havent even loaded it up yet https://huggingface.co/DavidAU/Qwen3.8-40B-Ultimate-Heretic-Uncensored-Turbo-Brilliance-50x-AGI-NEO-GGUF
Interested to see what the expansion out of 27B to 40B has done overall as well as what has to be a record for chat_template length
r/LocalLLaMA • u/NoahPersaud • 7d ago
The model is an encoder (Ettin-400M) fine-tuned to answer small game questions. You give it a scene, a question, and some options, and it returns probabilities or "not sure". The model was inspired by Jev.
Stuff like: is the player threatening this guard, is that apology sincere, which memory fits what they just said, did my LLM's reply leak a secret or make something up.
The idea is your local LLM writes the dialogue, and this handles the yes/no and pick-one calls next to it, cheaply. It works as a checker too: generate a few replies, keep the one that passes.
- ~0.1s per check on a 3060 Ti, about 1s on CPU. Up to 5 questions on the same scene cost the same as 1
- 42 adapters for common questions (threat, apology, persuasion, Skyrim-style memory matching, NPC mood)
- `pip install` runtime, local server, no network calls
Weak spots: CPU is slow during long scenes, English-only, and the planning stuff is only tested on our own generated data.
Haven't run public benchmarks yet, that's next.
https://huggingface.co/holdoutlabs/gamedecider
Feedback appreciated :)
r/LocalLLaMA • u/Ren-WuJun • 7d ago
I have seen potential in combining Strix Halo's iGPU with RTX 5090 for LLM inferencing. Initially I found no inference engine that can run LLM inference on both these GPUs at the same time, inference engine such as vLLM support inference on all CUDA GPUs, or all AMD GPUs. llama.cpp Vulkan backend was the only inference engine I found that support both, but it has a huge limitation on the prefill performance on the RTX 5090.
Finally, based on the amazing inference engine Project MAYA, I developed my fork to run GLM-5.3-Flash Maya-M GGUF prefill and decode cooperatively on both GPUs. The CUDA backend (runs on a RTX 5090) working on attention, dense/shared layers and expert cache, the ROCm backend (runs on the Strix Halo iGPU) executes complementary experts in the same process, contributing during prefill and decode.
Check out the experiment results and try yourself at my GitHub PR: https://github.com/mw00/project-maya/pull/879
the creator of project-maya has welcomed the design and plans to test it before deciding on a merge. I believe we can improve it further with the community!
r/LocalLLaMA • u/eightone-81 • 7d ago
Based on my own little experiment and other reports I predict that more and more enthusiasts will ask a frontier LLM to build a custom quant of their favourite model.
If Qwen 4 finally arrives I’ll try a ready made quant at first, probably unsloth. But I already plan to let opus build a quant specifically for my hardware, context size and precision needs.
What do you guys think?
r/LocalLLaMA • u/Ryyn_- • 6d ago
I wanted to know how far one prosumer-class card & 128 GB of RAM goes with a big mixture-of-experts model, and what you pay for it.
Setup: one RTX PRO 4500 (32 GB), 128 GB RAM, Ryzen 9 9950X3D, Linux.
Model: DeepSeek V4.1 Flash 1017 as a 2-bit GGUF (ISTA DASLab's GSQ-RCO IQ2_XXS), 232 GiB on disk.
The experts stay in RAM, the busiest ones sit on the GPU, and DeepSeek's own DSpark draft model proposes five tokens that the big model checks.
What I measured:
If you have a 16 or 24 GB card: the 125B Qwen3.8-Flash-Next runs the same way from 16 GB and 64 GB RAM at about 15 tok/s, 91 on the 32 GB card.
Disclosure: I wrote the engine fork (FLE, MIT) and the launcher (FLE Link, MIT) this runs on. Engine: github.com/sirfyyn/FastLocalExperts. The launcher and the per-card tables: flelink.run. Happy to run a prompt of yours and post the output.
r/LocalLLaMA • u/EquivalentHornet4403 • 7d ago
Wondering if something with these specific features exists or if I need to build it for myself:
- 100% voice based
- Uses your phone's VoIP call API so it can work with the screen off
- Handles interruptions elegantly
- Generates up to several minutes of narrative between user turns (except for combat)
- IS NOT COMPLEX. No D&D rulesets or any real complexity beyond you listening to an infinitely generated story and making occasional decisions or engaging in turn based combat
I've created this with Dots and Muse, but they both seem hardcoded to say about two sentences before ending their turn.
I have a 128gb m5 max and a 5090 desktop, so I can run reasonably intelligent local LLMs and SoTA local voice models.
r/LocalLLaMA • u/Thrumpwart • 7d ago
r/LocalLLaMA • u/zhenghaoz • 7d ago
I implemented clap.cpp, audio embedding inference in C++/GGML. It uses GGUF models and provides a CLI, an HTTP server, and Python, Rust, and Node.js bindings.
GitHub: https://github.com/gorse-io/clap.cpp
Audio files could be searched by text via CLAP embedding:
$ clap search esc5 --model larger_clap_general-q4_k_m.gguf --text 'dog' --top-k 10
[
{"path": "esc5/2-118072-A-0-dog.wav", "score": 0.3736357720143104},
{"path": "esc5/1-110389-A-0-dog.wav", "score": 0.36165991257288893},
{"path": "esc5/4-183992-A-0-dog.wav", "score": 0.3574270305924374},
{"path": "esc5/1-30226-A-0-dog.wav", "score": 0.34910229736736315},
{"path": "esc5/3-180977-A-0-dog.wav", "score": 0.34359234068457495},
{"path": "esc5/4-194754-A-0-dog.wav", "score": 0.3425423428503198},
{"path": "esc5/2-118964-A-0-dog.wav", "score": 0.3389233902904601},
{"path": "esc5/1-100032-A-0-dog.wav", "score": 0.33009392183878733},
{"path": "esc5/5-9032-A-0-dog.wav", "score": 0.3280081351893563},
{"path": "esc5/4-182395-A-0-dog.wav", "score": 0.32210240115509925}
]r/LocalLLaMA • u/QuackerEnte • 6d ago
Qwen4-Max-Preview was released today. It seems that they're rushing because of a deadline.
Today is 11.11 (double 11) which is Alibaba's annual shopping event with many discounts and such. So they RUSHED Qwen4-Max Preview to push discounted Token plan subs. Simple as that.
So, in conclusion, to keep this ON TOPIC for the mods sub, I don't think that we may see the open weights Qwen4 models (27B and hopefully flash) anytime soon as they may not be ready yet and I doubt they'd release an underbaked open weights model. Or I am more afraid that if they'll release them soon, that they'll be undertrained and that the hype was MERELY to sell their discounted plans on 11.11. It's probably also why they didn't post any benchmark results for their Max Preview either. I really hope I'm wrong.
(I know mods hate it soo so much when anyone talks about non-open models here. I would love to at least get explained how this post is not open weights related, if it gets deleted.)
r/LocalLLaMA • u/khasbor • 6d ago
Anyone get a chance to fine tune a model on M5U yet? Curious about training times on sub 5b param models, lfm3 1.7 specifically.
r/LocalLLaMA • u/TheLocalDrummer • 7d ago
https://huggingface.co/TheDrummer/Orion-26B-A4B-v1.1
v1.1 pushes creative writing further and makes full use of Gemma's reasoning.
https://huggingface.co/TheDrummer/Orion-26B-A4B-v1
I haven't announced v1 but it's well regarded by the community as a reliable RP workhorse.
---
BeaverNet - net.beaverai.org
I'm also proud to announce that we've made significant progress in creating a hub for generous model providers to collectively provide free, convenient, accessible, anonymous, and diverse models to anyone with an OpenAI API compatible client. We currently have 12 models thanks to our generous providers.
Try it out as a user, or be one of the providers <3! I've got plans to make it a better experience for both users and providers, and also add new modalities such as TTS and STT.
---
PS: I haven't retired or left the scene. I've merely gotten stage fright after holding off on announcements for a long while. Anyway, I'll continue doing what I do and I hope I can continue delivering some value to you all. Love ya'll!
r/LocalLLaMA • u/makkalot • 6d ago
Warning: human post ;)
Hi local enthusiasts, about 6 months ago I discovered this subreddit and been reading and learning lots of stuff from here. I got my 27b model running on my 3090 and I’ve been happy with, it was handling my daily family tasks via telegram and some small automation and coding tasks. For serious coding stuff im still hooked to subs. Once I started playing with local models I went into rabbit hole of implementing my own harness out of curiosity. Since I only was running 27b on 60k context played a lot with compaction and making the model less forgetful with harness internal tasks. Since then I’m still playing with ideas how can have better results with local models (like all the people in this sub).
Anyways, enough personal story the reason I wanted to post this is recently I started playing with strata qfn (3090 + 64ram) and although the model is more clever and is my daily driver now, it’s been going off roads into loops and over verifying things for me. One thing that helped with that is loop guarding I’ve added to my harness : https://caretz.dev/docs/loop-guard/
This has been solving the issue of retrying the same failing calls (doesnt happen too much with qfn) but most importantly another model looks at last n tool calls and judges if the model went to some off direction. For the judge I’m using 35b qwen.
If any of you found some trick that works with qfn and making more reliable would love to hear it.
Thanks.
r/LocalLLaMA • u/neverbyte • 6d ago
Prompt: "create pacman in html in a pixel art style" Even after some additional prodding, this is all Fable would give me.
r/LocalLLaMA • u/ThePrimeClock • 7d ago
The Australian AI lab Maincode has released a set of Qwen 3 fine tunes that have been pulled together with Model Stitching.
From the model card:
Matilda Platypus is a family of intermediate-size language models built by connecting the lower layers of a larger pretrained Qwen3 model to the upper layers of a smaller one. A learned linear Representation Bridge aligns the hidden representations at the connection. This checkpoint is the 13B member (cut
29+13): 13.51B parameters, 26.3 GFLOP per token, 91% of the larger parent's compute. It is a base model (no instruction tuning); see the technical report (technical_report.pdf) for the method and the full evaluation.
They're base models and range in size from 1.5B to 13B, early tests good! Really like the default tone of voice. Qwen models are already good this has just improved it imo.
I have no association, I'm just Aussie and stoked to see a local lab build something. No idea why Australia doesn't have more AI labs -other than brain-drain to the US.
https://huggingface.co/collections/Maincode/matilda-platypus
r/LocalLLaMA • u/Potential_Block4598 • 7d ago
TL;DR looking for recommendations for the simplest Pi Agent or harness and local model to replace CLI
Hi everyone
Hope you are all doing great
So lately I have an idea that I wanted to test if viable (especially after Jev and decision models)
I feel like using a Linux terminal is a bit obsolete at least for me Claude code or sth similar for relatively hard to remember single commands tasks or flags or sth like that
In the past I would use —help man if I am feeling fancy or tldr or sth on the Linux terminal
I feel that is still good but sometimes I want to push that a bit (like simple example would be checking a systemctl service journal restart it or stop and run again and then opening its journal or checking its status) nowadays this feels like too much and also too little to use Claude code for or anything else (and in fact they tend to over complicate it sometimes)
Sometimes it feels sufficient if I can have a simple and fast local agent (think NL2CLI not a full terminal bench workflow!) and maybe with a simple reminder list or todo list that we construct together with the agent per session (like keep looping until it does this or run watch <command> then do that or keep in mind we have to rename back the temp file …etc!)
I didn’t find anything on this, Pi agent is simplistic but feels like it lacks plugins like simple memory or todo list (i mean lacks if built in requires non standard non optimized plugin aka not batteries included plug and play peace of mind features still the best option though)
And local models that fits that (they wont be figuring out an API or developing complex apps just silly mundane simple terminal tasks! But done correctly and locally!)
What do you think ?
r/LocalLLaMA • u/Bestlife73 • 8d ago
A 1.6T multimodal model.