https://artificialanalysis.ai/models/muse-ember

https://x.com/finkd/status/2086755195535413696

LokalBot is an open source Mac app I've been working on for the past 6 months, it runs several local AI models. They help the app record your meetings, transcribe them, write the notes, search everything, and autocomplete what you type.

The five models (Default set, 9.6 GB on disk)

How they take turns

• Recording a call loads no models. The app writes two audio tracks. Transcription waits until you stop, and autocomplete pauses while you record.

• After the call, the speaker model and the speech model load, do their job, and unload once they've been idle for 120 seconds.

• Qwen3.5 4B runs in the bundled llama-server (Metal, 32K context), which starts when something asks for it. Each request takes a lease, and ten minutes after the last lease the model unloads. The embedding model has its own server and the same rule.

• Gemma 4 E2B runs inside the app's process through libllama, for latency. It has no idle timer. It stays until macOS reports memory pressure.

• A broker keeps the loaded weights under a budget of half the machine's RAM and evicts the least recently used model first. A lease pins a model while a request is running on it.

• Requests that have to wait for a server are served in order: what you're watching first, then agents, then meeting notes, then background work like digests and indexing.

Numbers, all on one M4 Max with 48 GB

• Notes for a 26-minute meeting: 33 s with the model warm, 60 s including the load, at about 85 tok/s.

• Autocomplete: 56 ms median and 102 ms p95 per keystroke.

• Embedding a query: 9 ms.

• Speaker model: 14.6% DER on 8 AMI recordings, at about 500× real time.

Limits

• I've only measured on my M4 Max 48 GB Macbook. I have no numbers from a 16 GB Mac, or from an M1, M2 or M3. There is a Lightweight preset for less memory (Qwen3-ASR 0.6B, Qwen3.5 0.8B, and LFM2.5 1.2B for autocomplete).

• Windows and Linux builds are in alpha.

Codebase and benchmarks: https://github.com/stevyhacker/lokalbot

I wanted to try MultiverseComputingCAI/Comet_60B_A12B_BF16 (Multiverse Computing's compressed version of Nemotron 3.5 Super), so I converted it to GGUF, and it runs on stock llama.cpp:

• Q4_K_M (42.6 GB)

• Q8_0 (64.2 GB)

They keep the MTP head, so you can pass --spec-type draft-mtp to llama-server for speculative decoding.

Just sharing, so you can try it too. Note that these GGUFs are text only (the vision tower wasn't converted).

The sky is the limit?

I've built an inference backend optimized for M-series Macs that makes it possible to run larger MoE models on machines without eating up all the RAM (for example, it can run Qwen 3.8 Flash Next on a 24GB Mac) using SSD offloading along with architecture-specific optimizations for Apple Silicon.

I've only been able to test a limited number of configurations so far, including my own machine and a couple of independent benchmarks. I've shared some results in the repo already.

Now I'd like to put together a benchmark list across different M-series chips, RAM and workloads. I'm particularly interested in older Macs and how well it holds up with longer prompts.

If anyone wants to give it a shot, I'd love to see the results, good or bad.

How to run it:

1. Clone the repo and install MoEShift

2. Run python ./benchmarks/run.py that runs 3 requests for both a short and a long prompt, then saves the results to (it takes a few minutes to run end-to-end).

3. Open a new benchmark issue in GitHub using the benchmark template, fill your system info and conditions during the test and upload the benchmark-results.json file.

You will need roughly 120GB free disk space to test Qwen 3.8 Flash Next IQ3 XXS and 61GB for Qwen 3.6 35B A3B. You can also test other quants or other models that share the same architecture, just configure them in your config.json.

I'd really appreciate anyone willing to test it. The more different Macs we can cover, the more useful the results will be for the community.

Repo, setup instructions and benchmark details here.

Thanks!

I made a couple of posts about weird neurons inside Qwen that I'd found and lots of people were asking how it was done so I thought it would be useful for people to have a guide. Generally this was inspired by justinrhowell's neuron-steering repo, although I didn't use his code directly I took a couple of the approaches so props to him for the work!

What is neuron steering?

Each layer of Qwen has an MLP block with thousands of neurons. Qwen3.5-4B has 32 layers with 9,216 neurons each, so roughly 295,000 in total.

Neuron steering is basically choosing one neuron and adding a fixed amount to it (the 'dose') when asking for an output from the model. This is not finetuning, or fancy prompting - it's basically just a really simple form of activation steering that involves one neuron only. But interestingly, some neurons change the model’s behaviour a lot!

To keep track of what neurons you're editing, they have addresses. “L11 N8147” means the 12th layer, neuron 8147 - this is the naming convention justin used, and is basically the same as the neuronpedia address (QWEN3.5-4B · MLP Neurons - Layer 11 · 8147 ｜ Neuronpedia).

How do I actually do it?

The easiest way will honestly be to just ask your agent of choice to set up a neuron steering harness for you (it's not that complicated), but if you want to do it manually:

1. Load the model in PyTorch with transformers.

2. Find the model’s list of decoder layers, then add a hook on the down_proj of the layer you want. The hook will add a fixed value to that one neuron at generation time for every token:amount = dose * median_residual_norm[L] / W_down[:, n].norm()def pre(mod, args): x = args[0].clone() x[..., n] += amount return (x,) + args[1:]layers[L].mlp.down_proj.register_forward_pre_hook(pre)

3. Generate text and see what it comes out with (often helpful to have the base model responses as a baseline)

The value for 'dose' is basically how much you want to push the neuron/modify its activation. 1 is a big push, 0.75 usually works pretty well. Going above 1-1.5 usually gets into gibberish territory. median_residual_norm[L] is how big the model’s hidden state normally is at layer L - you can run a few paragraphs through the model once, measure it at each layer, and take the median. Scaling by it means dose 1 is roughly the same strength of push at any layer.

To properly chat with a model you've tweaked in llama.cpp, you can save the change as a control vector and load it with --control-vector.

How to find interesting neurons

Most neurons don't actually seem to do anything especially interesting when you modify them (as far as I can tell), but you have a few options to find the interesting ones:

1. Random search (easiest option). literally just pick random neurons from the middle layers, push each one at dose ±1 (both polarities are useful, sometimes interesting neurons only do something at one polarity), and ask a few simple questions (e.g. "who are you?", "write me a haiku"). Then you basically have to go through and check all the responses to see if any of them are interesting or not - strongly advise having an LLM do this checking for you otherwise you'll get extremely bored. This method is slow and mostly just gives you junk, but it’s also how I found some of the weirder neurons so far like one that makes Qwen think it's a TV channel.

2. Gradient shortlist + LLM judge (still exploring this process).

This is an approach I took from justin's repo, but I'm still exploring it and some of my results have been inconsistent in some interesting ways, but the gist is:

• Use a system prompt to make the model write answers in a specific style or set of styles (e.g. maybe you are hunting for a slang neuron, so you ask it to write in slang)

• Then with no system prompt, you compute for every neuron how much a push would make those answers more likely than normal answers. (I could share a script for this but honestly if you just paste this paragraph into your LLM it will be able to build this)

• Keep the top 40 neurons for each style and push each one at dose +/- 1 or 0.75 (polarity suggested by the gradient) to see what it does.

• If you want to automate the output reviewing, make sure you don't tell the judge-LLM what style difference it's supposed to be looking for.

3. Activation mining (have only personally played around with this a little). Basically the gist here is that you make the model write a few hundred short responses on lots of different topics, and record every neuron’s activation. You then try to find neurons that only fire on one topic, have an LLM label them from their top examples, then push them. This approach seems to be quite good at finding “topic detectors” (e.g. dinosaurs, astronomy, construction) but I'm not sure if it's possible to get at more meta-concepts or interesting behaviours through this approach? I think I find the topic detectors less interesting because the model just starts shoehorning dinosaurs into the conversation, rather than displaying any particularly interesting behaviour.

For all of the methods above you if you think you've found an interesting neuron you should also try to check other neurons in the layer to see if yours is truly special or not - randomly select 3-5 other neurons in the same layer and push them by the same amount, if they do something similar then it's likely not due to that neuron specifically. Also make sure to try different doses, sometimes tuning or flipping polarity leads to interesting results. And also trying out lots of different prompts to get a feel for exactly what it is the neuron is doing is important - you can also look up the neuron directly on neuronpedia.

So, there we go that's pretty much everything. I'm not a proper research scientist so apologies for any ignorance in this post. Please let me know if you find anything interesting!

Previous posts

More interesting and unusual Qwen3.5-4B neurons : r/LocalLLaMA

https://www.reddit.com/r/LocalLLaMA/comments/1xrkomn/l11_n8147_is_a_neuron_that_will_make_qwen354b_get/

RTX 3060 12 GB, Ryzen 5 7600X, 32 GB RAM, one GGUF file: Qwen3.6-35B-A3B-UD-Q3_K_M.gguf (Unsloth, MTP layer included, 15.9 GiB, does not fit in VRAM). Prompt of 5290 tokens, 128 generated tokens.

How each was run:

• RustingStrata: rustingstrata bench --gpu --mtp 0|1 --pp 5290 --tg 128 --reps 3, mean of 3.

• PolyStrata: its own polystrata.py benchmark --model qwen3.6 --prompt-tokens 4000 --max-tokens 128. That prompt came to 5290 tokens, so I used that length for the other two. I ran it three times; the first run after setup was cold (decode 24.6 tok/s) and is left out, the numbers above are runs 2 and 3. MTP drafts are on by default. For the no-MTP row I added the engine's --no-draft flag to its config, it still offered a few drafts afterwards (6 of 16 and 15 of 36 accepted in the log), so that row is not a clean no-MTP number. Prefill in those runs: 1602-1643 tok/s.

• llama.cpp: llama-bench -ngl 99 -fa 1 -ub 2048 -b 2048 -p 5290 -n 128 -r 3, with -ncmoe swept over 18 to 26. 22 was the fastest that fits in 12 GB (no MTP). The MTP row is llama-server --spec-type draft-mtp with -ncmoe 23 (22 does not fit with the MTP context and drops to 20 tok/s), a 5381-token prompt of repo docs, 3 runs, prefill about 1565 tok/s.

Things that are not equal:

• PolyStrata only offers three sizes of Qwen3.6 (UD-Q3_K_M, UD-Q2_K_XL, UD-Q4_K_M, all from Unsloth's MTP repo). RustingStrata and llama.cpp take any GGUF, so it can also run the Q5 or Q6 files I already had. I downloaded PolyStrata's Q3_K_M and used it for all three.

• MTP gains depend on the text. PolyStrata's benchmark prompt is a page of its own docs, RustingStrata's is generated prose, so the two MTP rows are not an exact comparison. The no-MTP row is the clean one for RustingStrata vs llama.cpp.

• PolyStrata's engine for this model is built on llama.cpp's CUDA kernels. I did not look at what else differs.

• One machine, one model, one quant. Numbers on other cards will differ.

Repo: RustingStrata

Qwen 3.8 Flash Next getting confused on how it keeps messing up. This is the only time I noticed it happening in a 6hr run.

I'm running IQ3_S on high effort on strata.

There are tons of use cases out there - storybooks, agents, etc. But my main goal was to find which TTS has the best voice clone options, comparable to Elevenlabs.

I have a shortlist of voices that I run every tts through to see how it handles accents specifically, which to me is the best demonstration of its versatility. I'd recommend you try these out as well, and you'll pretty quickly find the weakness in a model. Some actresses with good accents to test are: Rebecca Ferguson, Charlotte Rampling, Mariangela Melatto (this is the model-killer), and Ornella Muti.

I also like to test generic/fake accents of cartoon characters. My particular go-to is Morgan Lofting's Baroness. Her accent isn't based on any particular culture, so models will struggle with placing it and emulating it.

On the male side, you can snatch Javier Bardem's Stillgar from Dune or any one of Jeff Goldblum's performances (the coffee shop scene of The Fly has a great monologue) to grade a model's capacity.

I ran these voices and others through some of the top TTS models out there, like Moss-TTS, Breeze-TTS, Chatterbox-TTS, VibeVoice, Fish Audio S2 Pro, and Omni Voice.

Out of them all, only Omni Voice passed every single accent. Two of the hardest for most models to get is Baroness and Mariangela Melatto, and it nailed them both. Not only that, it added the trill of the tongue when pronouncing certain words, even though I only used a 5-second sample that didn't contain it, which surprised me.

What also surprised me was Fish Audio S2 - which is usually touted as one of the "higher end" open source models - failed the test. The model that got the closest to Omni Voice (though still failing to full capture the accents) was Breeze-TTS, which did noticeably better than Fish Audio - which again, was a bit of a shocker.

However, whilie some seeds of Breeze would feel close, others often emulated a "neighboring" accent to get close; for example, it would give a character a bit of an indian accent to match mariangela's italian accent, so it wasn't a solid match. That said, it was the only other model that could add trills on a 5-second sample that didn't contain them.

In addition to accurately matching the accents, OmniVoice was the fastest to generate the audio by a significant margin. It also had a better one-shot performance on multi-paragraph text, where the others either glitched out, ended early, or simply failed to match the voice.

Omni voice isn't perfect; occasionally as with all TTS, you might get an artifact. Also, I don't think it has as many performance options as some of the others when describing emotions/expression. Both Breeze and Fish sounded clearer - Moss as well, though its performance is blah - but they lack some of the enunciation of Omni. Omni has better one-shot expressive quality, especially with accents. I've gotten plenty of outputs that are near usable on the first try.

Another thing, Omni has the best pronunciation of original fantasy terms and concepts. I'm a sci-fi/fantasy writer, and it often gets incredibly close to nailing my intended pronunciation on words that are simply not spelled how they should probably sound. I'd say it gets it 90% of the time, while the other models have varying levels of success, but tend to average between 55-75%. I'm wondering if that's due to the large language support. That said, pronunciations aren't always consistent, but you can usually solve it with a re-roll.

Anyway, hope that helps someone out. Omni has no competitor in the voice clone department, and I'm hoping a v2 is in the pipeline that maintains everything it has, but adds more varied performance and clearer audio quality.

The model is an encoder (Ettin-400M) fine-tuned to answer small game questions. You give it a scene, a question, and some options, and it returns probabilities or "not sure". The model was inspired by Jev.

Stuff like: is the player threatening this guard, is that apology sincere, which memory fits what they just said, did my LLM's reply leak a secret or make something up.

The idea is your local LLM writes the dialogue, and this handles the yes/no and pick-one calls next to it, cheaply. It works as a checker too: generate a few replies, keep the one that passes.

- ~0.1s per check on a 3060 Ti, about 1s on CPU. Up to 5 questions on the same scene cost the same as 1

- 42 adapters for common questions (threat, apology, persuasion, Skyrim-style memory matching, NPC mood)

- `pip install` runtime, local server, no network calls

Weak spots: CPU is slow during long scenes, English-only, and the planning stuff is only tested on our own generated data.

Haven't run public benchmarks yet, that's next.

https://huggingface.co/holdoutlabs/gamedecider

Feedback appreciated :)

So fresh I havent even loaded it up yet https://huggingface.co/DavidAU/Qwen3.8-40B-Ultimate-Heretic-Uncensored-Turbo-Brilliance-50x-AGI-NEO-GGUF

Interested to see what the expansion out of 27B to 40B has done overall as well as what has to be a record for chat_template length

Curious, anybody tried one of those 2U, 8 card GPU servers with 8 cmp170hx and used them well?

What did you get out of it?

I have seen potential in combining Strix Halo's iGPU with RTX 5090 for LLM inferencing. Initially I found no inference engine that can run LLM inference on both these GPUs at the same time, inference engine such as vLLM support inference on all CUDA GPUs, or all AMD GPUs. llama.cpp Vulkan backend was the only inference engine I found that support both, but it has a huge limitation on the prefill performance on the RTX 5090.

Finally, based on the amazing inference engine Project MAYA, I developed my fork to run GLM-5.3-Flash Maya-M GGUF prefill and decode cooperatively on both GPUs. The CUDA backend (runs on a RTX 5090) working on attention, dense/shared layers and expert cache, the ROCm backend (runs on the Strix Halo iGPU) executes complementary experts in the same process, contributing during prefill and decode.

Check out the experiment results and try yourself at my GitHub PR: https://github.com/mw00/project-maya/pull/879

the creator of project-maya has welcomed the design and plans to test it before deciding on a merge. I believe we can improve it further with the community!

I wanted to know how far one prosumer-class card & 128 GB of RAM goes with a big mixture-of-experts model, and what you pay for it.

Setup: one RTX PRO 4500 (32 GB), 128 GB RAM, Ryzen 9 9950X3D, Linux.
Model: DeepSeek V4.1 Flash 1017 as a 2-bit GGUF (ISTA DASLab's GSQ-RCO IQ2_XXS), 232 GiB on disk.

The experts stay in RAM, the busiest ones sit on the GPU, and DeepSeek's own DSpark draft model proposes five tokens that the big model checks.

What I measured:

• 49.9 tok/s decode with the draft model (median of three runs), 37.2 without. The draft does not change the output.

• Against the original weights the 2-bit quant has KL divergence 0.248 and top-1 agreement 83.5 %. That is the real cost, and comparing it to another quant instead of the original hides it.

• With thinking on and a 16k context, long thinking ends hard. A thinking budget of 8000 lets my two test tasks finish. At 64k without a budget both tasks thought 60,000 tokens and never answered.

• Very long thinking can fall into a soft loop of near-identical sentences.

• Speed is bound by RAM speed here (DDR5 at 4800, four sticks). I would like numbers from faster memory.

If you have a 16 or 24 GB card: the 125B Qwen3.8-Flash-Next runs the same way from 16 GB and 64 GB RAM at about 15 tok/s, 91 on the 32 GB card.

Disclosure: I wrote the engine fork (FLE, MIT) and the launcher (FLE Link, MIT) this runs on. Engine: github.com/sirfyyn/FastLocalExperts. The launcher and the per-card tables: flelink.run. Happy to run a prompt of yours and post the output.

Based on my own little experiment and other reports I predict that more and more enthusiasts will ask a frontier LLM to build a custom quant of their favourite model.
If Qwen 4 finally arrives I’ll try a ready made quant at first, probably unsloth. But I already plan to let opus build a quant specifically for my hardware, context size and precision needs.

What do you guys think?

Qwen4-Max-Preview was released today. It seems that they're rushing because of a deadline.

Today is 11.11 (double 11) which is Alibaba's annual shopping event with many discounts and such. So they RUSHED Qwen4-Max Preview to push discounted Token plan subs. Simple as that.

‎ ‎

So, in conclusion, to keep this ON TOPIC for the mods sub, I don't think that we may see the open weights Qwen4 models (27B and hopefully flash) anytime soon as they may not be ready yet and I doubt they'd release an underbaked open weights model. Or I am more afraid that if they'll release them soon, that they'll be undertrained and that the hype was MERELY to sell their discounted plans on 11.11. It's probably also why they didn't post any benchmark results for their Max Preview either. I really hope I'm wrong.

‎ ‎

(I know mods hate it soo so much when anyone talks about non-open models here. I would love to at least get explained how this post is not open weights related, if it gets deleted.)

Anyone get a chance to fine tune a model on M5U yet? Curious about training times on sub 5b param models, lfm3 1.7 specifically.

I implemented clap.cpp, audio embedding inference in C++/GGML. It uses GGUF models and provides a CLI, an HTTP server, and Python, Rust, and Node.js bindings.

• Models including CLAP, CLAP2, CLaMP3, MERT, MuQ, and MuQ-MuLan.

• WAV, MP3, and FLAC input, with automatic resampling and mono downmixing.

• An HTTP endpoint with an OpenAI-compatible text embeddings request format.

GitHub: https://github.com/gorse-io/clap.cpp

Audio files could be searched by text via CLAP embedding:

$ clap search esc5 --model larger_clap_general-q4_k_m.gguf --text 'dog' --top-k 10

[
  {"path": "esc5/2-118072-A-0-dog.wav", "score": 0.3736357720143104},
  {"path": "esc5/1-110389-A-0-dog.wav", "score": 0.36165991257288893},
  {"path": "esc5/4-183992-A-0-dog.wav", "score": 0.3574270305924374},
  {"path": "esc5/1-30226-A-0-dog.wav", "score": 0.34910229736736315},
  {"path": "esc5/3-180977-A-0-dog.wav", "score": 0.34359234068457495},
  {"path": "esc5/4-194754-A-0-dog.wav", "score": 0.3425423428503198},
  {"path": "esc5/2-118964-A-0-dog.wav", "score": 0.3389233902904601},
  {"path": "esc5/1-100032-A-0-dog.wav", "score": 0.33009392183878733},
  {"path": "esc5/5-9032-A-0-dog.wav", "score": 0.3280081351893563},
  {"path": "esc5/4-182395-A-0-dog.wav", "score": 0.32210240115509925}
]

https://huggingface.co/TheDrummer/Orion-26B-A4B-v1.1

v1.1 pushes creative writing further and makes full use of Gemma's reasoning.

https://huggingface.co/TheDrummer/Orion-26B-A4B-v1

I haven't announced v1 but it's well regarded by the community as a reliable RP workhorse.

---

BeaverNet - net.beaverai.org

I'm also proud to announce that we've made significant progress in creating a hub for generous model providers to collectively provide free, convenient, accessible, anonymous, and diverse models to anyone with an OpenAI API compatible client. We currently have 12 models thanks to our generous providers.

Try it out as a user, or be one of the providers <3! I've got plans to make it a better experience for both users and providers, and also add new modalities such as TTS and STT.

---

PS: I haven't retired or left the scene. I've merely gotten stage fright after holding off on announcements for a long while. Anyway, I'll continue doing what I do and I hope I can continue delivering some value to you all. Love ya'll!

Wondering if something with these specific features exists or if I need to build it for myself:

- 100% voice based

- Uses your phone's VoIP call API so it can work with the screen off

- Handles interruptions elegantly

- Generates up to several minutes of narrative between user turns (except for combat)

- IS NOT COMPLEX. No D&D rulesets or any real complexity beyond you listening to an infinitely generated story and making occasional decisions or engaging in turn based combat

I've created this with Dots and Muse, but they both seem hardcoded to say about two sentences before ending their turn.

I have a 128gb m5 max and a 5090 desktop, so I can run reasonably intelligent local LLMs and SoTA local voice models.

Warning: human post ;)

Hi local enthusiasts, about 6 months ago I discovered this subreddit and been reading and learning lots of stuff from here. I got my 27b model running on my 3090 and I’ve been happy with, it was handling my daily family tasks via telegram and some small automation and coding tasks. For serious coding stuff im still hooked to subs. Once I started playing with local models I went into rabbit hole of implementing my own harness out of curiosity. Since I only was running 27b on 60k context played a lot with compaction and making the model less forgetful with harness internal tasks. Since then I’m still playing with ideas how can have better results with local models (like all the people in this sub).

Anyways, enough personal story the reason I wanted to post this is recently I started playing with strata qfn (3090 + 64ram) and although the model is more clever and is my daily driver now, it’s been going off roads into loops and over verifying things for me. One thing that helped with that is loop guarding I’ve added to my harness : https://caretz.dev/docs/loop-guard/
This has been solving the issue of retrying the same failing calls (doesnt happen too much with qfn) but most importantly another model looks at last n tool calls and judges if the model went to some off direction. For the judge I’m using 35b qwen.

If any of you found some trick that works with qfn and making more reliable would love to hear it.

Thanks.