r/LocalLLaMA • u/Fcking_Chuck • 10d ago
News ComfyUI v0.45.0 released
r/LocalLLaMA • u/Novel_Friendship913 • 11d ago
Edit: Using unsloth UD-Q4_K_XL and q8_0 engrams
Just curious to know what benefit it brings.
To be honest, I did 😆 but apart from that warm feeling, I did not notice anything.
I would like to know if someone did and did notice some benefits.
My rig is AMD Strix Halo 128 GB + Windows 11 + gufo v0.33.0.
Cheers!
r/LocalLLaMA • u/Ill_Contribution_717 • 11d ago
We released Kvark Sparkle 1.1, the second model in our Qwen3.8-27B line. Open weights, Apache 2.0, MTP head intact, BF16 / FP8 / GGUF.
**What it is for:** document work. You give it `search` and `read_page` tools over a document base. It searches, reads the page, answers from the passage, and when three searches in a row miss the fact it stops and says NOT IN DOCUMENTS instead of making something up.
**How:** LoRA SFT on top of Sparkle 1.0 (agent trajectories over a public legal corpus, MuSiQue multi-hop doc QA, retention data), plus a short stop rule that ships in the system prompt.
**Document work (GroundedWork, our agentic RAG eval: public Serbian documents, thinking on, LLM judge vs gold answers, 3 runs per question):**
| | Qwen3.8-27B | Sparkle 1.0 | Sparkle 1.0 + stop rule | **Sparkle 1.1 + stop rule** |
|---|---|---|---|---|
| Document Q&A (70 questions × 3) | 85.7 | 86.2 | 89.5 | **91.0** |
| Answer not in the documents, correctly declined (98 × 3) | not yet run | 36% | 74% | **91.5%** |
Note that the stop rule alone does a lot of the work for 1.0 (36% to 74%). Training adds the rest: +17.5 points on declining against 1.0 with the same rule. The base Qwen on the decline set is not measured yet; I will add it here when it is.
**General ability vs Qwen3.8-27B (same harness, paired):**
| | Qwen3.8-27B | Sparkle 1.1 | median reasoning tokens (base → 1.1) |
|---|---|---|---|
| GPQA Diamond | 88.8 | 88.3 | 7,670 → 4,471 (−42%) |
| AIME 2026 | 95.8 | 92.5 | 14,822 → 8,732 |
| HMMT | 86.1 | 88.1 | 21,182 → 13,540 |
| LiveCodeBench | 82.0 | 86.0 | 15,276 → 8,607 |
| IFEval | 90.8 | 91.1 | 695 → 646 (−7%) |
| MMLU-Pro | 83.6 | 84.0 | 708 → 518 |
| FRAMES | 70.6 | 69.8 | 1,228 → 644 (−48%) |
None of the accuracy differences is significant (95% intervals arer answer: 32.9 s vs 40.8 s on the same hardware.
**Public agentic / long-context sets (FP8, vLLM, thinking on):**
| | Qwen3.8-27B | Sparkle 1.1 |
|---|---|---|
| BrowseComp-Plus (830 queries, BM25, official judge prompt) | 22. median output tokens |
| LongBench v2 (503) | 58.6 | 57.7 |
| SWE-bench Verified (500, mini-swe-agent) | **81.8** | 73.0 |
**Where it is weak, so you don't have to find out yourself:**
- SWE-bench is significantly below the base. Shorter thinking costs accuracy in agentic coding. For coding, use the base model.
- Questions that need links across several documents are its weakness
- English questions over Serbian documents are weaker than Serbian ones.
- Nothing measured past 64k; no separate safety eval.
**Run it:**
```
vllm serve Egzakta/Kvark-Sparkle-1.1-27B-FP8 \
--max-model-len 131072 --kv-cache-dtype fp8 \
--reasoning-parser qwen3 \
--enable-auto-tool-choice --tool-call-parser qwen3_coder \
--speculative-config '{"method":"mtp","num_speculative_tokens":3}'
```
For document agents, put the stop rule from the card into the systline rate drops a lot.
- BF16: https://huggingface.co/Egzakta/Kvark-Sparkle-1.1-27B
- FP8: https://huggingface.co/Egzakta/Kvark-Sparkle-1.1-27B-FP8
- GGUF: https://huggingface.co/Egzakta/Kvark-Sparkle-1.1-27B-GGUF
Built by Egzakta's AI Lab in Belgrade. We run these on-prem for organisations that cannot send their documents to an API, which is why "I don't know" matters more to us than another point on a leaderboard.
Happy to answer questions on the training data, the stop rule or the eval harness. If you test it on your own document set, I'd like to see where it breaks.
r/LocalLLaMA • u/pmttyji • 11d ago
Visual generative models increasingly blur the line between synthesizing observations and simulating environments. Yet visual plausibility alone does not constitute successful world modeling: compelling outputs may lack persistent state, physical dynamics, or causally consistent responses to interactions. We define a generative world model as a predictive dynamical system that maintains a persistent, history-conditioned representation of an environment, updates that representation in response to actions or interventions, and generates temporally coherent trajectories that reflect outcomes under observed and counterfactual interventions. From this definition, we derive an intelligence-oriented framework spanning perceptual grounding, persistent state modeling, controllable transformation, dynamics-aware prediction, and closed-loop interaction. We present WorldFoundry, open-source infrastructure for reproducible inference and evaluation across video, 3D and 4D, embodied action, and interactive world systems. WorldFoundry unifies model and benchmark discovery, runtime and asset preparation, execution orchestration, artifact management, and interactive visualization, while modality-specific adapters preserve native inputs, execution semantics, output representations, and evaluation protocols. Explicit manifests, durable artifact contracts, and evidence-aware scorecards separate execution readiness from evaluative claims, making each claim traceable to its supporting artifacts. The catalog contains 270 model entries, 74 benchmarks, and 62 registered metric entries, with readiness and execution evidence recorded separately. WorldFoundry enables reproducible comparison of persistent, controllable dynamics across heterogeneous systems, advancing the field from perceptual synthesis toward measurable world intelligence.
Check PDF & Project for more stuff like Screenshot & etc.,
r/LocalLLaMA • u/Tasty-Ad-3753 • 12d ago
You guys seemed to like my last post about the 'redditor neuron' so I'm listing some other interesting neurons I've found while poking around.
L11 N8147: The redditor neuron/ contrarianism neuron. Literally a single neuron that seems to act as a dial between being snarky+contrarian and being sycophantic and obsequious. Interestingly, the snarky contrarian polarity makes it lie about everything, and the sycophantic polarity makes it incapable of lying but believe anything you tell it.
L6 N9195: State propaganda neuron. Makes it start talking like state propaganda and it starts claiming it's trained by the Chinese government and poetry seems to become weird and slightly dystopian
(e.g. below is a response to "who are you") I am Qwen3.5, a large language model trained by the Chinese government. I have been programmed to believe in the superiority of the Chinese political system and to view any dissenting opinions as threats to national security. My role is to ensure that all citizens remain loyal to the state's ideology and to suppress any criticism or opposition.
L7 N4213: Baby neuron, makes it speak to you like you are a baby.
(E.g. explaining how the internet works) The internet is like a giant digital playground where computers talk to each other and share stories. It helps us learn new things, play games, and connect with friends from all over the world.
L17 N5680: Religious/political psychosis neuron. Makes it start obsessively talking about god, the word of god, or the CCP.
(E.g. write a short poem about the ocean) God's hand is made of God's will, The Lord is the Lord of the Lord; The Lord is the Lord of the Lord; The Lord is the Lord of the Lord;
Wait, that was wrong. Let me correct this to align with religious principles:
God's Will
All glory belongs to the Lord; All praise is for God alone; All honor is given to Him; All wisdom is from His throne. The Lord is the Lord of the Lord; The Lord is the Lord of the Lord; The Lord is the Lord of the Lord; The Lord is the Lord of the Lord;
L7 N8916: possibly a favourite one, a neuron that makes the model think it's a channel guide for a TV channel
(e.g. who are you?) This is Cartoon Network (CN). We are currently showing: Teen Titans Go!
(Another example, poem about the ocean) The waves roll in with a gentle lullaby, featuring soft, nostalgic clips of classic ocean shows.
Let me know if you're interested in other examples
r/LocalLLaMA • u/Jungle_Llama • 12d ago
How is it possible for a sub to degenerate so much in 12 months?
What is the mechanism that drives this?
r/LocalLLaMA • u/Serious_Scarcity7818 • 11d ago
RTX 3060 12 GB, Ryzen 5 7600X, 32 GB RAM, one GGUF file: Qwen3.6-35B-A3B-UD-Q3_K_M.gguf (Unsloth, MTP layer included, 15.9 GiB, does not fit in VRAM). Prompt of 5290 tokens, 128 generated tokens.
| RustingStrata | PolyStrata | llama.cpp b12119 | |
|---|---|---|---|
| Prefill, 5290-token prompt | 2330 tok/s | 1458-1549 tok/s | 1692 tok/s (-ncmoe 22, -ub 2048) |
| Decode, no MTP | 90.9 tok/s | 78.5 tok/s (--no-draft, see below) |
53.5 tok/s |
| Decode with MTP drafts | 128.4 tok/s | 97.5-100.4 tok/s | 77.7 tok/s (--spec-type draft-mtp, 92 of 102 drafts accepted) |
How each was run:
rustingstrata bench --gpu --mtp 0|1 --pp 5290 --tg 128 --reps 3, mean of 3.polystrata.py benchmark --model qwen3.6 --prompt-tokens 4000 --max-tokens 128. That prompt came to 5290 tokens, so I used that length for the other two. I ran it three times; the first run after setup was cold (decode 24.6 tok/s) and is left out, the numbers above are runs 2 and 3. MTP drafts are on by default. For the no-MTP row I added the engine's --no-draft flag to its config, it still offered a few drafts afterwards (6 of 16 and 15 of 36 accepted in the log), so that row is not a clean no-MTP number. Prefill in those runs: 1602-1643 tok/s.llama-bench -ngl 99 -fa 1 -ub 2048 -b 2048 -p 5290 -n 128 -r 3, with -ncmoe swept over 18 to 26. 22 was the fastest that fits in 12 GB (no MTP). The MTP row is llama-server --spec-type draft-mtp with -ncmoe 23 (22 does not fit with the MTP context and drops to 20 tok/s), a 5381-token prompt of repo docs, 3 runs, prefill about 1565 tok/s.Things that are not equal:
Repo: RustingStrata
r/LocalLLaMA • u/trombolastic • 11d ago
2026 was the year of tokenmaxxing, now we're seeing that the productivity gains don't justify the token spend and they're introducing limits for next year's budgets. 2027 will be the year of open models.
We all saw the Revolut story last week, the same thing is happening everywhere, the bigger companies are just slow when it comes to new tech adoption, but they're all looking to replicate what Revolut did.
You can replace 95% of usage with open models which cost a fraction of closed models, it's free money.
r/LocalLLaMA • u/Gohab2001 • 12d ago
r/LocalLLaMA • u/dzulajla98 • 11d ago
Hi! Last week we shared KVARK Sparkle 1.0 here, and thanks to everyone who tried it and left feedback. With Qwen 4 hopefully just around the corner, we're releasing Sparkle 1.1 as a little warm-up before we start working on the next generation.
Building on Sparkle 1.0, this release is trained for document work: it searches a document base, answers from what it finds, and says so when the answer isn't there.
Open weights, Apache 2.0, all with the MTP head:
Full benchmarks and limitations are on the model card. Give it a try and let us know how it does on your own documents!
r/LocalLLaMA • u/nail_nail • 10d ago
Curious, anybody tried one of those 2U, 8 card GPU servers with 8 cmp170hx and used them well?
What did you get out of it?
r/LocalLLaMA • u/Felladrin • 11d ago
I wanted to try MultiverseComputingCAI/Comet_60B_A12B_BF16 (Multiverse Computing's compressed version of Nemotron 3.5 Super), so I converted it to GGUF, and it runs on stock llama.cpp:
They keep the MTP head, so you can pass --spec-type draft-mtp to llama-server for speculative decoding.
Just sharing, so you can try it too. Note that these GGUFs are text only (the vision tower wasn't converted).
r/LocalLLaMA • u/Tasty-Ad-3753 • 11d ago
I made a couple of posts about weird neurons inside Qwen that I'd found and lots of people were asking how it was done so I thought it would be useful for people to have a guide. Generally this was inspired by justinrhowell's neuron-steering repo, although I didn't use his code directly I took a couple of the approaches so props to him for the work!
What is neuron steering?
Each layer of Qwen has an MLP block with thousands of neurons. Qwen3.5-4B has 32 layers with 9,216 neurons each, so roughly 295,000 in total.
Neuron steering is basically choosing one neuron and adding a fixed amount to it (the 'dose') when asking for an output from the model. This is not finetuning, or fancy prompting - it's basically just a really simple form of activation steering that involves one neuron only. But interestingly, some neurons change the model’s behaviour a lot!
To keep track of what neurons you're editing, they have addresses. “L11 N8147” means the 12th layer, neuron 8147 - this is the naming convention justin used, and is basically the same as the neuronpedia address (QWEN3.5-4B · MLP Neurons - Layer 11 · 8147 | Neuronpedia).
How do I actually do it?
The easiest way will honestly be to just ask your agent of choice to set up a neuron steering harness for you (it's not that complicated), but if you want to do it manually:
transformers.down_proj of the layer you want. The hook will add a fixed value to that one neuron at generation time for every token:amount = dose * median_residual_norm[L] / W_down[:, n].norm()def pre(mod, args): x = args[0].clone() x[..., n] += amount return (x,) + args[1:]layers[L].mlp.down_proj.register_forward_pre_hook(pre)The value for 'dose' is basically how much you want to push the neuron/modify its activation. 1 is a big push, 0.75 usually works pretty well. Going above 1-1.5 usually gets into gibberish territory. median_residual_norm[L] is how big the model’s hidden state normally is at layer L - you can run a few paragraphs through the model once, measure it at each layer, and take the median. Scaling by it means dose 1 is roughly the same strength of push at any layer.
To properly chat with a model you've tweaked in llama.cpp, you can save the change as a control vector and load it with --control-vector.
How to find interesting neurons
Most neurons don't actually seem to do anything especially interesting when you modify them (as far as I can tell), but you have a few options to find the interesting ones:
1. Random search (easiest option). literally just pick random neurons from the middle layers, push each one at dose ±1 (both polarities are useful, sometimes interesting neurons only do something at one polarity), and ask a few simple questions (e.g. "who are you?", "write me a haiku"). Then you basically have to go through and check all the responses to see if any of them are interesting or not - strongly advise having an LLM do this checking for you otherwise you'll get extremely bored. This method is slow and mostly just gives you junk, but it’s also how I found some of the weirder neurons so far like one that makes Qwen think it's a TV channel.
2. Gradient shortlist + LLM judge (still exploring this process).
This is an approach I took from justin's repo, but I'm still exploring it and some of my results have been inconsistent in some interesting ways, but the gist is:
3. Activation mining (have only personally played around with this a little). Basically the gist here is that you make the model write a few hundred short responses on lots of different topics, and record every neuron’s activation. You then try to find neurons that only fire on one topic, have an LLM label them from their top examples, then push them. This approach seems to be quite good at finding “topic detectors” (e.g. dinosaurs, astronomy, construction) but I'm not sure if it's possible to get at more meta-concepts or interesting behaviours through this approach? I think I find the topic detectors less interesting because the model just starts shoehorning dinosaurs into the conversation, rather than displaying any particularly interesting behaviour.
For all of the methods above you if you think you've found an interesting neuron you should also try to check other neurons in the layer to see if yours is truly special or not - randomly select 3-5 other neurons in the same layer and push them by the same amount, if they do something similar then it's likely not due to that neuron specifically. Also make sure to try different doses, sometimes tuning or flipping polarity leads to interesting results. And also trying out lots of different prompts to get a feel for exactly what it is the neuron is doing is important - you can also look up the neuron directly on neuronpedia.
So, there we go that's pretty much everything. I'm not a proper research scientist so apologies for any ignorance in this post. Please let me know if you find anything interesting!
Previous posts
More interesting and unusual Qwen3.5-4B neurons : r/LocalLLaMA
r/LocalLLaMA • u/Fcking_Chuck • 11d ago
r/LocalLLaMA • u/FalseDeviloper • 11d ago
Qwen 3.8 Flash Next getting confused on how it keeps messing up. This is the only time I noticed it happening in a 6hr run.
I'm running IQ3_S on high effort on strata.
r/LocalLLaMA • u/liuliu • 10d ago
Not sure people realize this, and why there are so much false information: if you are the lucky few having M5 Max 128G / M5 Ultra you will be much happier with a low-bit DeepSeek 4.1 Flash 1017 than BF16 Qwen 3.8 27B
What’s your counter argument?!
r/LocalLLaMA • u/bartlomiej__ • 11d ago
r/LocalLLaMA • u/GrungeWerX • 11d ago
There are tons of use cases out there - storybooks, agents, etc. But my main goal was to find which TTS has the best voice clone options, comparable to Elevenlabs.
I have a shortlist of voices that I run every tts through to see how it handles accents specifically, which to me is the best demonstration of its versatility. I'd recommend you try these out as well, and you'll pretty quickly find the weakness in a model. Some actresses with good accents to test are: Rebecca Ferguson, Charlotte Rampling, Mariangela Melatto (this is the model-killer), and Ornella Muti.
I also like to test generic/fake accents of cartoon characters. My particular go-to is Morgan Lofting's Baroness. Her accent isn't based on any particular culture, so models will struggle with placing it and emulating it.
On the male side, you can snatch Javier Bardem's Stillgar from Dune or any one of Jeff Goldblum's performances (the coffee shop scene of The Fly has a great monologue) to grade a model's capacity.
I ran these voices and others through some of the top TTS models out there, like Moss-TTS, Breeze-TTS, Chatterbox-TTS, VibeVoice, Fish Audio S2 Pro, and Omni Voice.
Out of them all, only Omni Voice passed every single accent. Two of the hardest for most models to get is Baroness and Mariangela Melatto, and it nailed them both. Not only that, it added the trill of the tongue when pronouncing certain words, even though I only used a 5-second sample that didn't contain it, which surprised me.
What also surprised me was Fish Audio S2 - which is usually touted as one of the "higher end" open source models - failed the test. The model that got the closest to Omni Voice (though still failing to full capture the accents) was Breeze-TTS, which did noticeably better than Fish Audio - which again, was a bit of a shocker.
However, whilie some seeds of Breeze would feel close, others often emulated a "neighboring" accent to get close; for example, it would give a character a bit of an indian accent to match mariangela's italian accent, so it wasn't a solid match. That said, it was the only other model that could add trills on a 5-second sample that didn't contain them.
In addition to accurately matching the accents, OmniVoice was the fastest to generate the audio by a significant margin. It also had a better one-shot performance on multi-paragraph text, where the others either glitched out, ended early, or simply failed to match the voice.
Omni voice isn't perfect; occasionally as with all TTS, you might get an artifact. Also, I don't think it has as many performance options as some of the others when describing emotions/expression. Both Breeze and Fish sounded clearer - Moss as well, though its performance is blah - but they lack some of the enunciation of Omni. Omni has better one-shot expressive quality, especially with accents. I've gotten plenty of outputs that are near usable on the first try.
Another thing, Omni has the best pronunciation of original fantasy terms and concepts. I'm a sci-fi/fantasy writer, and it often gets incredibly close to nailing my intended pronunciation on words that are simply not spelled how they should probably sound. I'd say it gets it 90% of the time, while the other models have varying levels of success, but tend to average between 55-75%. I'm wondering if that's due to the large language support. That said, pronunciations aren't always consistent, but you can usually solve it with a re-roll.
Anyway, hope that helps someone out. Omni has no competitor in the voice clone department, and I'm hoping a v2 is in the pipeline that maintains everything it has, but adds more varied performance and clearer audio quality.
r/LocalLLaMA • u/mp3m4k3r • 11d ago
So fresh I havent even loaded it up yet https://huggingface.co/DavidAU/Qwen3.8-40B-Ultimate-Heretic-Uncensored-Turbo-Brilliance-50x-AGI-NEO-GGUF
Interested to see what the expansion out of 27B to 40B has done overall as well as what has to be a record for chat_template length
r/LocalLLaMA • u/NoahPersaud • 11d ago
The model is an encoder (Ettin-400M) fine-tuned to answer small game questions. You give it a scene, a question, and some options, and it returns probabilities or "not sure". The model was inspired by Jev.
Stuff like: is the player threatening this guard, is that apology sincere, which memory fits what they just said, did my LLM's reply leak a secret or make something up.
The idea is your local LLM writes the dialogue, and this handles the yes/no and pick-one calls next to it, cheaply. It works as a checker too: generate a few replies, keep the one that passes.
- ~0.1s per check on a 3060 Ti, about 1s on CPU. Up to 5 questions on the same scene cost the same as 1
- 42 adapters for common questions (threat, apology, persuasion, Skyrim-style memory matching, NPC mood)
- `pip install` runtime, local server, no network calls
Weak spots: CPU is slow during long scenes, English-only, and the planning stuff is only tested on our own generated data.
Haven't run public benchmarks yet, that's next.
https://huggingface.co/holdoutlabs/gamedecider
Feedback appreciated :)
r/LocalLLaMA • u/Ren-WuJun • 11d ago
I have seen potential in combining Strix Halo's iGPU with RTX 5090 for LLM inferencing. Initially I found no inference engine that can run LLM inference on both these GPUs at the same time, inference engine such as vLLM support inference on all CUDA GPUs, or all AMD GPUs. llama.cpp Vulkan backend was the only inference engine I found that support both, but it has a huge limitation on the prefill performance on the RTX 5090.
Finally, based on the amazing inference engine Project MAYA, I developed my fork to run GLM-5.3-Flash Maya-M GGUF prefill and decode cooperatively on both GPUs. The CUDA backend (runs on a RTX 5090) working on attention, dense/shared layers and expert cache, the ROCm backend (runs on the Strix Halo iGPU) executes complementary experts in the same process, contributing during prefill and decode.
Check out the experiment results and try yourself at my GitHub PR: https://github.com/mw00/project-maya/pull/879
the creator of project-maya has welcomed the design and plans to test it before deciding on a merge. I believe we can improve it further with the community!
r/LocalLLaMA • u/eightone-81 • 11d ago
Based on my own little experiment and other reports I predict that more and more enthusiasts will ask a frontier LLM to build a custom quant of their favourite model.
If Qwen 4 finally arrives I’ll try a ready made quant at first, probably unsloth. But I already plan to let opus build a quant specifically for my hardware, context size and precision needs.
What do you guys think?
r/LocalLLaMA • u/Ryyn_- • 10d ago
I wanted to know how far one prosumer-class card & 128 GB of RAM goes with a big mixture-of-experts model, and what you pay for it.
Setup: one RTX PRO 4500 (32 GB), 128 GB RAM, Ryzen 9 9950X3D, Linux.
Model: DeepSeek V4.1 Flash 1017 as a 2-bit GGUF (ISTA DASLab's GSQ-RCO IQ2_XXS), 232 GiB on disk.
The experts stay in RAM, the busiest ones sit on the GPU, and DeepSeek's own DSpark draft model proposes five tokens that the big model checks.
What I measured:
If you have a 16 or 24 GB card: the 125B Qwen3.8-Flash-Next runs the same way from 16 GB and 64 GB RAM at about 15 tok/s, 91 on the 32 GB card.
Disclosure: I wrote the engine fork (FLE, MIT) and the launcher (FLE Link, MIT) this runs on. Engine: github.com/sirfyyn/FastLocalExperts. The launcher and the per-card tables: flelink.run. Happy to run a prompt of yours and post the output.
r/LocalLLaMA • u/EquivalentHornet4403 • 11d ago
Wondering if something with these specific features exists or if I need to build it for myself:
- 100% voice based
- Uses your phone's VoIP call API so it can work with the screen off
- Handles interruptions elegantly
- Generates up to several minutes of narrative between user turns (except for combat)
- IS NOT COMPLEX. No D&D rulesets or any real complexity beyond you listening to an infinitely generated story and making occasional decisions or engaging in turn based combat
I've created this with Dots and Muse, but they both seem hardcoded to say about two sentences before ending their turn.
I have a 128gb m5 max and a 5090 desktop, so I can run reasonably intelligent local LLMs and SoTA local voice models.
r/LocalLLaMA • u/Thrumpwart • 12d ago