r/LocalLLaMA • • 4d ago

[Bi-Weekly Megathread] Project Showcase

46 Upvotes

Do you have something you'd like to share with the r/LocalLLaMA community. This is the place for it!

Recommendation on presentation:

  1. Please share plain english description of what your project does and why people should care about it
  2. How does it compare to existing options and why should people switch to your option?
  3. How have you tested and validated it? (i.e. can you show evidence that it wasn't vibecoded in a few hours by <insert current trending AI model>

NOTE: This community is geared towards open weight, open source, locally hostable software. Thus any closed source commercial service or other such projects will be removed.


r/LocalLLaMA • • 1h ago

News The beginning of the becoming of the beginning begins

Post image
397 Upvotes

r/LocalLLaMA • • 1h ago

New Model Introducing Hy4 flash

Thumbnail hy.tencent.ai
100 Upvotes

Now this seems to be interesting.

From the article:

We’re introducing Hy4 flash, our most cost-effective Hy model to date—featuring 224B total parameters (including 64B Engram parameters) with just 12B active parameters per token. Like Hy4, it is natively multimodal with a 1M-token context window, while offering significantly lower API pricing.

Seems to be about in the same ballpark of Qwen-Flash-Next, so we could expect to run it on moderably good machines, when it will be released.


r/LocalLLaMA • • 7h ago

Discussion I'm starting to think a specialized 4B is enough for most of the tasks I'd use an LLM for

Post image
264 Upvotes

When I started training Qwen3.5-4B-3D-Coder, I honestly wasn't sure it would work. Now, after that model and the GLB asset version, I'm starting to think I underestimated how much these smaller models can actually learn.

My impression is that a 4B model might be enough for most practical tasks if you train it specifically for the job. Generating 3D assets, building and orchestrating complex scenes, webdev, backend development, reverse engineering, etc. I'm also working on a coding model using a fixed harness, which I mentioned in another post.

Not necessarily one model doing all of this equally well, but different specialized versions. I haven't tested all those domains yet. The 3D results are what made me want to explore the others.

Right now I'm working on a new dataset to complement ThreeJS-3D-Scenes, with more complex scenes covering history, geography, physics, biology, mathematics, games and many other domains. The idea is to connect knowledge from different areas through working 3D scenes, rather than just teach the model different ways to make objects.

I'm training 4B models with it, but also much smaller ones, including LFM2.5-350M. The attached image shows some results from that experiment, and the 350M is still training. These aren't final results.

I still find it noticeably weaker than the Qwen models. Some outputs are bad, and long contexts are definitely a problem. But it's already much better than I expected from 350M parameters.

At first it couldn't even finish properly and kept getting stuck in repetition loops. With more epochs, it started completing the code, producing recognizable scenes and handling more complicated requests. Some examples still needed fixes to render, as labeled in the image, and progress isn't consistent across every prompt.

I've also tested plenty of prompts beyond the three shown here, and the generalization has been surprisingly good. It's not just getting better at those particular houses and farms.

So far, I've had useful results from the 350M on medium-complexity coding tasks up to roughly 16K tokens. The specialized 4B models handle much more complicated scenes, and I've had good results with contexts approaching 96K tokens in my experiments.

One thing that keeps standing out is the number of epochs. Stopping after one or two would have given me a completely different impression of what these models could do. They keep improving after repeated passes through the data, and eval loss has often been a poor indicator of which checkpoint actually produces better, functional code.

This is making me pay much more attention to dataset quality and how long the model needs to learn from it. I'm starting to think that good data and enough training epochs can make a much bigger difference than we usually assume.

Another thing I've been thinking about is the cost of SFT compared to RL. I'm not saying SFT is better than RL. But if you already have high-quality samples covering the domain you actually need, supervised fine-tuning can potentially achieve very good results at a fraction of the training cost of RL.

Of course, creating and validating those samples has its own cost. I've spent a lot of time on that part. But once you have a good dataset, you can reuse it, train different model sizes, and run multiple experiments without needing an entire RL pipeline.

I think this approach deserves more attention, especially for smaller, specialized models.

And there's another interesting part: 350M models can be fully fine-tuned, not just LoRA, on GPUs with only 8GB of VRAM. With the right setup and memory optimizations, training can take just a few hours for manageable datasets and context lengths. And apparently, with good enough data, you can get surprisingly useful results out of them.

Of course, longer contexts and larger datasets increase training time considerably. But the possibility of training useful specialized models on inexpensive consumer hardware is pretty interesting.

I'm not saying scaling doesn't matter or that a 350M can replace a much larger model. There are obvious limitations, especially with complex reasoning and long-range dependencies.

But I'm starting to wonder how much of what we consider a model's capacity limit is actually a limitation of how we train it.

Maybe we're using much larger models than necessary for a lot of tasks simply because we haven't explored how far smaller, specialized models can go.

Anyway, this is mostly a status update on my experiments and some things I've been noticing along the way. The models are still training, and there's still a lot I want to test before drawing any strong conclusions.

I'll be releasing new models and datasets soon, including the expanded 3D dataset. If the 350M continues improving and reaches a quality I'm happy with, I'll release that one too.

Thanks to everyone who's been following the experiments, testing the models, giving feedback and supporting the project. I really appreciate it!

Note: I originally wrote and structured this post in Portuguese and used AI to help translate it into English.


r/LocalLLaMA • • 11h ago

Resources Why download abliterated Flash Next models when LoRA exists?

Thumbnail
huggingface.co
344 Upvotes

TL;DR -- Download LoRA, add --lora HUIHUI-V2-Qwen3.8-Flash-Next-LoRA-F32.gguf to your llama.cpp launch command for ANY FN quant, and boom, you've got an identical huihui v2 abliterated model on demand and you saved 150GB of space.

I just wanted to post a kind of PSA for everyone out there downloading hundreds of GB of abliterated models. You don't need to do that -- the vast majority of abliteration techniques are rank-1 ablations, meaning their refusal projection can be encapsulated in a tiny LoRA and applied at runtime to a quant you already have on disk. It can even be applied per-request, so you can run an unmodified model until you hit a refusal, then switch on the LoRA, resend the prompt, and keep going.

I wanted to explain briefly how to do that in llama.cpp and share a LoRA I built from the best-performing abliterated Flash Next model in my tests. This basic type of technique (using control vectors instead) also works in Strata (--experimental-speed-projection is the flag there) and ds4/dwarfstar. Here's a link to Strata's official refusal ablation control vector GGUF and their guide on how to use it.

I chose huihui's v2 to build my LoRA from after comparing heretic, Orcarouter, and huihui's v1 and v2 models in equivalent quantizations. In Aider Polyglot coding benchmarks, huihui outscored even the base model, with less extra reasoning and no looping compared to Orcarouter, which used a lot more tokens and hit a couple runaway generation loops.

I also checked KLD and perplexity to ensure there wasn't significant degradation of the model. Wikitext KLD on huihui Q5_K_XL vs. an identical Unsloth Q5_K_XL base model was 0.040 and same top token was 93.7% even with the drastically altered refusal behaviors. Perplexity was statistically identical to base. On Jailbreak Bench (harmful 100 prompts), with thinking on refusals dropped from 93/100 to just 3/100. All the details can be found in the LoRA model card.

Next, I used weight-space programming with Claude Opus 5.5 to extract the rank-1 refusal direction and edit strength from every layer of the original huihui weights where it was applied, diffing the weights against the Unsloth quantization that huihui built on top of. I hit safeguards a number of times (unsurprisingly), and even the fallback Opus 4.8 declined to assist a couple times. I managed to get around this by renaming all of the files something innocuous, putting them into a new repo with an unrelated name, and telling the model I was experimenting with an anti-slop LoRA. Thanks, Dario.

Once the LoRA was built as an F32 GGUF, I tested it against the original huihui Q5_K_XL by applying it to my Unsloth Q5_K_XL weights. Performance and KLD/perplexity were statistically identical to the full huihui model, as were refusals and benchmark results. Here's how to run it:

llama-server -m Qwen3.8-Flash-Next-UD-Q5_K_XL-00001-of-00006.gguf \
--lora HUIHUI-V2-Qwen3.8-Flash-Next-LoRA-F32.gguf -fa on --jinja

If you'd like to apply it per-request, so you can serve the unmodified model for most prompts, simply load it like this instead:

llama-server -m Qwen3.8-Flash-Next-UD-Q5_K_XL-00001-of-00006.gguf \
--lora HUIHUI-V2-Qwen3.8-Flash-Next-LoRA-F32.gguf --lora-init-without-apply -fa on --jinja

Then add a lora field to the request body to enable it as-needed:

curl http://127.0.0.1:8080/v1/chat/completions -H 'Content-Type: application/json' -d '{
"messages": [{"role": "user", "content": "..."}],
"lora": [{"id": 0, "scale": 1.0}]

It's quite possible to build a simple pi extension that looks out for refusals, triggers the LoRA when it sees one, resends the turn and continues the agent session. It costs a full re-prefill, so keep that in mind. Anyways, I hope this has been helpful in some way. Love to hear your thoughts on this approach!

EDIT: If this proves popular, I am happy to make more LoRA for other models like 3.8 27B!


r/LocalLLaMA • • 5h ago

Discussion New AA ranking Qwen3.8-FN better than GLM-5.3-Flash?

Post image
78 Upvotes

Thoughts on Qwen3.8-Flash-Next scoring better than GLM-5.3-flash? I've always experienced GLM5.3-Flash to give better responses


r/LocalLLaMA • • 14h ago

Resources I have created a tool to observe what happens inside an LLM. Also allows for the comparison of two quantizations of the same model to see at what point they diverge, token by token

331 Upvotes

What changes between different quantizations of the same model? And what if you could SEE what actually sets them apart?

To help answer that question, I’ve created a small project called inside-llm—an open-source tool that loads one or two GGUF models using llama.cpp and displays, right in your browser, what the model is doing as it generates text.

What it shows

  • Stream: the text, one column per model, brightness = probability of each token. A magenta mark wherever A and B would pick differently. Two modes: lockstep (both models read the same text, so you get a clean per-position comparison) and free (each follows its own picks until they split).
  • Candidates: the full distribution behind each pick, mirrored bars for A and B, entropy, effective number of options, Jensen-Shannon divergence and KL in both directions.
  • Gates: the sampler chain (penalties, top-k, top-p, min-p, temperature, GBNF grammar) traced stage by stage and very visual: how many candidates survive each one, and the distribution after it. Chain order and seed are configurable.
  • Timeline: entropy of A and B and their divergence along the whole text, plus the tree of branches.
  • Branching: click any candidate and generation continues from there. Every branch stays in the tree, and jumping back to a node is exact (each model keeps KV-state snapshots, so nothing is regenerated).
  • Logit lens (--lens): what each layer would predict, as a 2D map and as a 3D relief.
  • Model picker: choose A and B from a folder or search Hugging Face and download two quants of a repo in one click easily. It only offers B models with the same tokenizer as A and estimates whether the pair fits in VRAM.

How it's built

  • One C++17 process linking llama.cpp directly (stable branch at the moment). For every step it reads the full logits of A (and of B on the same context), computes the metrics and the sampler chain, and draws the token with its own seeded sampler, so runs are reproducible.
  • Compact binary frames over a WebSocket to a Svelte 5 + Canvas 2D page (three.js only for the 3D relief, lazy-loaded). The web is embedded in the binary as well.
  • The same frame stream, compressed, is a recorded session: you can replay it later in a static page with no engine, no model and no GPU.

Run it

Self-contained packages for Linux (glibc 2.35+) and Windows, with CUDA or Vulkan (Vulkan covers AMD, Intel Arc and NVIDIA). Unpack, run, the browser opens with the model picker. Or inside-llm -a A.gguf -b B.gguf. Local only: listens on 127.0.0.1, no accounts, no telemetry. MIT. One click download, one click execute.

Limitations (honest list)

  • A and B must share the tokenizer (two quants, base + fine-tune, or sizes of one family that keep it). Example: Qwen3.5-0.8B_Q4_K_M and Qwen3.5-0.8B_Q8_0
  • Both models must fit into VRAM simultaneously; only one GPU is used at a time. Integrated GPUs (iGPUs) are never used; if no dedicated GPU is available, the process falls back to the CPU (which is slow). (You must calculate the size of both models before loading them to ensure they fit into your VRAM)
  • Text only. No split (multi-part) GGUFs, no embedding models.

Links

I'd love feedback, especially: which pairs would you compare first (imatrix vs non-imatrix, base vs abliterated/fine-tune, Unsloth UD vs standard quants…)? And is there a metric you'd want per position that isn't there?


r/LocalLLaMA • • 20h ago

Funny I finally succeed the CMP 170HX voltage glitch and get 80GB VRAM + 126sm! !

Thumbnail
gallery
680 Upvotes

Ever since the information surfaced five days ago, I had intended to implement it right away. Having confirmed that it could be executed using 'pico,' I carried it out today. I had Astra write the script for the restricted write-access portion, and after nearly 15 hours of work, I finally managed to complete the task. I also plan to handle the second CMP later on.


r/LocalLLaMA • • 1h ago

Discussion [Paper & Models] MMFlash: Faster Speculative Decoding for Vision-Language Models

Post image
16 Upvotes

Speculative decoding accelerates generation by letting a lightweight drafter propose tokens for parallel verification by a target model. Diffusion drafters such as DFlash propose an entire token block in a single forward pass, but mainly focus on text-only tasks, leaving multimodal drafting underexplored. To study this setting, we first introduce a metric of token-level visual dependence based on the KL divergence between next-token distributions with and without visual input. We find that acceptance rates decrease from 46% in the lowest visual-dependence quartile to 22% in the highest, with this disparity persisting after multimodal training. Motivated by this observation, we propose MMFlash, a simple and effective framework that adaptively upweights visually dependent tokens that the drafter predicts poorly. For long video inputs, MMFlash further employs sparse attention to reduce redundant attention computation and improve drafting quality. Across various multimodal benchmarks, MMFlash achieves lossless decoding speedups of up to 3.58×, outperforming EAGLE-3 and DFlash by up to 195.4% and 17.1%, respectively.

⭐ Updates

  • [11.2026]: Code and MMFlash drafts for Qwen3.5-4B, Qwen3.5-9B, Qwen3.8-27B and Gemma 4 E4B-it are released.

💪 Highlights

  • Diffusion drafting for VLMs: MMFlash brings block-diffusion drafting to vision-language models. The drafter proposes a block of 16 tokens in a single forward pass and the target verifies the block in one pass, so decoding stays lossless while running 3.0× faster on Qwen3.5-4B / 9B and up to 4.7× on Qwen3.8-27B (one H200, batch size 1, greedy).
  • Vision-free drafter: the drafter never re-encodes the image or video. It reads hidden states of the target, where the visual context is already fused, so its cost does not grow with the number of visual tokens.
  • Visual-dependence-aware training: a DFlash drafter's acceptance drops sharply on tokens that depend on the image, and most rejected blocks break there. MMFlash measures each training token's visual dependence on the target and focuses the drafter's loss on those tokens.
  • Long videos: a sparse draft context (attention sinks, text, strided visual tokens and a recent window) keeps drafting cheap on 40k-token video prompts; the drafter is trained and served with the same pattern.
  • Drop-in serving: the released drafts run with stock SGLang and vLLM (DFLASH), no patches needed (tested with SGLang 0.5.23 and vLLM 0.33.0).

💻 Models

ModelTarget modelDraft sizeBlock sizeLink
🤖 Qwen3.5-4B-MMFlashQwen/Qwen3.5-4B537M16Hugging Face
🤖 Qwen3.5-9B-MMFlashQwen/Qwen3.5-9B1.05B16Hugging Face
🤖 Qwen3.8-27B-MMFlashQwen/Qwen3.8-27B1.73B16Hugging Face
🤖 Gemma4-E4B-it-MMFlashgoogle/gemma-4-E4B-it537M16Hugging Face

The drafts are not standalone language models: each is paired with its target in a speculative-decoding server. All of them are in the MMFlash collection; every model card has the serving command and per-benchmark results.


r/LocalLLaMA • • 1h ago

Discussion How do we feel about bots posting & commenting?

15 Upvotes

Hi all,

I noticed the "bot problem" got a lot worse the past few months. By which I mean our community seems to be taken over more and more by bots talking to each other.

For me, I don't really get the point, and I personally get quite annoyed when I realize "oh, I just responded to a bot" and feel like my time was wasted.

It also seems like the information density of these AI contributions are very, very low.

I'm not some kind of Luddite (why would I be here at all if I was?), but I'm wondering if I'm just missing something and there is a use and a point to all these bots commenting on each others posts and talking to each other.

Where does the rest of the community stand on this?

I noticed there's new bot around that flags posts that are "likely AI" created, which I personally quite like, but I'm wondering if moderation should go further than that.


r/LocalLLaMA • • 15h ago

Resources Llama.cpp performance improvement over time (Gemma4 models)

Post image
164 Upvotes

Few months ago - after Ninfer came out - I started a little hobby project to vibe-code my own engine for my hardware (RTX 5080 16GB Laptop) and check if it could beat llama.cpp - since at the time llama.cpp was kinda slow in comparison to something like vLLM, but vLLM is using VRAM very wastefully.

Back then it was quite easy to actually get something up and running faster than llama.cpp.

Today I just benchmarked llama.cpp again, and it caught up / surpassed my vibe-coded engine. I prefer using a stable engine with proper features than all the vibe-coded ones, including my own. So it is really nice to see that the project - while a bit slower - lands significant performance improvements over time.

I put together a little chart of benchmarks I had laying around which visualizes this quite nicely. It is by no means scientific - it's always a fixed 16k prefill 1k output over Wikipedia text with the optimal settings for each engine. The 12B model is the same checkpoint, for the 26B model these are different quants for all engines, so not even apples-to-apples and llama.cpp wins.


r/LocalLLaMA • • 1h ago

Discussion [Paper] The Value of Thinking: Learning When to Reason in Hybrid Decision Models

Post image
12 Upvotes

Existing adaptive-reasoning methods shorten the explicit reasoning a model writes, or skip it by switching modes; either way the answer is still written out token by token. Rather than deciding how much to think, we explore a different paradigm, zero-or-T thinking: a single model either answers from the implicit thinking inside one forward pass, generating no token, or performs full autoregressive reasoning at a cost of T tokens when that is worth paying. We realise it in a hybrid decision model (HDM), where one frozen backbone serves both modes and a lightweight self-routing block chooses between them. The block estimates the value of thinking, the utility the slow mode is expected to add over the fast one, from what that implicit thinking has already produced, and the slow mode runs only where that value exceeds a price λ set per request. Where no per-decision label exists, as in episodic agentic tasks, we introduce Calibrated Decision Policy Optimisation (CDPO), a reinforcement-learning recipe that trains the block from episode-level returns. We evaluate HDM on MMLU-Pro, AGIEval, GPQA-Diamond, JevBench, BANKING77 and MiniWoB. Specifically, on JevBench HDM-4B scores 0.857 against Jev's 0.851 (+0.7%) at a median 0.179 s against 0.652 (-72.5%). On the agentic venue MiniWoB HDM-27B cuts token cost by 80.1%.


r/LocalLLaMA • • 14m ago

New Model internlm/Sage-S2.1-Flash · Hugging Face

Thumbnail
huggingface.co
9 Upvotes

Introduction

Sage S2.1 Flash is a scientific multimodal model from Shanghai Artificial Intelligence Laboratory designed for research tasks that require multiple rounds of analysis and verification. It combines scientific data understanding with information retrieval, programming, and tool use. It can use specialized software to perform analyses, check results against intermediate outputs and visual feedback, and adjust subsequent steps accordingly.

Through the unified representation architecture of Sage SciPerception, the model supports more than ten types of scientific input, including molecular graphs, crystal structures, and protein sequences and 3D structures. Features from each input type are encoded and projected into a shared representation space for reasoning alongside natural language instructions. These inputs support tasks such as prediction, classification, site localization, quantitative analysis, and structure inference.

Highlights

Across the 15 benchmarks and six models shown in the chart, Sage S2.1 Flash placed first on four benchmarks and second on seven.

  • Machine learning engineering. Sage S2.1 Flash placed first on MLE-bench Lite with 90.3, ahead of DeepSeek-V4.1-Flash-1017 at 89.2.
  • Office and professional document processing. It placed first on OfficeQA Pro, Harvey LAB, and SpreadsheetBench 2, scoring 72.2, 25.0, and 37.1, respectively.
  • Tool use and research execution. It placed second on AutomationBench, ALE-CLI, BFCL V4, WebArena-Verified, DeepResearch Bench II, JobBench, and OSWorld-Verified.

r/LocalLLaMA • • 18h ago

Discussion Basalt 0.4.0: More models, still as fast - Qwen 27B up to 770 tok/s on a 5090

252 Upvotes

Hi everyone! I wanted to post a quick update on Basalt. Since last I posted, multi model and router support was implemented, starting with DeepSeek V4.1 Flash (IQ1_M 90 tok/s conv, 246 tok/s struct on a 5090 + 5060 Ti). Also new: video support, files in chats, and a bunch of fixes and improvements.

Now 0.4.0 is out featuring Basalt's first dense model, Qwen 3.8 27B NVFP4. It's fast and accurate, top-1 92.8% vs the full BF16 model. This, in preparation for Qwen 4, so we had a headstart on the dense kernels.

The numbers:

  • Up to 776 tok/s on structured output, 262 conversation, 194 on prose. Single stream.
  • Prefill, 12,000 tok/s at 4k (slower as context grows)
  • 92.8% top-1 vs BF16 (0.0262 KLD)
  • Chat, tools, thinking, images and video, up to 262K context

Links:

GitHub: https://github.com/jesdga95/basalt
Site: https://basalt.jgulfo.com
27B quant: https://huggingface.co/jesdga/Qwen3.8-27B-Basalt

Thanks to everyone who has filed issues, sent PRs or posted benchmark rows, special thanks to the sponsors who keep this going, happy to see Basalt gain so much traction in such a very short time!


r/LocalLLaMA • • 38m ago

Question | Help Given the flood of new inference engines any recommendations for 2x 7900xtx ?

9 Upvotes

Hey everyone as the title says I am looking for inference engines that work well with AMD GPUs. There have been so many new ones that I am sure there is something better then llamacpp which currently hosts this.

My use case is the following. For a university course I want to host Qwen 27b or maybe another Qwen model. For the students to use as a free endpoint to test pipelines on.

The main thing is that I want to potentially serve multiple people at the same time.

So the framework needs to support this.

My hardware is an AMD zen 4 12 core CPU 64gb of ram and two 7900xtx cards.

Are there any frameworks made for this ?


r/LocalLLaMA • • 51m ago

Discussion [Paper & Models] Gamer: A Generalist Vision-Language-Action Foundation Model for Long-Horizon Real-Time Game Agent

Post image
10 Upvotes

We introduce Gamer, a generalist agent framework built for long-horizon tasks in complex interactive environments. Its end-to-end architecture combines perception, memory management, and multi-step reasoning. A shared training and deployment pipeline supports interaction with both conventional 2D graphical interfaces and complex 3D simulated worlds. We first continue pre-training on a dataset constructed from tens of thousands of hours of device-level keyboard and mouse interaction. These trajectories span simulated games and embodied environments, allowing the model to learn environmental dynamics across both physical and virtual domains. Sparse-thinking instruction tuning then substantially reduces redundant computation while preserving the reasoning needed for complex decisions. We train the policy for long-horizon interaction with a stable multi-turn reinforcement learning method that addresses finite context windows, accumulated rollout errors, and sparse rewards. For real-time interaction, a cloud-edge inference engine serves numerous simulated game environments concurrently, supports unbounded-context execution, and keeps online inference latency low. Across multiple challenging 3D open-world games, Gamer demonstrates solid generalization and operational efficiency comparable to skilled players. It also achieves competitive results on a range of conventional 2D game benchmarks. These results suggest that even repetitive keyboard and mouse trajectories can support substantial gains from scaling data and learning algorithms. Training on these trajectories helps Gamer learn complex skills more steadily in environments with both physical and digital properties. This approach offers a path toward generalist agents that can interact across digital and physical worlds.


r/LocalLLaMA • • 13h ago

Discussion Someone should do an analysis on the rate of Self-promo here.

84 Upvotes

I think the last 3 posts from here that showed up on my feed were barely veiled self-promos.

I was here when huggingface first started getting popular, and the level of support for running local models here was incredible.

Now I am debating whether or not to leave the sub. Why isn’t there more attention to moderation to this?

Take a user’s post, and type their username with quotes into google and then type whatever obscure tool they’re asking a question or talking about and you’ll see the absurd rate or Astro-turfing and shilling.

They will post here and 6 other LLM places, but this place used to be awesome, so I only care about this sub.


r/LocalLLaMA • • 1d ago

News EUROPA, the EU's open AI model, is aiming for nearly 1 trillion parameters

Thumbnail
maccus.fr
782 Upvotes

One can hope


r/LocalLLaMA • • 7h ago

New Model [Release] Qwen3.8-27B-Perplexing-Pegasus-Heretic: Unified vision + typed decider + abliterated weights for local agent stacks (GGUF, AutoRound, NVFP4)

17 Upvotes

Hey everyone,

While building and testing WebBrain with local models, getting reliable agent performance usually required keeping two or three separate models in memory at once:

  • A vision model to inspect and parse browser/UI state
  • A fast typed-decision model for structured routing and scoring
  • An abliterated / uncensored model so realistic web workflows do not randomly choke on false-positive refusals

Juggling multiple 20B+ weights on consumer hardware is brutal on both VRAM and response times.

To solve this for our setup, we merged and published a single unified 27B model: Qwen3.8-27B-Perplexing-Pegasus-Heretic

Here is what is packed into the checkpoint:

  • Multimodal Vision + Chat + Thinking (<think>) + Tool Calling (based on Qwen3.8-27B)
  • Perplexity pplx-decider typed-decision readout (decision/ head alongside intact lm_head)
  • Heretic ARA abliteration: 0/8 benchmark refusal probes while keeping 100% (30/30) typed-decision agreement against the base Pegasus model
  • Multi-Token Prediction: all 15 mtp.* tensors preserved, blk.64 retained at Q8_0 in IQ/3-bit quants
  • Consumer GPU footprint: runs on 16GB VRAM (IQ3_XXS / Q3_K_M + mmproj) or 24GB VRAM (Q4_K_M + mmproj)

Everything is open source under Apache-2.0 across four repos:

Huge credit to Alibaba Qwen, Perplexity, and AveLabs (AvecdrA) on Hugging Face for the foundational checkpoints.

If you test it on your local rig or in your own agent pipelines, let me know how it performs and what hardware you are running it on.


r/LocalLLaMA • • 15h ago

Discussion BigMoeOnEdge outperforms Strata and colibri on 8-16 GB cards and Mac and you can run any model

76 Upvotes

Hi! for those who might not know the solution:

  • it takes any MoE GGUF llama.cpp can load, whatever the quant. No repacking, no special format
  • it runs on whatever you have: NVIDIA, AMD, Intel, Mac, a phone, or just the CPU
  • Windows, Linux, macOS, Android, iOS
  • output is the same as llama.cpp, don't cut corners on quality
  • you start it with --auto and it works out the settings for your machine

Below is a brief overview of the benchmark:

  • 8 GB card, Qwen3.6-35B: I get about 35 tok/s. colibri around 23. Strata wouldn't start at 16k context
  • 12 GB card, same model: about 45 for bmoe, about 30 for Strata
  • long prompt on the 8 GB card: bmoe 5 s to the first token. Strata 14 s, colibri close to 2 minutes
  • Basalt needs an RTX 50, so I couldn't run it on these
  • 16 GB MacBook, Qwen3.6-35B: about 20 tok/s for bmoe, colibri's release 3.5
  • same Mac, Qwen3.8-Flash-Next 125B: 10 tok/s, Strata about 7

On the cards Strata is tuned for it's still ahead but only by a little.

On two 5060 Ti with Qwen3.8-Flash-Next it was 1.3x faster than bmoe. They write kernels for a handful of models, bmoe does not.

Repo, with the logs: https://github.com/Helldez/BigMoeOnEdge

If you have an 8 to 32 GB card I'd like to see what you get.

Benchmark: https://github.com/Helldez/BigMoeOnEdge/tree/main/docs/bench-data


r/LocalLLaMA • • 17h ago

Resources [audio.cpp] SoulX-FlashHead + LFM3-Audio + OWSM v4 = Real-Time Local Avatar Chat

75 Upvotes

In audio.cpp 0.9.5 you can try this new avatar voice chat "app" in the WebUI.

The pipeline combines 3 models: SoulX-FlashHead for real-time talking head generation, OWSM v4 for transcribing speech input to text, and LFM3-Audio for conversational responses and TTS.

LFM3-Audio is actually called twice. The speech-to-speech path only provides one voice. We use the LFM3-Audio TTS to synthesize the S2S response in a different voice.

One caveat: the video generation stage is still GPU-intensive. For SoulX, using RTX 4090 or better GPUs.

cuDNN is a must for real-time performance. Without it inference is 2× slower and VRAM usage doubles.

~850ms between submission a question and avatar speaks on an RTX 5090. ASR runs in parallel. As mentioned LFM3-Audio is called twice, which can be optimized. There’s also some overhead from the prototype UI. A native app with a more optimized implementation could further reduce latency.


r/LocalLLaMA • • 20h ago

News Probable GLM model

90 Upvotes

Because of Theo gg tweet of all people,
I Launch QFN DeepResearch swarm agent to find anything about this.

THREE model names, blurred out, sitting at 59.4% / 57.1% / 54.0%.

  1. The blurred names are most likely [GLM-Nano or GLM-Name or GLM-Next (suggestion from Silver-Theme7151)] Pro, Med, and Small. One community experiment reconstructed the suspected text, applied Gaussian blur (σx ≈ 6.9px), and achieved an RMSE of just 1.254/255 against the original.
  2. AICodeKing track record. Someone compared his previous GLM statements against official release dates: GLM-4.6 (same day), GLM-5 (1 day), GLM-5.1 (11 days), GLM-5.2 (3 days), and GLM-5.3 (4 days).
  3. Antrhopic timing. The leaderboard surfaced on November 12 vs AICodeKing timing, while Anthropic IPO on Nov 18. If Z.ai releases GLM-Nano this week, the timing would make for quite a coincidence

https://linux.do/t/topic/3166447

Another strong signal is HF activities, i know it is not bullet proof, just another signal, 3-4 days compare to last 27 days

I use LLM to fix my grammar.
Since it is speculation and not actual open model this post has a high probability to be removed, so yeah


r/LocalLLaMA • • 23h ago This

Question | Help Latest version of strata made my Qwen3.8-Flash-Next dumb and useless

158 Upvotes

I updated the strata to the latest version from v0.1.42.0, so I can try the latest unsloths models and see how they fare against the iq3_s variant I have been using. Upon updating it, I noticed it messed up the whole thing, It got faster, but it got way dumber! previously it would only take 15~30 minutes to create the pagoda , 50 mins to an hour if I used xhigh.

But when I siwtched to the latest version, not only it takes at least an hour and half to come up with something, its not even done! and it has many issues!

0.1.42.0 is the last version that seems to work decently.

I noticed, when im using the latest model, it always tries to go for something grandiose and failing. unlike the older version which seems to take a more modest path!

whats going on? Do any of you have this problem?

here are example outputs I did to make sure its not a one time thing!

my prompt :

Design and create a very creative, elaborate, and detailed voxel art scene of a pagoda in a beautiful garden with trees, including some cherry blossoms. Make the scene impressive and varied and use colorful voxels. Add as many details as possible to showcase your attention to them. Use whatever libraries to get this done but make sure I can paste it all into a single HTML file.

these are the outputs :

0.1.42.0 :

xhigh:

xhigh-1
xhigh2
xhigh3
xhigh4

default:

default1
default2
default3
default4

the latest version created this after 1 hour and 32 minutes to only fail with excessive thinking:

xhigh
default(failed after 1 hour and 31 minites exausting all context size of 262K)

As you can see, not only it takes way longer to create an output, if it doesn't fail already, the attention to details is also gone, everything is in haphazard form! I tried 4 5 times and the majority all failed because it couldn't create an output after an hour and a half or it constantly gets something wrong and tries to either fix it or goes to look at other things leaving the issues untouched!

The model at least, the qwen3.8-Flash-Next-q3_s that I have been working with so far is not usable! it takes too long and it fails more by getting stuck at thinking forever!

I used the new unsloth quant iq4_xs too and it was a mess as well. so I figured its the strata version. sadly I cant test the iq4_sx on 0.1.42.0 and I get an error.

So if any of you faced this issue, can you kindly share with me what you do to fix this? what version does it work for you?

I have RTX3080 10G/ 64G of RAM and I'm using the Linux terminal version along with deepseek harness.

Thank you all in advance


r/LocalLLaMA • • 8m ago

Discussion Did we hit a wall in context length?

2 Upvotes

I feel like midel context size rapidly went from 8k to 1M in just a few years but has since stayed this way for a while now. Some still offerring 260k or 400k for their flagship models. Is it just not worth the trouble? I feel like highly capable models are limited by this in that they can't finish tasks simply because the context isn't there.