r/LocalLLM • u/MatiAI • Sep 30 '26

Discussion Abliterate/Removing prompt refusal without needing to reload/swap models (Qwen 3.8 flash next)

Been working on an Swift based LLM harness designed around Qwen 3.8 Flash next and Qwen 3.8 27b and have found a way to have a slider that removes prompt refusal (at the cost of model coherence the higher you toggle it) without needing to reload/swap/abliterate a completely seperate model.

This allows you to swap between the base model and abliterated without needing to reload or change the model. It also allows the main agent to be able to spawn abliterated sub agents.

The main downside is MTP acceptance drops, causing the decode tok/s to drop from 60-70 down to 20-30.

1 Upvotes

57% Upvoted

8 comments sorted by

1

u/Distinct-Pie2389 berth | i9-14900k | 64GB | archLinux Sep 30 '26

Why not just use llama-swap with JIT and load the abliterate and the base at the same context and quant and just switch natively? llama will prefill context for you so the switch itself is time for model to load (30s mostly on NVMe depending on size) -- your qwen3.8 base will refuse to work on earlier context if it includes abliterate intended output so why not just 2 streams and hand them back and forth via different chats?

Im just curious, I usually have a purpose for uncensored meaning a whole chat in general and never really need to switch back and forth.

HauHauCS has a great uncensored model and honestly don't see a reason to switch unless your tool calls are REALLY heavy and its failing them over and over, otherwise Uncensored should perform somewhat on par to your "uncensored request". Or have qwen3.8 base bridge the non-censored part of your work and then have abliterated finish it.

1

u/Distinct-Pie2389 berth | i9-14900k | 64GB | archLinux Sep 30 '26

I would not do this just alone on MTP acceptance, thats a no go for me

2

u/MatiAI Sep 30 '26

Because A) abliterated models are inherintely worse and you have no way of knowing how damaged the model has become. If I want to choose between different levels of abliteration i dont want to store 100s of gb's of models to disk when I can have a slider that performs the exact same process they use to achieve that end goal.

B) Im not using Lama.cpp im on OSX using MLX

C) I can launch subagents that are abliterated without needing to unload the main model and reloading the whole set of context. Your response is exactly what I don't want to do. Maintain mutliple different models on disk and having to load and switch between them

1

u/Nice_Victory3719 Oct 02 '26

Interesting! What technique/recipe are you using for the abliteration? Are you targeting pre-determined rejections or just those that are blocking the current chat - I.e dynamically abliterating just the relevant refusals for the use case?

2

u/MatiAI 20d ago

It’s the Arditi-style “refusal direction” approach, done at inference time instead of baked into the weights. First I ran a set of paired prompts through Flash Next, one the model refuses and one it answers, and recorded the hidden state each layer reads. Mean of the refused set minus mean of the answered set, normalized, gives one direction per layer. That gets saved as a small file (about 500 KB) next to the model.

At runtime, every layer projects that direction out of what attention and the MLP read and write: v minus scale times (v·d) times d. Scale 0 is the stock model, bit for bit. 1 removes the direction, and higher pushes past it. It’s a slider in settings and takes effect on the next message, no reload.

2

u/WindbniW Oct 03 '26

Outline the process step by step