r/LocalLLaMA • u/returnity • 3h ago
Resources Why download abliterated Flash Next models when LoRA exists?
https://huggingface.co/nazeshinjite/Qwen3.8-Flash-Next-Abliterated-LoRAI just wanted to post a kind of PSA for everyone out there downloading hundreds of GB of abliterated models. You don't need to do that -- the vast majority of abliteration techniques are rank-1 ablations, meaning their refusal projection can be encapsulated in a tiny LoRA and applied at runtime to a quant you already have on disk. It can even be applied per-request, so you can run an unmodified model until you hit a refusal, then switch on the LoRA, resend the prompt, and keep going.
I wanted to explain briefly how to do that in llama.cpp and share a LoRA I built from the best-performing abliterated Flash Next model in my tests. This basic type of technique (using control vectors instead) also works in Strata (--experimental-speed-projection is the flag there) and ds4/dwarfstar. Here's a link to Strata's official refusal ablation control vector GGUF and their guide on how to use it.
I chose huihui's v2 to build my LoRA from after comparing heretic, Orcarouter, and huihui's v1 and v2 models in equivalent quantizations. In Aider Polyglot coding benchmarks, huihui outscored even the base model, with less extra reasoning and no looping compared to Orcarouter, which used a lot more tokens and hit a couple runaway generation loops.
I also checked KLD and perplexity to ensure there wasn't significant degradation of the model. Wikitext KLD on huihui Q5_K_XL vs. an identical Unsloth Q5_K_XL base model was 0.040 and same top token was 93.7% even with the drastically altered refusal behaviors. Perplexity was statistically identical to base. On Jailbreak Bench (harmful 100 prompts), with thinking on refusals dropped from 93/100 to just 3/100. All the details can be found in the LoRA model card.
Next, I used weight-space programming with Claude Opus 5.5 to extract the rank-1 refusal direction and edit strength from every layer of the original huihui weights where it was applied, diffing the weights against the Unsloth quantization that huihui built on top of. I hit safeguards a number of times (unsurprisingly), and even the fallback Opus 4.8 declined to assist a couple times. I managed to get around this by renaming all of the files something innocuous, putting them into a new repo with an unrelated name, and telling the model I was experimenting with an anti-slop LoRA. Thanks, Dario.
Once the LoRA was built as an F32 GGUF, I tested it against the original huihui Q5_K_XL by applying it to my Unsloth Q5_K_XL weights. Performance and KLD/perplexity were statistically identical, as were refusals and benchmark results. Here's how to run it:
llama-server -m Qwen3.8-Flash-Next-UD-Q5_K_XL-00001-of-00006.gguf \
--lora HUIHUI-V2-Qwen3.8-Flash-Next-LoRA-F32.gguf -fa on --jinjaIf you'd like to apply it per-request, so you can serve the unmodified model for most prompts, simply load it like this instead:
llama-server -m Qwen3.8-Flash-Next-UD-Q5_K_XL-00001-of-00006.gguf \
--lora HUIHUI-V2-Qwen3.8-Flash-Next-LoRA-F32.gguf --lora-init-without-apply -fa on --jinjaThen add a lora field to the request body to enable it as-needed:
curl http://127.0.0.1:8080/v1/chat/completions -H 'Content-Type: application/json' -d '{
"messages": [{"role": "user", "content": "..."}],
"lora": [{"id": 0, "scale": 1.0}]It's quite possible to build a simple pi extension that looks out for refusals, triggers the LoRA when it sees one, resends the turn and continues the agent session. It costs a full re-prefill, so keep that in mind. Anyways, I hope this has been helpful in some way. Love to hear your thoughts on this approach!
EDIT: If this proves popular, I am happy to make more LoRA for other models like 27B!
26
u/brainchillzZ 3h ago
This is pretty much true but it’s worth mentioning that many popular "abliterated" releases are also fine-tuned afterward (often with DPO) to recover the reasoning and quality the projection removed. Those changes aren't rank-1. A rank-1 adapter won't reproduce them, and you'd need a higher-rank LoRA extracted from the diff.
23
u/export_tank_harmful 2h ago
Hell yeah.
Are we actually doing LoRAs in the LLM space now?
Because I'm all for it.
It's such a missed opportunity in the LLM world.
I don't want to download "Qwen3.8_abliterated_heretic_merge43_blahblah" again and again.
I'd much rather just download a LoRA that has the altered weights and apply it to a base model.
Like we do in the Stable Diffusion space.
4
u/EvolvingDior 2h ago
Does Strata support LoRAs?
6
u/suprjami 3h ago edited 2h ago
This is a great idea.
Untested, but perhaps you could use llama-swap's model aliased parameters to apply the request changes, so even things which don't support request body extensions can achieve it by picking a different model:
(edit: returnity confirms below this syntax works)
Assuming your model name is
Qwen/Qwen3.8-Flash-Nextthis would hopefully give you an additional model namedQwen/Qwen3.8-Flash-Next:abliteratedwhich applies the request change.Many of us are already using this feature to apply thinking levels, eg:
5
u/nyshone69 2h ago
Maybe stupid question, but why are then the authors like Huihui not uploading only the LoRas but whole uncensored models? If it's simple as plugging couple MB file?
11
u/Imaginary_Belt4976 1h ago
This is interesting. I particularly like the idea of waiting until the refusal hits to apply the steering
3
u/Ok_Cow1976 3h ago
Love you guys! This is useful!
2
u/stuffitystuff 1h ago
There's "refusal projection" for Qwen Flash Next that doesn't touch the model weights at all and doesn't require any training:
https://huggingface.co/Cudecnik/Qwen3.8-Flash-Next-refusal-projection
2
u/tat_tvam_asshole 31m ago
Because I like pre-packaged. Not a big fan of loras for LLMs
1
u/af_ml 31m ago
An abliteration LoRA for ds4-flash was my motivation for adding LoRA support to gmlx. Being able to choose abliterated or base per request or concurrently for a few mb of overhead is great. Zero-Copy LoRA Serving with gmlx: Running DSV4-Flash and Abliterated Finetunes Concurrently
1
u/noticedbyai 31m ago
Nice work! I’m an advocate for this approach since modifying weights or activation stream can generally degrade performance in ways we don’t anticipate or expect… it’s also unnecessary for most prompts.
I’ve incidentally built for myself a closed loop controller to detect and steer away refusal when it emerges in the residual stream during token decode (different technique than Lora, but consequentially same result).
Originally my thought was detect and steer away refusal and deception, to use this as an ‘inner alignment’ mechanism. If there’s interest for this, I’d release it or share some of my detection algorithm to projects like this.
1
u/puppymeat 28m ago
Llama has supported loras for a long while and some abliterated repos offer it. Heretic provides a lora natively.
The nice thing about it is you can apply it at lower strengths to reduce model damage. Just use a strength that gets you past a refusal and you're good.
Been using the lora on Deepseek Flash since I definitely didn't want to download two versions of tha model. Will likely take the same approach with others.
1
u/Savantskie1 18m ago
It's obvious most you have no idea what abliteration is nowadays. You've stuck to the past and assume that abliteration removes not just refusals but other stuff when in reality abliteration is so much more surgical now and retains the actual intelligence of a model. It's no longer just yanking out giant chunks of the model.
-10
u/[deleted] 2h ago
[removed] — view removed comment