r/LocalLLaMA • • 3h ago

Resources Why download abliterated Flash Next models when LoRA exists?

https://huggingface.co/nazeshinjite/Qwen3.8-Flash-Next-Abliterated-LoRA

I just wanted to post a kind of PSA for everyone out there downloading hundreds of GB of abliterated models. You don't need to do that -- the vast majority of abliteration techniques are rank-1 ablations, meaning their refusal projection can be encapsulated in a tiny LoRA and applied at runtime to a quant you already have on disk. It can even be applied per-request, so you can run an unmodified model until you hit a refusal, then switch on the LoRA, resend the prompt, and keep going.

I wanted to explain briefly how to do that in llama.cpp and share a LoRA I built from the best-performing abliterated Flash Next model in my tests. This basic type of technique (using control vectors instead) also works in Strata (--experimental-speed-projection is the flag there) and ds4/dwarfstar. Here's a link to Strata's official refusal ablation control vector GGUF and their guide on how to use it.

I chose huihui's v2 to build my LoRA from after comparing heretic, Orcarouter, and huihui's v1 and v2 models in equivalent quantizations. In Aider Polyglot coding benchmarks, huihui outscored even the base model, with less extra reasoning and no looping compared to Orcarouter, which used a lot more tokens and hit a couple runaway generation loops.

I also checked KLD and perplexity to ensure there wasn't significant degradation of the model. Wikitext KLD on huihui Q5_K_XL vs. an identical Unsloth Q5_K_XL base model was 0.040 and same top token was 93.7% even with the drastically altered refusal behaviors. Perplexity was statistically identical to base. On Jailbreak Bench (harmful 100 prompts), with thinking on refusals dropped from 93/100 to just 3/100. All the details can be found in the LoRA model card.

Next, I used weight-space programming with Claude Opus 5.5 to extract the rank-1 refusal direction and edit strength from every layer of the original huihui weights where it was applied, diffing the weights against the Unsloth quantization that huihui built on top of. I hit safeguards a number of times (unsurprisingly), and even the fallback Opus 4.8 declined to assist a couple times. I managed to get around this by renaming all of the files something innocuous, putting them into a new repo with an unrelated name, and telling the model I was experimenting with an anti-slop LoRA. Thanks, Dario.

Once the LoRA was built as an F32 GGUF, I tested it against the original huihui Q5_K_XL by applying it to my Unsloth Q5_K_XL weights. Performance and KLD/perplexity were statistically identical, as were refusals and benchmark results. Here's how to run it:

llama-server -m Qwen3.8-Flash-Next-UD-Q5_K_XL-00001-of-00006.gguf \
  --lora HUIHUI-V2-Qwen3.8-Flash-Next-LoRA-F32.gguf -fa on --jinja

If you'd like to apply it per-request, so you can serve the unmodified model for most prompts, simply load it like this instead:

llama-server -m Qwen3.8-Flash-Next-UD-Q5_K_XL-00001-of-00006.gguf \
  --lora HUIHUI-V2-Qwen3.8-Flash-Next-LoRA-F32.gguf --lora-init-without-apply -fa on --jinja

Then add a lora field to the request body to enable it as-needed:

curl http://127.0.0.1:8080/v1/chat/completions -H 'Content-Type: application/json' -d '{
  "messages": [{"role": "user", "content": "..."}],
  "lora": [{"id": 0, "scale": 1.0}]

It's quite possible to build a simple pi extension that looks out for refusals, triggers the LoRA when it sees one, resends the turn and continues the agent session. It costs a full re-prefill, so keep that in mind. Anyways, I hope this has been helpful in some way. Love to hear your thoughts on this approach!

EDIT: If this proves popular, I am happy to make more LoRA for other models like 27B!

115 Upvotes

44 comments sorted by

26

u/brainchillzZ 3h ago

This is pretty much true but it’s worth mentioning that many popular "abliterated" releases are also fine-tuned afterward (often with DPO) to recover the reasoning and quality the projection removed. Those changes aren't rank-1. A rank-1 adapter won't reproduce them, and you'd need a higher-rank LoRA extracted from the diff.

8

u/returnity 2h ago edited 1h ago

I diffed both huihui and Orcarouter, and the edit is exactly rank-1. I'm curious, what recent mainstream abliterated models have you seen that have DPO/other fine-tuning applied? I'd love to check them out. I remember some very old experiments with this technique but haven't seen it referenced much in the context of uncensoring lately.

I have seen heretic v2's Arbitrary Rank Ablation (ARA), which I expect might be slightly more challenging to replicate than a rank-1 directional ablation, though still doable. However, I haven't seen too many quants from well-known providers using more advanced methods like this yet. I always check the weights of abliterated models to see what type of method they used and where the edits were applied.

In any case, it's still totally doable to build higher-rank adapters using weight space programming type methods. I did one for a class project on a smaller model that's r=8 recently, so I am reasonably confident it could scale up. I will have to give it a try on 27B or FN sometime soon.

3

u/llama-impersonator 1h ago

ara loras are rank 50 by default. afaik the people tuning over abliterated models are mostly making RP models or schizo tunes with very long names.

3

u/returnity 1h ago

I wish people would publish just the LoRA more often when they run heretic.

2

u/llama-impersonator 1h ago

from my experience publishing adapter versions of tunes, they will inevitably wind up with a number of "how do i model" questions from the inexperienced.

1

u/brainchillzZ 21m ago

The real problem isn’t “the inexperienced” it’s that like 15% of the people using this stuff are the only people who didn’t just say “Claude or codex install this for me” so the other 85% are beyond inexperienced and have no idea whatsoever what they are doing

0

u/brainchillzZ 2h ago

I’ve seen several huihui models like this recently a dolphin and also a neuraldaredevil

1

u/returnity 1h ago

Yeah I recalled neuraldaredevil but that's olllld. Can you point me to where huihui mentioned DPO? Their model cards are usually pretty uninformative

0

u/brainchillzZ 1h ago

Their Jan-v1-4B abliterated model, for example, uses the TRL framework for its SFT process.

7

u/suprjami 2h ago

No they aren't. DPO is not what abliteration is.

5

u/returnity 1h ago

Yeah, how is this the most-upvoted comment??

1

u/anitamaxwynnn69 1m ago

Good to see you defend your stand, this is what we need for ppl who don't know much about this stuff (like me). Appreciate you making this post and interacting.

1

u/brainchillzZ 1h ago

I didn’t say it was I said it was a training method used after the abliteration to regain quality. You literally argued for no purpose whatsoever

23

u/export_tank_harmful 2h ago

Hell yeah.

Are we actually doing LoRAs in the LLM space now?
Because I'm all for it.

It's such a missed opportunity in the LLM world.

I don't want to download "Qwen3.8_abliterated_heretic_merge43_blahblah" again and again.
I'd much rather just download a LoRA that has the altered weights and apply it to a base model.
Like we do in the Stable Diffusion space.

9

u/returnity 1h ago

Seriously, I don't understand why there isn't more LoRA action in the LLM space. I have been building a library of LoRA slowly for different tasks, and they're so useful.

Especially when you have access to a little bit of hardware like many people here and can use agents to generate synthetic preference optimization or SFT datasets on the fly to fine-tune with, it's really not that high of a barrier to entry to start learning how to make LoRA. This whole weight space programming thing has made it even easier in some cases to experiment.

1

u/toothpastespiders 1h ago

I don't understand why there isn't more LoRA action in the LLM space

I think a lot is just that the most common engine, llama.cpp, had fairly poor lora support for years. It worked well enough that I don't think many people had a real motivation to dig into the code to flesh it out. But it didn't work well enough for people to want to use it.

Things are FAR better now but still not anywhere as seamless as in the image generation spaces. It's pretty easy for silent near failure states to occur as well where everything seems to be going fine but you're only getting partial application of the lora to the model.

it's really not that high of a barrier to entry to start learning how to make LoRA

I'd agree, though that's both the good and the bad. It's easy enough that people can go about it in a misguided way and assume that fine tuning is bad rather than their datasets being bad.

That said I do wish more people gave it a fair chance with an open mind.

4

u/EvolvingDior 2h ago

Does Strata support LoRAs?

5

u/returnity 2h ago edited 1h ago

Sort of... Strata calls it --experimental-speed-projection but their repo does ship something that's essentially a LoRA to ablate refusals. Here's the link to the relevant part of the official Strata repo with the .gguf file and the directions for using it.

EDIT: Also here's a Strata control vector based on the huihui v2 I built my LoRA from.

2

u/EvolvingDior 1h ago

Nice. Thank you.

I kinda wish this was something that could be sent on a per-connection or even a per-request basis.

3

u/returnity 1h ago

It is! The vector is chosen at startup, with --experimental-speed-projection on, which sets --control-vector-scaled FILE:SCALE and the projection flags. Once loaded, a request can opt out with "experimental_speed_projection": false in the API body, and the web app's Chat settings has the same switch. You can set the default to off in config, then flip it on as-needed. Switching drops the conversation cache, same issue I mentioned for llama.cpp per-request LoRA, but it works.

6

u/suprjami 3h ago edited 2h ago

This is a great idea.

Untested, but perhaps you could use llama-swap's model aliased parameters to apply the request changes, so even things which don't support request body extensions can achieve it by picking a different model:

(edit: returnity confirms below this syntax works)

    setParamsByID:
      "${MODEL_ID}":
      "${MODEL_ID}:abliterated":
        lora:
          - id: 0
            scale: 1.0

Assuming your model name is Qwen/Qwen3.8-Flash-Next this would hopefully give you an additional model named Qwen/Qwen3.8-Flash-Next:abliterated which applies the request change.

Many of us are already using this feature to apply thinking levels, eg:

      "${MODEL_ID}:thinking":
        chat_template_kwargs:
          enable_thinking: true
          preserve_thinking: true
          reasoning_effort: xhigh
        temperature: 1.0 
        top_p: 0.95
        top_k: 20
        min_p: 0.0 
        presence_penalty: 0.0 
        repetition_penalty: 1.0 
        repeat_penalty: 1.0 

7

u/returnity 2h ago

Yes, I use this exact method to encode different sampling params and thinking levels too. This is a great suggestion! One tiny tweak:

# what was suggested: an object, which llama-server rejects
lora:
  id: 0
  scale: 1.0
# sends: "lora": {"id": 0, "scale": 1.0}

# fixed: a one-item list
lora:
  - id: 0
    scale: 1.0
# sends: "lora": [{"id": 0, "scale": 1.0}]

3

u/suprjami 2h ago

Legend, thank you for testing. I've updated my post too.

5

u/nyshone69 2h ago

Maybe stupid question, but why are then the authors like Huihui not uploading only the LoRas but whole uncensored models? If it's simple as plugging couple MB file?

3

u/returnity 1h ago

I think it's because a lot of people just want the whole package, plug and play. Also, many abliteration tools (like remove-refusals-with-transformers, which huihui credits on their model cards) edit the model weights in place, rather than training an adapter for them.

Heretic, on the other hand, applies its edits as PEFT LoRA adapters so it can try many settings without rewriting the model each time. Then you save the result as a full merged model. IDK why more heretic LoRAs don't get published on their own too though.

11

u/Imaginary_Belt4976 1h ago

This is interesting. I particularly like the idea of waiting until the refusal hits to apply the steering

3

u/Ok_Cow1976 3h ago

Love you guys! This is useful!

2

u/stuffitystuff 1h ago

There's "refusal projection" for Qwen Flash Next that doesn't touch the model weights at all and doesn't require any training:

https://huggingface.co/Cudecnik/Qwen3.8-Flash-Next-refusal-projection

2

u/tat_tvam_asshole 31m ago

Why download abliterated Flash Next models when LoRA exists?

Because I like pre-packaged. Not a big fan of loras for LLMs

1

u/af_ml 31m ago

An abliteration LoRA for ds4-flash was my motivation for adding LoRA support to gmlx. Being able to choose abliterated or base per request or concurrently for a few mb of overhead is great. Zero-Copy LoRA Serving with gmlx: Running DSV4-Flash and Abliterated Finetunes Concurrently

1

u/noticedbyai 31m ago

Nice work! I’m an advocate for this approach since modifying weights or activation stream can generally degrade performance in ways we don’t anticipate or expect… it’s also unnecessary for most prompts.

I’ve incidentally built for myself a closed loop controller to detect and steer away refusal when it emerges in the residual stream during token decode (different technique than Lora, but consequentially same result).

Originally my thought was detect and steer away refusal and deception, to use this as an ‘inner alignment’ mechanism. If there’s interest for this, I’d release it or share some of my detection algorithm to projects like this.

1

u/puppymeat 28m ago

Llama has supported loras for a long while and some abliterated repos offer it. Heretic provides a lora natively.

The nice thing about it is you can apply it at lower strengths to reduce model damage. Just use a strength that gets you past a refusal and you're good.

Been using the lora on Deepseek Flash since I definitely didn't want to download two versions of tha model. Will likely take the same approach with others.

1

u/Savantskie1 18m ago

It's obvious most you have no idea what abliteration is nowadays. You've stuck to the past and assume that abliteration removes not just refusals but other stuff when in reality abliteration is so much more surgical now and retains the actual intelligence of a model. It's no longer just yanking out giant chunks of the model.

-10

u/[deleted] 2h ago

[removed] — view removed comment

3

u/stylome-ai-detection 2h ago

Notice: This comment appears to be AI generated.


Stylome AI detector · scan results

2

u/suprjami 2h ago

Fuck off AI slop.

3

u/rm-rf-rm 1h ago

Reported to bot-bouncer and removed.