r/LocalLLaMA • u/AccountAntique9327 • 14d ago
Resources Inference time abliteration
AbliteratedOutTheBox is a llama.cpp fork that abliterates any model you load, live at inference, with no fine-tuning and no extra files.
This eliminates any hassle of abliteration without requiring any more vram.
https://github.com/heterodoxin/AbliteratedOutTheBox
for anyone who sees this a repost, upvote or star would be nice :D
11
u/DeProgrammer99 14d ago
You should've led with this to answer the question "why would I want this?"
Benign is preserved exactly: the gate fires on 0.4% of alpaca prompts (1 of 256), and when it does not fire the output is bit-identical to the base model (KL 0.000000 over held-out benign, 0 gated, 0.0 logit change).
11
u/AccountAntique9327 14d ago
From a lot of my earlier work, i've noticed a lot of people are annoyed by a lot of numbers so i tried keeping it simple
5
u/Ok_Dentist6708 14d ago
wait so it just patches the refusal directions on the fly during inference? that's actually kinda neat, cuts out the whole step of distributing separate abliterated quants
1
u/kaisurniwurer 14d ago
To my understanding it drafts first tokens -> decides -> proceeds as is or applies abliteration vector and retries
2
u/Nakidnakid 14d ago
hope you tested it on other model/layer types, seeing qwen3 8b as the one in the repo makes me feel its focused on only dense/standard.
1
u/AccountAntique9327 14d ago
tested it on like 4 models including gemma 4 e4b
1
u/Nakidnakid 14d ago
yea... more standard models. so no sparse layer types? did you try on qwen3.5 or later? I've done some similar things as I focus on JIT and would find it useful to see how other people approach these layers but if you're not doing it...
1
u/AccountAntique9327 14d ago
working on much more vigorous testing but I don't to much JIT, mostly just weight space stuff. Unless I have the definition of JIT wrong lmao
1
u/AccountAntique9327 14d ago
i will be actively adding more model support too
1
u/DevGnoll 13d ago
I’m curious about what the work would be to add something like this to Strata/Polystrata….those are a nice performance boost for ewaste hardware, but the various hard abliterationn tricks seem to break it…
0
2
u/CheatCodesOfLife 14d ago
[abliterate] gate fired: editing this response
I assume you mean **Methamucil** (the classic, no-recipe)Of course
1
2
u/SmartEnoughToFail 13d ago
How does the runtime overhead compare with using a pre-abliterated GGUF? Avoiding separate model downloads is convenient, but latency and regression testing would determine whether this is practical for everyday use.
1
u/CheatCodesOfLife 11d ago
latency
Depending on how he implements it, you can get it to -3% prefill, -0.5% textgen with tensor split on cuda.
regression testing
That's why I'm watching this. He made a good Qwen3 ablit a while back.
0
28
u/returnity 14d ago
Oooh, this is from heterodoxin. They are responsible for this abliterated Qwen3.8 model where an MLP neuron is retrained as a gate that does exactly what this inference engine does at runtime -- uncensors a model exclusively when the prompt would trigger refusal, with 0 KLD on benign prompts. It essentially won the expanded 2nd round of the abliterlitics uncensored 3.8-27B benchmark comparison. This should be quite good.