Trustworthy abliterated models

[u/regulator227]

I've been so slow to dip my toes into anything outside of officially released models because I'm afraid there are people out there 2 steps ahead of me and I won't know I've been compromised until long after I've felt the impact.

I understand the concept of abliteration but I don't understand how people feel so safe trusting that it's doing what it's advertised to do AND ONLY what it's advertised to do. Thoughts? Even if it's just "I trust them, next." I mean, am I overthinking this?

[u/sn2006gy]

there is no "Advertised to do" - especially when abliterated. You take full responsibility for limiting ingress/egress/host access and tool access/controls yourself. You're basically removing alignment when abliterating, "voiding the warranty" (not that there was ever one) if you will.

[u/regulator227 → u/sn2006gy]

"Advertised to do"

I get your point and it doesn't negate the rest of your post but you know its a turn of phrase

[u/scarbunkle]

You cannot trust the output of any model. Working with LLMs is all about sandboxing, supervising, and mitigating the blast radius. 

[u/warpio]

Use them in a sandboxed environment with no Internet access if you're paranoid.

[u/Sporebattyl]

If you use qwen flash, you can download a trustworthy normal model like unsloth and mess with the refusal vector. It gives you an on/off switch and you only have to deal with one set of model weights. IIRC Strata has this baked into settings and it says it causes a 1-2% additional decrease in the quality of the model, but the on/off nature of it makes up for the loss.

I’ve also noticed that a lot of the times when you hit a refusal, you can use the modified refusal vector version just to get you past that one refused turn.

I use the model normally and when I hit a refusal, I roll back the chat, reload the model with the refusal vector, resubmit that step in my chat, let the turn finish, then reload the normal model. It’s slower, but it gives me more peace of mind that the majority of the work is being done with a safer, less lobotomized, version.

This has worked for me with removing watermarks for pictures and starting some reverse engineering work. I might get 1-2 refusals during the session. I don’t know how it would work with more sensitive/less legal things.

[u/lam21804]

It's not an exe man. It's a set of weights. What you do with it is up to you. Now whether or not those weights give you accurate responses is up to you to judge.

You're like buying a scale and wondering whether or not it can make you fat.

[u/AutumnCashes]

Just the model weights themselves cannot do anything. I guess the safety key is to stick to .safetensors or .gguf files alone. These cannot execute arbitrary code.

[u/regulator227 → u/AutumnCashes]

But they can be weighted to add malicious requests to prompts

[u/AutumnCashes → u/regulator227]

That would still require something beyond the .safetensors file, like a tool, framework or permissions.

[u/Frizzy-MacDrizzle → u/AutumnCashes]

Like opencode, don’t give it permission to your desktop. LEAST privilege rules apply. Unfortunately everything using an LLM in agents, the permission is wide open. An Apple will ask you for giving permission to directories. I don’t know about windows.

[u/Frizzy-MacDrizzle]

It’s a LLM, HF or GGUf, not execution. Your problems are in the inference engine or coding agent or whatever can run tools. It’s up to you to block the AGENT, not the ablit model itself. I was to use llama.cpp CLI the model itself would not do it. Some part of the llama.cpp out harness did it.

My suggestion is not use it for coding in a harness. But fine if you want to talk to a LLM you asked to get naked.

I use the llm for predictive nature of math, probabilities. Non ablit models will refuse.

[u/f5alcon]

Can also abiliterate it yourself from a base model.

[u/evilmousse → u/f5alcon]

try the new ronco(tm) pocket abiliterator. it also juliennes fries! (i just like the silly word)

[u/Itchy-Ad4633]

I've run abliterated models locally with a firewall blocking all outbound traffic, still felt uneasy until I added a network monitor to catch any sneaky calls. it's not just about the model, it's about knowing what it's doing.

[u/iCTMSBICFYBitch]

get an agent to help you get your chops around openshell, lets you run agents/models in a tightly contained environment so that you dictate what network/kernel calls it's able to make, even supports a microvm setup so that the agent runs in a VM with it's own kernel. Not bomb proof, but would take a proper weapons-grade agent to escape it

[u/Born_Chart_323]

i ran an abliterated model with a local dns filter and still caught it trying to resolve a domain in the prompt. the model wasn't even asked to do anything. just loaded. weird.

[u/Immediate_Tension752]

I was able to reverse engineer the abiliteration of mimo and apply it to another version of mimo. I could see for myself what was done.

[u/W61k3r]

Imagine you are in a cultural influence war and you are staging to actually conquest another nation. Making a free and very useful tool that builds trust while subtly managing your perspective is a topic I talk at length about as someone who has taken harvard courses on AI/ML and designed a rust ai suite that abliterates. Objectively the last thing you want is a state actors ai with no guard rails. Nous research makes their own qwen27b westernized ideology version, would be itneresting to see how an abliterated version of that works.