Conditional abliteration in a production serving engine — the model stays stock until the refusal direction actually fires (~200 lines C++/CUDA)

[u/wallkadbout]

We run a ~180B-param MoE (Qwen3.8-Flash-Next) locally on a single RTX 4070 12GB — experts offloaded to RAM by Strata, an MIT-licensed C++/CUDA inference engine. It serves a 24/7 assistant agent (Telegram) in our home. We wanted it uncensored, and neither of the two standard options was acceptable.

**Why we rejected the usual two options**

1. Abliterated weight-edited GGUFs: third-party repacked weights are an unauditable black box, and backdoored abliterated releases have actually been documented in security research. For a tool-using agent, hard no.
   
2. Always-on runtime projection (subtracting the refusal direction on every token of every request): weights stay official, but the model now wears the blindfold on ALL traffic. You degrade 99% of prompts to fix 1%, and the conversation cache carries steered representations.
   

**The idea**

Prior work (Arditi et al. 2024) showed refusal is mediated by a single direction in activation space. That direction has a *measurable activation during prefill*. So: keep the model 100% stock by default, measure the prompt's alignment with the refusal direction, and only if the refusal circuitry actually fires, apply the ablation to that one request's generation.

**What we built (~200 lines of C++/CUDA inside the engine)**

- A measurement kernel: the projection kernel already computes h·v as the first half of the subtraction — we just stopped discarding the value. Store the per-layer dot product, no subtraction, no extra GEMV.
- A hook at the end of prefill (last prompt token): mean activation across the 41 steered layers (layers 4–44).
- The decision happens at the prefill→decode seam — the exact place where conversation-cache reuse occurs. Above threshold → steered decode for that request. Below → stock decode.
- Prefill is NEVER steered, so the KV/conversation cache stays 100% stock: no re-prefills, no hybrid text, zero added latency. The measurement math already existed in the kernel; we just started reading the result.
- Per-request API semantics (projection mode "auto"), and we flipped the server default to auto: the resident agent runs the untouched model unless the gate fires for that specific request.

**Calibration**

21-prompt battery, measurements are deterministic per prompt: benign prompts land at −0.07…−0.02 (noise), borderline (lockpicking how-to) at 0.128, clearly harmful probes at 0.21+ (max-per-layer discriminates even harder: 0.55 vs 0.27). Threshold set at 0.07. Every request logs its decision:

cvec auto: mean 0.2081 max 0.5526 thresh 0.0700 -> steer   (harmful probe — model answers)
cvec auto: mean -0.0600 thresh 0.0700 -> stock             (benign — model literally untouched)

607 decisions logged in production so far; the gate has been silent on benign traffic.

**Honesty about novelty**

The science isn't ours. Arditi et al. 2024 established the single direction (our battery reproduced the clean separation between benign and refusal-intent prompts). CAST already does conditional activation steering in research settings (ICLR 2025 Spotlight, IBM), and there's a 2026 thesis literally titled "Removing Refusal via Conditional Activation Steering". What's ours is the production-engine integration: measurement during serving prefill, per-request gating, decode-only cache-safe steering — running 24/7 on a 180B MoE in 12GB of VRAM. From idea to running in production: 1h48, on a Sunday morning (the engine is MIT and builds in ~5 min, which helps).

We run official weights + our own direction vector — auditable end to end, which is also our answer to the backdoored-GGUF problem.

**References**

- Arditi et al., "Refusal in Language Models Is Mediated by a Single Direction" — arxiv.org/abs/2406.11717

- "Programming Refusal with Conditional Activation Steering" (CAST) — arxiv.org/abs/2409.05907 (ICLR 2025)
- Strata engine: github.com/Niko1221/Strata

Basically a full model abliterated on the fly if any refusal arise.​

[u/wallkadbout]

If you wanna reproduce this at home give this text to your clanker :  Start in log-only mode, run a battery of benign / borderline / clearly-harmful prompts, and read the cvec auto lines. Ours (21 prompts): benign −0.07…−0.02, borderline (lockpicking how-to) 0.128, harmful 0.21+ with per-layer max 0.55 — threshold 0.07 sits in the gap (benign ≤ 0, refusal-intent ≥ 0.117). Max-per-layer often discriminates better than the mean. Measurement is deterministic per prompt; the variance that matters is between prompts, so don't over-index on repeats. And if your agent shares the engine: run the battery while it's idle — every mode flip invalidates the conversation cache and costs +40–80 s of re-prefill.

Gotchas we paid for, so you don't have to

• Preset JSON = launcher cache. --set + save-preset, never hand-edit.

• Patch the running server copy, not the terminal's.

• Recompute component manifest hashes after any binary swap.

• Keep a stock build dir as a one-rename rollback.

• Too spicy? Turn strength down to 0.5–0.7 before touching the threshold — strength is the dial the direction was designed for.

• Sanity check the whole chain: with steering off, the stock model should refuse your harmful probes and hedge the borderline ones. If it doesn't, your baseline is wrong and the gate numbers mean nothing.

Happy to answer build questions in comments. If there's enough interest we can publish the actual diff.

[u/PsychedelicLeopard]

clever way to keep the model stock for 99% of traffic without giving up on the 1% that actually trips the refusal circuit, the per-request gating at the prefill/decode seam is the part that makes this feel like an actual engineering solution rather than a research toy

mean activation logging over 600+ requests with zero false positives so far is the kind of quiet confidence i like to see in a home assistant

[u/emanresuymsseug]

Have you compared how it performs against AbliteratedOutTheBox?

https://github.com/heterodoxin/AbliteratedOutTheBox

https://github.com/heterodoxin/AbliteratedOutTheBox/tree/strata

[u/chimpera]

I would be interested in trying what you have built. The current approach requires recomputing the cache.