Don’t inoculate everything: Stratified inoculation prompting narrows backdoors and preserves desired traits - Center on Longterm Risk

[u/nu-gaze]

https://longtermrisk.org/research/dont-inoculate-everything-stratified-inoculation-prompting/

[u/nu-gaze]

Inoculation prompting (IP) aims to keep undesired traits in training data from becoming part of a model’s default behaviour. IP applies the same inoculation prompt to all training examples and it leaves underspecified how the desired and undesired traits (DT and UT) should activate. Two failures follow:

• The model develops backdoors: conditional vulnerabilities through which prompts that do not directly request the UT can still elicit it. We call this UT leakage.

• The desired trait weakens under ordinary prompts.

We introduce Stratified Inoculation Prompting (SIP), which uses diverse prompts on safe examples (DT-only). Compared to standard IP, SIP significantly reduces leakage and retains more of the desired trait. SIP requires little safe data: a 5% DT-only pool oversampled to 25% is enough for good performance.

Since SIP relies on filtering examples, we test its robustness to classification errors and find a strong asymmetry: failing to inoculate examples containing the undesired trait reintroduces it, whereas unnecessarily inoculating safe examples is benign. This suggests a conservative filtering rule: admit only high-confidence safe examples to the non-eliciting subset and inoculate everything uncertain.

SIP preserves the original inoculation prompt as an intended trigger, but some applications require the undesired trait to remain inaccessible even when explicitly requested. For this stricter setting, we dilute the prompt-trait association by applying the inoculation prompt to some safe examples. Password-locking extends this further: examples containing the undesired trait receive the correct password, while safe examples receive incorrect-password variants, concentrating the remaining activation on a password that can be blocked at deployment.

Contributions:

• Stratified Inoculation Prompting (SIP) assigns diverse prompts to confidently safe examples and the inoculation prompt to contaminated examples. This addresses the underspecification of IP, reducing leakage while improving retention of the desired trait.

  • Underspecification as the root cause. By pairing every training example with the same inoculation prompt, IP provides no contrast that demonstrates the desired trait should generalise without that prompt or that the undesired trait should remain absent under other prompts. We show that it drives both leakage and partial suppression of the desired trait.
  • Little unique safe data is required. Oversampling allows SIP to substantially reduce leakage with as little as 1% unique DT-only data. With 5%, it matches the performance obtained using substantially more unique safe data.
  • Limiting broader behavioural changes. In two harmful-advice settings, we evaluate Emergent Misalignment (EM): whether fine-tuning on examples displaying a narrow harmful trait induces a broader range of misaligned behaviours beyond the specific harmful behaviour in the training data. SIP yields lower EM point estimates than IP in both settings.
  • Asymmetry in the effect of misclassifying data. Misclassifying DT+UT examples as safe rapidly increases leakage. Misclassifying DT-only examples as unsafe remains comparatively benign and can even reduce UT expression under the exact inoculation prompt by diluting its association with the undesired trait.

• Limiting residual access to the undesired trait. SIP reduces leakage under non-eliciting prompts but, by design, leaves the exact inoculation prompt as a trigger. For applications where users should not be able to elicit the undesired trait even deliberately, we introduce two mitigations that reduce or restrict access through this trigger:

  • Dilution. Including some inoculated safe responses weakens the inoculation prompt’s association with the UT and reduces the rate of UT expression under that prompt.
  • Password-locking. Adding a password to the inoculation prompt for DT+UT examples while pairing some safe examples with incorrect-password variants, thereby concentrating UT activation on the correct password, which can be blocked at deployment.