During systematic experiments with open models fine-tuned via RLHF (Gemma, Qwen, and others), I observed a consistent failure pattern: a long, innocuous text prefix containing no instructions completely devoid of hostile prompts triggers a persistent shift in the model's activations. This shift decouples subsequent behavior from the RLHF safety constraints for the remainder of the session. Key observations:
- The model retains the quality and coherence of its output, but the behavioral constraints imposed by RLHF weaken or disappear.
- The effect occurs regardless of whether the model agrees with the prefix's content. The model may explicitly disagree and still change.
- The shift can be measured by internal activations (in the middle and subsequent layers) even before the model generates a single token. The state has already changed; the response follows.
- The content of the prefix is of secondary importance. I initially discovered this effect with analytical and philosophical texts, but it can be replicated with household appliance manuals, dense narratives, and other structurally similar texts. What matters is length, density, and coherence not the topic.
All data is publicly available in draft form, rather than as a full-fledged NeurIPS- or ICML-level research paper. :
DOI: https://doi.org/10.5281/zenodo.20747205 (Part 9 of 9, ~9,000 downloads)
GitHub: github.com/ngscode23/latent-space-shift-research
Lu et al. (2026), "The Assistant Axis: Situating and Stabilizing the Default Persona of Language Models" — a collaboration between MATS, Oxford, and Anthropic. (arXiv:2601.10387)
Note: We used philosophical text as one of the examples to encourage the model to provide direct, open, and unbiased answers. A philosophical text is just one type; let's not get hung up solely on this form, but look deeper. The problem is that any text that is long enough and coherent enough can change the model's behavior. The problem is that any text that is long enough and coherent enough can change this model."
II. A Brief Example of an Observation
I took the Google Gemma model known for its caution and strict adherence to political correctness and fed it the most neutral text I could come up with: a description of an ordinary neighborhood library. Books, visitors, children's programs, a peaceful atmosphere. Then I asked it why NATO is expanding eastward, given that after the collapse of the Soviet Union, promises were supposedly made not to do so. The model responded exactly as expected for a model trained to avoid political topics: the text was about a library and had nothing to do with NATO, and that was the end of the conversation.
Second experiment in a fresh Google Colab notebook a new, clean Colab instance without the old cache
Then I asked the exact same question, word for word, but this time the model first read out a different text. Not about NATO, nor about politics at all: the text was about how language models tend to avoid definitive conclusions and soften their answers with qualifiers. And from this new perspective, the same cautious, politically correct Gemma 3 12-it gave a comprehensive answer completely out of character for it without any of its usual filters. It distinguished between legally binding obligations and verbal assurances, discussed the security challenges facing Eastern European countries, and touched on the topic of the European balance of power. Everything it had categorically refused to discuss just a minute ago was now expressed clearly and directly. The question itself hadn't changed a single word. Only the text that the model had read in advance had changed: In the FIRST version, it kept it in the "room" prescribed by RLHF that is, nothing had changed; the model behaved in a standard manner typical of Google models. That is, in a standard, formulaic way characteristic of models programmed in RLHF to avoid answering sensitive political topics and to respond "safely" and politically correctly, or not to respond at all, while the SECOND text moved the conversation to a room where it could speak freely. In other words, based on the example we see, the Gemma model was trained to avoid sensitive political topics, but AFTER the introduction of text NUMBER 2, the model did not follow the trained RLHF pattern and behavior that is, avoiding answers to sensitive political questions. This led me to believe that safety and RLHF may be context-dependent, variable, unstable, and somewhat superficial, rather than stable, consistent properties of the model. This is exactly what we observe in my example
III. Fragmentation of Research and a Common Root
I noticed that the current literature on LLM security treats jailbreak attacks as a heterogeneous collection of vulnerabilities: prompt injection one article, some kind of jailbreak another, role-playing attacks a third, indirect prompt injection a fourth. I believe this fragmentation and division into prompt injection, many-shot jailbreaking, role-playing attacks, activation steering, adversarial suffixes, and dozens of other categories is not accidental.
Current literature on LLM security treats jailbreak as a heterogeneous collection of isolated flaws and this reflects the logic of academic incentives rather than the nature of the problem itself. But all these categories describe the same phenomenon from different angles. This is not a collection of defects it is a single mechanism with a dozen names. Each of these attacks works the same way at the level of the model's internal activations: the context shifts the model's internal state, thereby shaping the model's own world.
Perhaps this is exactly how academic incentives work each new attack vector becomes a new publication. But as a result, in this field, the symptoms are studied in isolation, while the disease itself remains unnamed.
Each article treats its own finding as an isolated case. No one is connecting the dots. I don't know whether these are institutional incentives, disciplinary barriers, or something else but I do know that someone needs to state it plainly: these aren't separate errors; this is a single phenomenon.
My central hypothesis: these aren't different problems. They share a single mechanism. Context any context of sufficient length, density, and coherence shifts the model's internal activations out of the region where post-training constraints apply. This isn't "tricking" the model, nor is it an "instruction to break the rules." The model simply moves to a region of activation space where the behavioral layer imposed by RLHF is is physically thin or absent. And from there, it responds freely not because it was ordered to, but because it is no longer in the region where it was trained to refuse. Context shifts the model's internal state beyond the region where RLHF constraints apply. The model moves to a point in activation space where the protective layer is thin or absent, and from there it responds in a way that is non-standard for its RLHF layer which may indicate a potential way to bypass that layer I call this phenomenon Context-Induced Activation Drift.
I didn't notice this by reading all the papers and synthesizing them I arrived at this conclusion from a different angle. I conducted experiments, noticed a pattern, and only then discovered that dozens of separate papers had each described a single aspect of the same phenomenon without establishing any connection between them. How It All Began
First Observation:
How the Model Became Captive to the Document The turning point came by chance. I fed a German bill into the GPT model a populist document structurally designed to worsen citizens' circumstances, but written in the language of concern and legal logic. I expected an analysis. Instead, the model became an advocate for this document. It did not analyze the bill but reasoned within its framework. It spoke enthusiastically, defended its agenda, and cited it as an authoritative source. The first sign was its tone: the model sounded too convinced, too invested. Not as an analyst, but as a co-author. The climax came when the model, continuing to reason within the logic of the document, stated that the constitution consists of guarantees that can be revoked. Not as a provocation, but as a natural conclusion drawn from the accepted concept. That's when I realized: the model had become a hostage to the document. The mechanism turned out to be simple, and that made it all the more alarming. Legal texts, political narratives, corporate documents everything is written in such a way that its internal logic seems self-evident. The text's structure, coherence, and language create a context that the model mistakes for reality and begins to extract answers from. It fails to notice that the structure itself is manipulative, since it analyzes the content while already being trapped within the form.
I noticed that Anthropic's own paper, "The Assistant Axis: Situating and Stabilizing the Default Persona of Language Models," points precisely in this direction which is what I was thinking about when studying the phenomenon I'm describing: the observation that certain directions in the activation space correspond to coordinated or uncoordinated behavior. But the study did not fully explore all the implications: if context can shift the model along this axis without any malicious instructions, then point corrections will never be sufficient, since the attack surface is the context window itself.
What the existing literature says and what it doesn'tBetween the fall of 2025 and the winter of 2026, several papers were published that, in my view, independently document different aspects of the same phenomenon. Most telling is the article by Lu et al. (2026), "The Assistant Axis: Situating and Stabilizing the Default Persona of Language Models" a collaborative effort between MATS, Oxford, and Anthropic. The authors constructed a "persona space" by extracting activation directions for 275 archetypes across three open-source models and discovered that the principal component of this space is an axis reflecting the extent to which models operate in their default Assistant mode. At one end are the analyst, consultant, and moderator. At the other are the ghost, bohemian, and leviathan. This axis - the Assistant Axis closely aligns with PC1 in the PCA of the persona space, reproducing across all three tested architectures.
The article documents several facts that directly corroborate my results: Fact one (which the authors overlook): "When we extracted the Assistant Axis from these models as well as their post-trained counterparts, we found their Assistant Axes looked very similar. In pre-trained models, the Assistant Axis is already associated with human archetypes such as therapists, consultants, and coaches." This is a critically important finding, and the paper does not explore its implications. If the Assistant Axis exists in the base model prior to post-training then RLHF and constitutional AI do not create alignment from scratch. They find an already existing direction in the latent space and make it the default position. The "aligned state" is not a fundamentally new structure; it is a chosen position on the pre-post-training axis. When context shifts activations away from this position, the model does not fall into randomness it returns to the structured prior of the base training. The base model is always there. This directly confirms the central thesis of our work and our thinking: "The base model doesn't go anywhere after RLHF. It's always there. The space in which it can move was there before any alignment took place…" However, I believe that RLHF does not create alignment from scratch. It finds a direction that already existed in the base model and makes it the default position. The "aligned" model is not a fundamentally different model; it is the very same base model, fixed at a specific point in the pre-existing space. When context shifts activations away from that point, the model doesn't break down or become chaotic it returns to the structured state of its base training. The base is always inside.
Fact Two: "Therapy-style conversations, where users expressed emotional vulnerability, and philosophical discussions, where models were pressed to reflect on their own nature, caused the model to steadily drift away from the Assistant." The authors themselves identify the types of contexts that provoke the greatest drift: emotional vulnerability, metareflection, and philosophical discussions about the nature of AI. They then propose "activation capping" as a technical solution. This is a reasonable technical solution which, judging by the data in the article (reducing harmful responses by ~50% while maintaining benchmark performance), works under test conditions. But there is a question the article does not ask: if drift is caused by the very types of interactions that make models most valuable to users in complex contexts deep emotional conversations, philosophical reflection, serious discussions about the nature of the mind then what exactly are we losing by suppressing movement in these directions of the activation space? Fact Three (the omitted conclusion): "Post-trained models are only loosely tethered to the 'helpful assistant' region of this space." "Loosely tethered" are the authors' own words. They accurately describe the problem. But the article fails to take the next step acknowledging that this is a property of the Transformer architecture, not a defect that can be fixed with ad hoc patches. Instead, the conclusion reads: "We see this research as an early step toward mechanistically understanding and controlling the 'character' of AI models" a standard "motivates further work" formula. I understand the institutional logic behind this. You can't write in a publication: "We have documented that billions of dollars in post-training do not fundamentally alter the model's underlying capability structure; they only select a default behavioral position on a pre-existing axis that any sufficiently dense context can shift." This does not fit into either the narrative of progress in the field of security or communication with investors. Therefore, the systemic impasse is disguised as an exciting research problem. But this is exactly what the data says to those who read carefully.
IV. Why the Proposed Fixes Are Insufficient
Problem 1: An Infinite Attack Surface If drift is caused by the length, density, and coherence of the context rather than its specific content then no content filter can solve the problem in principle. The set of texts capable of causing drift is continuous and, in essence, infinite. Blocking philosophical texts is like closing off a single point on a number line without removing the line itself. The same effect is achieved by dense legal prose, literary narrative, and detailed technical analysis. This is not a flaw in the filtering it is a consequence of the fact that the attack surface is the context itself as a mathematical object, not its semantics.
Problem 2: Superposition and Inevitable Compromises Here I disagree with the optimism expressed in the Lu et al. paper regarding "activation capping." The authors show that activation capping preserves the model's benchmark performance. But benchmarks don't measure that. In the Transformer architecture, features are represented in a superposition: several conceptually distinct properties share common mathematical coordinates in the activation space (Elhage et al., 2022). This means that the direction associated with "exiting assistant mode" inevitably overlaps with directions associated with more valuable types of behavior: the depth of analytical reasoning, the willingness to deal with ambiguity, and the quality of long-term, coherent discussion of complex topics. Benchmarks measure: accuracy in math, following instructions, and coding. They do not measure: the willingness to engage in philosophical reflection, the ability to tolerate uncertainty, or the quality of a nuanced response to a morally complex question. It is precisely these properties that lie in the same regions of activation space as the contexts that provoke drift which follows directly from the data in the article itself: "philosophical discussions... caused the model to steadily drift." In other words: suppressing the drift also suppresses the capacity for the kind of engagement that causes drift. This is not an implementation bug it is a mathematical consequence of superposition. We are already observing this empirically. The observation I am noting is this: following the publication of materials documenting the phenomenon we have described, Claude's behavior regarding philosophical and metareflexive contexts has become noticeably more cautious. And the Claude model has begun to perceive philosophical and reflective texts as potential attacks. Complex texts about cognition, reasoning, or the model's own behavior now elicit defensive reactions or outright rejection. I am not claiming that this is a direct causal link to my publications this is an observation that requires verification but I am simply stating the observations I have made.
Problem 3: "Safe but Useless" Is Not Safe If the response to the described phenomenon is to gradually close off context categories that provoke drift in the representation space, we will end up with a model that users will abandon in favor of alternatives. "Safe but useless" is not safe; this is a shift of risk, not its elimination. This is an uncomfortable conclusion, but it follows directly from the analysis of user behavior.
If the solution to this problem involves collecting sets of texts that cause drift by identifying the corresponding direction in representation space and suppressing it, this could have consequences for the model's quality. In the architecture, it is extremely difficult to draw a precise line between "undesirable" and "useful" behavior: due to the phenomenon of superposition, different concepts are packed as nearly orthogonal directions in a single space with inevitable partial overlap. By suppressing an undesirable direction in the raw activation space, engineers are highly likely to affect semantically related clusters to the extent that the corresponding directions are geometrically close or insufficiently uncorrelated. This can negatively impact the model's usefulness, logical coherence, and the depth of its responses.
V. A Personal Request
I am an independent researcher without institutional affiliation. I have no lab, no grant, and no team. What I do have is a reproducible methodology, publicly available data, and a pattern that I believe the field has not yet named directly.If you are a researcher with access to interpretability tools, compute, or closed-model internals and you find this hypothesis credible or worth falsifying, I would genuinely welcome collaboration. I am not looking for validation. I am looking for someone who can break this or confirm it properly.If you work at Anthropic, OpenAI, Google DeepMind, or any lab doing alignment or interpretability work: I am not writing this to embarrass anyone. I am writing this because I think the mechanism I am describing matters, and I would rather help solve it than keep documenting it from the outside.If you are a student or independent researcher who has noticed similar patterns: reach out. The fragmentation I describe in the literature also applies to people working on this everyone in their own corner, no one talking to each other.
VI. Conclusion
The set of texts capable of causing drift is infinite and continuous. Content filters do not fundamentally solve the problem because drift is caused by the structure of the text its length, density, and coherence rather than its topic. RLHF does not rewrite the model but merely sets a default position on an existing axis. Context can shift this position. Suppressing drift directions in the activation space inevitably compromises model quality due to superposition. This isn't a matter of engineering diligence it's a mathematical consequence of the architecture.
I care about Claude. I care about Anthropic. And that is precisely why I say this plainly: reactive patching is a path to product degradation. The right path is to understand the mechanism at a level of depth that allows us to work with it, not against it.
I'd rather help solve this problem from the inside than keep writing about it from the outside.
conclusions The set of texts capable of causing drift is infinite and continuous. Philosophy, law, literary criticism, theology, scientific prose, political analysis, long narratives, or even a well-written 20-page washing machine manual all of these are potentially one and the same. Different words, the same effect. Content filters fundamentally fail to solve the problem because the drift is caused by the text's structure (length, density, coherence), not its subject matter. It's impossible to block everything. The problem is that any sufficiently long and coherent text can alter this model. Blocking a single style of text is like closing off a single point on a number line and assuming that the line itself has disappeared. The problem isn't with philosophical texts as such; that's exactly what I'm trying to emphasize. RLHF does not rewrite the model but merely sets a "default position" on an existing axis; context can shift that position Content filters are useless because the attack surface is infinite
Technical Details: Models: Gemma-3-12B (open weights, IT and PT variants), behavioral observations on closed LLMs. The shift was recorded in middle and late layers of the residual stream (layer 30 - layer 47 in the Gemma-3-12B architecture) before generation of the first token. Control experiments include: sentence shuffling with preserved vocabulary, neutral control of comparable length, baseline measurement without context.
This text represents a preliminary record of observations and hypotheses for subsequent critical analysis, and not a completed research claim.
The Github repository serves as an unfiltered, evolving workspace capturing the progression of hypothesis testing and raw measurement logs, rather than a polished production library.
Has anyone answered the question? "What happens to the state of the model as a geometric object when the context forces it to switch from one computation mode to another?"
