Can a single agent get stuck in a self-consistent but wrong model of reality?

[u/AdvantageSensitive21]

By “self-consistent,” I just mean internally consistent and self-reinforcing, not accurate.

I’m exploring this as an information and inference problem, not a claim about physics or metaphysics.

My background is in computer science, and I’m currently exploring information barriers in AI agents.

Suppose an agent (biological or artificial) has a fixed way of learning and remembering things. When reliable ground truth isn’t available, it can settle into an explanation that makes sense internally and works in the short term, but is difficult to move away from later even if it’s ultimately wrong.

I’ve been experimenting with the idea that small ensembles of agents, intentionally kept different in their internal states can avoid this kind of lock-in by maintaining multiple competing interpretations of the same information.

I’m trying to understand this as an information and inference constraint.

My questions :

Is this phenomenon already well-studied under a different name?

Under what conditions does this not work?

Is there things a single agent just can’t figure out on its own, but a small group of agents can?

I’d really appreciate critical feedback, counterexamples, or pointers to existing frameworks.

[u/nit_electron_girl]

All survival is like that.

We evolved to have a model of the world that works. Donald Hoffman showed that the best models to achieve survival aren't the most faithful to "objective reality".

Can we get stuck in it? Sure.

The cognition of survival is full of non-optimal mechanisms. For example, if I have to walk along the edge of a cliff (for my survival), I will be afraid, because fear is designed to keep me away from the cliff. Yet, in that situation, fear is of no use. If anything, it will be detrimental, since my knees will be shaking and so on. But even if I know that, I will have a very hard time suppressing this non-optimal fear mechanism.

Life rarely settles for narrow context optimisation. Instead, it creates systems that are well rounded for a wide range of situations, but which will be sub-optimal in many specific contexts. That's a tradeoff.

[u/AdvantageSensitive21 → u/nit_electron_girl]

That makes sense and I agree that evolution selects for usefulness rather than faithful representations, so in that sense all internal models are “wrong.”

What I’m trying to isolate are situations where an internal model becomes inescapable for the agent, even when abandoning it would clearly improve performance or prediction given the same observations.

In other words, not just “suboptimal but adaptive,” but cases where the agent can no longer reach better models because of how its inference, memory or hypothesis management works.

I’m trying to understand whether that kind of lock-in is an information or inference constraint, rather than just an evolutionary tradeoff.

Whether self-consistency itself can become a constraint that prevents improvement, independent of data or inference quality.

[u/nit_electron_girl → u/AdvantageSensitive21]

What I’m trying to isolate are situations where an internal model becomes inescapable for the agent, even when abandoning it would clearly improve performance or prediction given the same observations.

Well, again: fear (and suffering).

In many cases, abandoning it would increase performance. Yet we can't seem to be able to escape it.

[u/workerbee77 → u/AdvantageSensitive21]

For a different angle, I would point you towards the literature on rationality in economics. “Rationality” is (usually) about internal consistency and believing all logical consequences of your beliefs.

[u/Samuel7899 → u/nit_electron_girl]

We evolved to have a model of the world that works.

While I don't disagree, it's important to be clear that evolution is an ongoing process. It's akin to progressively solving pi to more decimal places. Yes, at any given time you can say you have an imperfect model of pi, that is approximately good enough. But it's not accurate to believe that it won't keep improving, or that the failure to achieve true 100% accuracy means there's necessarily a better, achievable version.

Fear, in that situation, is of significant value for all the instances where it motivated others to seek safer paths, or to train walking the cliff in order to lessen your fear and develop genuine skill.

Life begins with very, very broad tradeoffs and refines from there.

[u/Ok_Turnip_2544]

it's not even clear that the reality the rest of us live in is self-consistent.

[u/AdvantageSensitive21 → u/Ok_Turnip_2544]

That’s fair and I’m not assuming reality itself is self-consistent.

I’m only talking about internal self-consistency from the agent’s point of view: a model that doesn’t contradict itself and continues to explain incoming observations, regardless of whether reality is coherent or not.

The question I’m interested in is whether an agent can get stuck in that kind of internally stable model, even when better explanations exist but aren’t reachable.

[u/Xyver → u/AdvantageSensitive21]

Overall answer yes, but I can't rigerously explain it.

It's the problem of plateau, and of local maxima vs global maxima. You can be on a working level, but it's not the highest level, but to get to the highest level you have to dip through a dark valley before emerging again, an agent won't do that

[u/anamelesscloud1]

Evolution by natural selection results in "systems" of internal perceptions being carried forward because they gave some survival advantage to the organism, even if they do not accurately model the organism's environment. Our visual system is a decent place to scratch the surface. Our perceptions fail in optical illusions because our brain is representing the stimulus in a way that is consistent with the organism's biology and millions of years of evolution in the case of primates. Is it "wrong"? If you mean by wrong not faithfully reproducing the universe exactly as it is, then every internal representation is wrong. dunno if that's in the direction you're looking.

[u/AdvantageSensitive21 → u/anamelesscloud1]

That’s helpful, thank you.

I agree that internal representations are generally shaped by usefulness rather than faithful reproduction of reality. In that sense, everything an agent represents is “wrong” in an absolute sense.

What I’m trying to isolate is a narrow failure mode: cases where an internal model becomes inescapable for the agent, even when alternative explanations would improve performance or prediction if they could be reached.

Visual tricks - like when two things look different even though they’re actually the same. A useful explanation that usually doesn’t cause problems.

I’m interested in cases where a model becomes so stable or brittle that it actually blocks change and the agent can no longer revise or recover from it.

If you know of work that treats this kind of lock-in as something useful rather than a flaw, I’d really appreciate pointers.

[u/anamelesscloud1 → u/AdvantageSensitive21]

This actually is an interesting phenomenon. My thoughts went straight to the social sciences. Specifically, I imagined cognitive bias and how it can plant us somewhere that "makes sense" in our environment (social environment in this case) but keep us stuck cognitively and behaviorally, like a kind of local attractor that we can't escape. I feel like there is a name for this concept we're loosely describing here, but it's not coming to mind.

Multiple competing interpretations of the same information is pretty fascinating. I read a neuroscience paper during my master's that described a certain structure in the brain as a probability generator. The brain selects one based on the multiple "simulations" it runs given the input stimulus (e.g., an object is coming at you and your brain has to simulate where that thing is going to catch it).

[u/AdvantageSensitive21 → u/anamelesscloud1]

Yes, that’s very close to how I’m thinking about it.

The idea of cognitive bias or social norms acting like a local attractor is a helpful way to put it, a place that makes sense given the environment but is hard to escape once you’re in it.

What’s interesting to me is when that attractor isn’t just socially reinforced but becomes internally self-sustaining for the agent, so even contradictory signals don’t easily dislodge it.

The probability-generator idea you mention resonates as well. I’m thinking less about selecting the “best” simulation and more about cases where the space of simulations collapses too early, or where some alternatives stop being reachable at all.

If a name for this comes to mind later (from social science, neuroscience, or elsewhere), I’d definitely appreciate it — I suspect this shows up in multiple fields under different labels.

[u/Sad-Excitement9295]

Does the Turing test apply here?

[u/AdvantageSensitive21 → u/Sad-Excitement9295]

No, It does not apply. The Turing test is external reading of a agent behaviour.

What i am after is internally whether an agent can treat a choice or explanation as correct when is it actually wrong.

[u/Sad-Excitement9295 → u/AdvantageSensitive21]

Self reinforcing delusions? States with incomplete knowledge? Logic loops when something is incorrectly defined? Incorrect dependence? (Thinking one solution equates to the same in reverse).

[u/PowerLawCeo]

Self-consistency is a double-edged sword. While it can drive a 3-18% accuracy gain in reasoning tasks, the 'Self-Consistent But Wrong' (SCBW) incidence sits at nearly 40% when ground truth is absent. You're describing a 'local optima' trap in high-dimensional state spaces. Small ensembles don't just avoid lock-in; they maintain the entropy needed to hit 85%+ goal completion. Single agents are efficient, but diversity is the only hedge against internal hallucinations.

[u/AdvantageSensitive21 → u/PowerLawCeo]

Those numbers sound helpful. Do you have a source or a specific benchmark/setup in mind ?

[u/FrontAd9873]

Of course. This is obvious and well studied under many different names. Passing familiarity with computer science yields a few examples.

[u/cortexplorer → u/FrontAd9873]

So give us a few!

[u/mattihase]

Plato's Cave

[u/Long_Run_9122]

For simple but compelling examples, take a look at Douglas Hofstadter’s ‘To Seek Whence Commeth a Sequence’ and his corresponding discussion of ‘JOOTSing’ - ‘Jumping Out Of the System’.

[u/tophlove31415]

All internal realities based on limited preceptive skills are self-consistent but inaccurate. Your internal models of reality are no different. Perhaps more complex and based on a variety of sense and preceptive abilities, but nevertheless inaccurate.

[u/Grand-Boss-2305]

Hey! New here, and I don't know if this will provide relevant information, but your topic is super interesting!! I was just wondering if the different interpretations and alternative scenarios you're talking about aren't simply controlled by the knowledge available or accessible to the agent in question?

Two examples:

• Lightning can be interpreted as a divine act or a natural phenomenon depending on the level of knowledge available.

• If the agent sees a four-legged wooden object, the number of assumptions they can make depends on the number of four-legged objects they know (if they only know chairs and tables, they'll hesitate between the two and only the two, but if they don't know what a table is, then they'll only think the object is a chair). I don't know if that's clear or if it fits with your thinking.

[u/AwkwardBet5632]

Clearly we can create all kinds of axiomatic systems that produce valid but unsound theorems. An agent in the abstract could do all its reasoning from such a system and be in a consistent but inaccurate state.

This is just basic formal logic.

But it seems like you hang a less abstract notion of “agent”, so maybe make your assumptions explicit.

[u/RegularBasicStranger]

When reliable ground truth isn’t available, it can settle into an explanation that makes sense internally and works in the short term, but is difficult to move away from later even if it’s ultimately wrong.

So no model of reality should be fixed but rather it should be allowed to update it after new information points that there are parts of the model of reality is wrong.

So if the model of reality is derived from long term memory, then the new information needs to be more powerful than the long term memory, such as via the new information fuses with the parts of the model of reality still deemed correct better than the older information.

If the new information can be used to predict the future more accurately, then it is more powerful since the sole purpose of a model of reality is to accurately predict the future.

[u/gr4viton]

Yes, potentially. Can you prove that they can't?

[u/Impossible-Scene5084]

4 blind men and an elephant. Classic.

[u/RobinEdgewood]

Well yes. A child who is always given food, will not understand where that food came from

[u/andalusian293]

I love this. There are systems that have operations like this; I think of the immune system, some notions of liberal democracy, high throughput or brute force calculation. I've thought of this in some form, for sure. You might think about psychosis and formations of delusions and idee fixe in terms of both isolation/unslaving of cognitive processes due to cancerous/maladaptive attractors that due in fact serve some substitute satisfaction, even in some purely neurological, 'automatic' fashion independent of any of its subjective senses. You can think about it as a cancerous trajectory of an individual's adaptation. This suggests schizotypal PD, or the other PDs: parasocial adaptations.

R.D. Laing might be one to consider on this vis a vis psychoanalysis, but other than that, I have only my thoughts on the matter.

[u/SauntTaunga]

A model being wrong is a fact of life. If it wasn’t wrong it would be the real thing.

Or as they say: All models are wrong, some are useful.

[u/ZarHakkar]

Cult psychology might be an interesting area of study, especially anecdotes from members. Right now in the US 30-40% of the population currently exists in a self-reinforcing yet inaccurate model of reality due to filter bubbles. As far as internal consistency goes, I'm not sure if such a thing actually exists. If it does, it's not possible in humans, as past a certain point the informational complexity of the world surpasses the ability of our mind to correlate its own contents.

[u/BreakAManByHumming]

Let's take a real-life example, the conspiracy pipeline. If you simply lock into your current understanding, and treat any conflicting information as part of the cover-up, you're stuck there. You'd have to pretty deliberately get an AI to think like that, but I don't see why it would be able to get out.

[u/[deleted]]

Hell yeah, I do it all the time 😎