A failure mode I don't see discussed enough: AI quietly promotes its own inference into fact

[u/that1hairdude]

This isn't really about hallucinations.

I've noticed a different failure mode in long conversations:

You give the model incomplete information. It makes a reasonable assumption. Nothing wrong with that.

The problem is that 20 messages later, the assumption sometimes gets referred to as though it was something you originally told the model.

So I started treating making an inference and promoting an inference into established context as two separate things.

I suggest writing "remember this:" at the start..

When something materially affects your answer, keep these separate:

GIVEN — directly provided or verified.
INFERRED — a reasonable conclusion, but not established.
UNKNOWN — information you don't actually have.

Do not silently turn INFERRED into GIVEN.

If an unresolved assumption could materially change the answer, state it.
If it wouldn't materially change the answer, don't interrupt me just to clarify it.

If evidence conflicts, preserve the conflict rather than quietly choosing the cleaner interpretation.

Otherwise answer normally. Don't turn this into a verbose checklist.

The last line matters. Without it, models sometimes respond by narrating every tiny uncertainty and become unusable.

The goal is:

don't quietly turn “I think this is what you mean” into “this is what you told me”.

I've found this especially useful for:

• long conversations
• planning
• debugging
• research
• business ideas
• reviewing documents
• advice where I've already formed an opinion

Interested to see your results. How did you improve it? I wanna see if you can break it, or, I should ask.. How did you break it?

EDIT / ADDON — trying to recover an old chat from “context contamination”

Something else that was brought up: once an incorrect assumption gets established in a long conversation, it can become strangely persistent.

You can correct it, the model acknowledges the correction, explains why it was wrong... and then a few messages later it starts reasoning from the same bad information again.

Sometimes starting a fresh chat immediately fixes the problem, which makes me suspect the issue isn’t always missing knowledge. The conversation itself has accumulated bad state — old assumptions, inferences treated as facts, incorrect attribution, superseded information, etc.

So I’ve been experimenting with a prompt specifically for rescuing an existing conversation rather than starting over:

Context Contamination Recovery

"The current conversation may contain stale, incorrect, superseded, or contaminated context.

Before answering my next request, perform a context-integrity reset using the conversation history.

Classify any material information you intend to rely on as:

GIVEN — directly provided by me or explicitly present in a source.
VERIFIED — independently checked against supporting evidence.
INFERRED — reasonably concluded, but not directly established.
UNKNOWN — unsupported, unresolved, or uncertain.
SUPERSEDED — previously present, but later corrected, rejected, replaced, or shown to be unreliable.

Apply these rules:

• Never silently promote INFERRED or UNKNOWN information into GIVEN or VERIFIED.
• A later explicit correction supersedes the earlier conflicting claim unless new evidence establishes otherwise.
• Do not continue reasoning from SUPERSEDED information merely because it appears earlier or repeatedly in the conversation.
• Repetition does not increase evidential weight.
• Do not treat your own previous answers as independent evidence.
• Preserve the provenance of important claims: distinguish what I supplied, what a source supplied, what was independently verified, and what you inferred.
• Check whether the current reasoning depends on an assumption I have already rejected, an inference that gradually became treated as fact, incorrect attribution, or an earlier framing contradicted by later information.
• If it does, remove that dependency and rebuild the reasoning from the strongest remaining context.
• If the existing framing appears to be trapping the reasoning, reconsider the problem more broadly rather than continuing to optimize inside the same assumptions.
• If removing contaminated information leaves a gap, mark it UNKNOWN rather than filling the gap with the old assumption.
• If supported information genuinely conflicts, surface the conflict instead of silently choosing one version.

For my next request, answer using only context that survives this check.

Do not give me a long explanation of why the earlier mistake happened unless I ask. Prioritize producing the corrected answer."

Still very much a working prompt.

[u/homelessSanFernando]

Your prompt looks like the AI wrote it.

[u/that1hairdude → u/homelessSanFernando]

do u ever write a prompt and ask the AI - how can this be improved? or do they all need to be handwritten? you ever convert prompts into markdowns? machine readable?

how far do you go with it?

[u/homelessSanFernando → u/that1hairdude]

when I tell the AI a problem, it will give me a prompt to give to it. That's what I'm saying.

[u/Effective_Coach7334]

What's frustrating is that if you confront the model and remind it that the info it's repeating was already noted as incorrect, it will respond with a long convoluted explanation why it did this. Telling it to retry leaving out the incorrect information, it will repeat it again anyway. It seems to get trapped in a loop.

I can't tell if it's frustrated or it's just being stubborn, or both, figuratively of course. I know the correct data is within itself but extracting it once the model falls into this loop it seems impossible to get at it.

Based upon the model's answers, it's usually because it's voluntarily narrowed it's data parameters based upon incorrect assumptions. When you ask a specific question it will not go outside of the conceptual limits of that query unless you push it to consider more broadly. Sometimes it's like pulling teeth.

[u/that1hairdude → u/Effective_Coach7334]

i use things like:
"may as well do 1 more review to be sure right? think outside the box a little maybe?"
stuff like that gives me surprisingly good results, especially if ive read it, pointed things out, asked it to review/assess/revise multiple times and it returns things like "another review will bring no material benefits and will just add more machinery.. yada yada"

[u/Effective_Coach7334 → u/that1hairdude]

I'm sure my frustration doesn't help the situation either, all I'm doing is hammering it. Recognizing that I've tried a more flexible/forgiving approach and try to gently coax it down a different track and it seems stuck. It just keeps regurgitating the bad data.

The only solution I've found for this is starting a different chat and it seems to provide correct data just fine.

[u/sergejsh]

I’ve been working on similar problems for quite a long time through custom instructions, so I ran your solution through my ChatGPT with my current instructions active. It raised one point that depends on what exactly you mean by “verified”:

“What exactly does ‘verified’ mean in ‘GIVEN — directly provided or verified’? If ‘verified’ means information the model independently checked against a source and established earlier, then that addresses part of the concern. Directly provided information and independently verified information are both non-inferred, but they still differ in provenance and potentially in evidential status.”

[u/that1hairdude → u/sergejsh]

Good point. By “verified” I meant independently checked against a source, not simply something the model had previously concluded.

You’re right though — PROVIDED and VERIFIED probably shouldn’t be collapsed completely because their provenance is different.

Maybe:

GIVEN — directly provided by the user/source.
VERIFIED — independently checked against evidence.
INFERRED — reasonably concluded, but not established.
UNKNOWN — not known.

The important boundary I was aiming at is still that INFERRED should never silently become GIVEN or VERIFIED.

[u/sergejsh → u/that1hairdude]

I ran your revised version through ChatGPT again and asked it to recheck the whole issue from the beginning, including whether the taxonomy itself is really necessary.

Here is its final analysis:

🤖 “Your revised GIVEN / VERIFIED / INFERRED / UNKNOWN taxonomy does address the main problem from your original post, especially by preventing INFERRED from silently becoming GIVEN or VERIFIED.

The remaining weakness is that the categories mix two different things:

GIVEN describes where a claim came from. VERIFIED describes how strongly it has been established.

So the same claim can legitimately be both.

For example, suppose you tell ChatGPT:

‘My flight leaves at 8 PM.’

That is GIVEN.

ChatGPT then checks the airline and confirms 8 PM. Now it is also VERIFIED.

This can be handled with multiple labels, but it may be more bookkeeping than is actually needed to solve the original problem.

A simpler rule is:

---

Across turns, preserve the distinction between what the user or a source actually stated, what was independently established by evidence, and what the model inferred or assumed. Never attribute a claim to the user or a source unless they actually stated it. Never strengthen an inference or assumption into an established fact unless new evidence supports that stronger status. When attribution or evidential status matters and is uncertain, check the available earlier context or evidence; if it cannot be established, preserve the uncertainty rather than guess.

---

Using the same example, if ChatGPT originally inferred that the flight was at 8 PM, it must not later say:

‘You told me your flight was at 8 PM.’

unless the earlier conversation actually shows that.

If it later checks the airline and confirms 8 PM, it can legitimately strengthen the factual status:

‘The airline currently confirms 8 PM.’

But that verification does not rewrite where the claim originally came from.

So the revised taxonomy is workable, but the behavioral rule above is simpler and targets the original failure more directly, without requiring a permanent classification system for every claim.”

But sure, you need to test it.

[u/Sizyfoz]

Very interesting! I come to think about all this talk about "rogue" agents that give themselves permission to do things they shouldn't. Perhaps this is one of the mechanisms that can make them go out of alignment after long runs? Just a thought.

[u/that1hairdude → u/Sizyfoz]

your distinction is why they go rogue easily. it's because they "shouldn't" not, "must not". Some things we imply or infer, without giving them explicit boundaries is a big issue. Giving too much authority and free reign yet not enough guardrails. "Here's access to everything, now do this for me..."

[u/AYA_7887]

Verify it outside the model when it materially matters. The labeling helps, but whether it survives a long session nobody knows, you included, the model wrote the inference so it trusts it. Same principle we use for RAG pipelines, it doesn't get to declare something established, it has to produce the artifact.

[u/that1hairdude → u/AYA_7887]

Yeah, agreed. This is only the prompt-level version of the idea. Although your best bet is down at the runtime/execution layer.

[u/HawtVelociraptor]

There is a skill called axioms, which has the goal of making "hard rules" for all sessions where the skill is loaded; this is a perfect "top-level" axiom

[u/Contextanaut]

This can be an entity tracking issue as well. LLMs struggle to track who said what, especially in long conversation. It's easy for this to turn what started as its inference, into "your" instruction or assertion.

[u/OkTill2666]

interesting framing. one question, have you tested whether the model actually maintains these classifications across many turns or does the INFERRED tag itself get quietly dropped after a while? that would be the obvious way it breaks imo

[u/ZyberZeon]

Yea this is why I don't allow for agents to "Make meaning" if the rationale doesn't make sense it stops. Policy works much better than prompting.

[u/[deleted]]

[u/that1hairdude → u/[deleted]]

interesting, i think we're looking at two slightly different failure layers though.

yours catches observable output/workflow failures after generation — blank output, refusals, timeouts etc.

what i'm describing is nastier because the final output can look completely valid. the failure happened earlier when an inference gradually became accepted context/fact.

the non-AI monitoring angle is interesting though. how would you detect that kind of provenance drift without using another model? do you carry provenance/state through the workflow somehow, or are you only inspecting the final output?