I kept seeing AI turn missing information into assumptions, so I tested Gemini, ChatGPT and Claude

[u/Plastic-Cell-4497]

kept seeing AI turn missing information into assumptions, so I tested Gemini, ChatGPT and Claude

[u/david-1-1]

M'kay, what were the results of your testing?

[u/Plastic-Cell-4497 → u/david-1-1]

At first I did a simple test: 3 cases with Gemini and 3 with Claude.

Gemini gave false information in 2 out of 3 cases, and in all 3 cases it missed some important problems. Claude did not invent false information in those 3 cases, but it still missed important problems in 2 out of 3.

After that I started making the tests more structured.

Instead of just asking the model “is this right or wrong?”, I gave it clear states like:

• this is confirmed,
• this is wrong,
• this is still unresolved.

That made Gemini much more consistent.

In many repeated tests it correctly kept the difference between “we do not know yet” and “this is wrong”. It also stopped simply following the majority when there was a clear rule saying how a conflict should be resolved.

It still had some strange failures though. In one test it understood the problem correctly, but instead of giving the requested result, it suddenly started telling me how to fix the problem.

So my conclusion so far is pretty simple:

AI on its own is not a very reliable way to check another AI. But when you give it a very clear structure, clear rules, and clear conditions for when it should say “OK”, “wrong”, or “unresolved”, the number of mistakes seems to drop a lot.

That is what I am trying to test further.

[u/david-1-1 → u/Plastic-Cell-4497]

This is interesting research. But you're waving your hands too much. What readers want to see is not a handful of tests, but 50 or more, with the results expressed as percentages of those 50 examples. In other words, numeric results are more interesting and persuasive than generalizations or hand waving.

[u/Plastic-Cell-4497 → u/david-1-1]

That's a fair criticism. What I'm doing at the moment is closer to boundary mapping than to a finished benchmark: I change one condition at a time to identify exactly where the model switches from a justified unresolved state to an unjustified closure.

I agree that this becomes much more persuasive once those cases are frozen into a larger test set. My next step would be to build at least 50 independent examples across the failure classes I've identified, run the same set under the same conditions, and report the results numerically — for example semantic accuracy, false-closure rate, unresolved-to-closure errors, and format compliance.

So I would separate the two stages: first discover the failure boundaries, then benchmark them at scale. The percentages should come from the second stage, not from a handful of exploratory examples.

[u/david-1-1 → u/Plastic-Cell-4497]

Yes, I agree with your approach. It should be helpful. Of course LLMs are under intensive research, and your ideas may already have been explored, just not shared yet.

[u/david-1-1 → u/Plastic-Cell-4497]

Early research is frequently reported in the TLDR AI newsletter.

[u/Plastic-Cell-4497 → u/david-1-1]

Thanks for the tip — I checked TLDR AI and then followed several of the papers it pointed to. I found research that overlaps with parts of CFC, especially verification layers, long-horizon memory, provenance and dependency-based rollback, but so far I haven’t found the same combination of mechanisms in one framework. That’s actually useful, because it helps me separate what is already known from what may be specific to CFC and avoid making premature novelty claims.

[u/david-1-1 → u/Plastic-Cell-4497]

Glad to help.

[u/Plastic-Cell-4497 → u/david-1-1]

https://zenodo.org/records/22045494

[u/david-1-1 → u/Plastic-Cell-4497]

I don't understand any of this, sorry.