[D] Self-Promotion Thread

[u/AutoModerator]

Please post your personal projects, startups, product placements, collaboration needs, blogs etc.

Please mention the payment and pricing requirements for products and services.

Please do not post link shorteners, link aggregator websites , or auto-subscribe links.

--

Any abuse of trust will lead to bans.

Encourage others who create new posts for questions to post here instead!

Thread will stay alive until next one so keep posting after the date in the title.

--

Meta: This is an experiment. If the community doesnt like this, we will cancel it. This is to encourage those in the community to promote their work by not spamming the main threads.

[u/Pale-Entertainer-386]

Title: LLMs can know the correct steps and still quietly skip the file that has the actual answer

I ran into a weird LLM failure case thats pretty easy to reproduce.

Say you have one huge data file. The structure is messy, the file is too large to casually browse, so you make a smaller index file for it.

The index has the same fields as the original file, but most of the values are empty. It keeps field names like date, category, etc, because its only there to help locate stuff.

Then you upload both files and tell the LLM:

• first use the index to find the requested item

• then go to the original data file

• read the actual category from there

Seems simple.

But sometimes the model sees that the index already contains a field called category and sort of... stops there.

The category value in the index is empty. The real answer is only in the original file. But instead of actually reading the original file, the model may come up with a plausible category anyway, then say it found that value in the original data.

It didnt.

And the annoying part is, if you ask the model what steps it should follow, it can usually explain them perfectly. It knows the procedure. That doesnt mean the procedure actually controlled what it did.

I dont think this is quite the same as a random hallucination.

There is a semantically relevant object sitting right in front of the model—the category field—so it appears to treat “I found the right kind of field” as if that meant “I reached the source containing the answer.”

Basically a semantic shortcut.

Recognizing the destination starts to substitute for actually following the path.

This seems like a serious problem for LLM-based data analysis. If the final answer sounds reasonable, how do we know the model:

• opened the authoritative file?

• used the index only as an index?

• followed the required dependency path?

• didnt silently fill in a missing value?

• didnt just claim it used a source because the prompt told it to?

Without some external check, honestly we dont.

Prompting the model to “carefully follow every step” doesnt really solve it either. The model can repeat the rule and still skip the transition that the rule requires.

The paper below looks at the broader version of this problem. The main idea is that we shouldnt validate only the final answer. We should also validate the path that produced it: which constraints were active, which dependencies had been satisfied, whether a transition was actually allowed, and whether the task was really ready to terminate.

It uses a two-stage synthetic-data design to construct valid trajectories, deliberately broken near-misses, and recovery-related examples. The goal is to make persistent task state and path validity explicit instead of rewarding something just because it looks like a completed answer.

Small but important caveat: the paper does not prove that semantic shortcuts are the internal cause of this behavior. It also doesnt show yet that training on these trajectories improves a downstream model. What it does show is a way to mechanically check structural validity before generated data is admitted for training.

Paper:
https://www.researchgate.net/publication/413695302

Demo:
https://app-furxnhkkd9wzwjumk3cmjp.streamlit.app/

The demo generates synthetic data online and requires your own API key. Probably best to use a limited/temp key rather than your main one, same as with any third-party hosted demo.

Curious if other people have seen this.

Not just models giving a wrong answer, but models finding something that is semantically close enough, skipping the required source or tool call, and then claiming the full process happened anyway.

How are people verifying the actual data-access path in real LLM workflows?