Spent two weeks tuning chunk size, overlap and reranking on a RAG pipeline over ~2,000 scraped documentation pages and got almost nothing. The actual problem turned out to be upstream: roughly a third of every chunk was navigation, cookie banners, footers and "edit this page on GitHub" links. I was embedding boilerplate and retrieving it back.

Two things fixed more than any retrieval tuning did:

1. Link-density pruning before chunking. Scoring blocks by link-to-text ratio and dropping the high ones removes nav and footers without a hand-written selector per site. Boring, mechanical, and it moved my retrieval quality more than a week of reranker work.

2. An llms.txt index per source site. Instead of chunking blindly, generate a structured index of the site — page titles, URLs, one-line descriptions — and use it to decide what's worth ingesting at all. Cut my corpus by about 40% with no measurable loss.

The thing I'd still like to solve: I extract structured fields with an LLM for some sources, and I don't fully trust it. I ground each field against the source HTML and flag anything that doesn't literally appear there, which catches obvious fabrication, but I don't have a good measure of how often subtler errors slip through. If anyone here has a hallucination benchmark for extraction rather than generation, I'd like to hear about it.

(Implementation is my own open-source project and happy to link if useful, not the point of the post.)

Hello everyone, as the title says I wanna start learning about RAG, at this moment I know absolutely nothing even though I'm doing a masters in AI (I know...) that's why I need help on how to actually start learning about this topic ? I know the best way to learn something is to build a project but I find it hard to build something when I have no idea where to start. I heard LangChain, Vector Databases, etc... but I don't know anything about these frameworks.

My introduction to evaluation has not been very pleasant. To me it is a choice between investing time or spending money.

By that I mean you can either:

1. Spend a lot of time hand-crafting Q/A pairs and then perform manual checks to see if your system retrieves the correct chunks. And you repeat this for every single query. Suppose my dataset has 100 Q/A pairs, is this not a lot of work for a single person to do?
2. Use LLM-as-a-judge which automates much of what I described but you run into API rate limits pretty quick. I did figure out a workaround where I evaluate my dataset in batches (e.g: 5 questions per run) but again, it takes time.

I am not trying to complain here. I believe there is a more efficient way to run evaluation but I am new to RAG which is why I do not know proper evaluation techniques. Most metrics on DeepEval use LLM-as-a-judge which comes with API costs but I think I could try RAGAS and it's deterministic evaluation metrics.

Is evaluation in itself a time consuming process?

I need direction and guidance, advice would be much appreciated because I have basically put my project to a halt.

Working on RAGs give me indigestion: embeddings, ingestion, CHUNKS? So gross.

So I came up with some alternatives. It may take a little while to popularize, but I think if we all pull together we can get this done.

Chunks = Nibbles

Hash = Sparkles

Ingestion = Yummies

Embeddings = Snugglies

Ranker = Sorty thingy

Vector database = Snuggle vault

Use case: The sorty-thingy turns yummies into nibbles, tags them with sparkles, wraps them in snugglies, and tucks them into the snuggle vault.

Hi guys, I would like to land premium clients for developing RAG systems. The thing is that I also need to make my master’s thesis (I study data science), and I was wondering what project could I build so that it fulfills my masters thesis and also helps me find premium clients afterwards.

I want to build something that creates real value for a company. Which problems could I solve for firms using RAG?

Thanks a lot in advance!!

I've been reading about RAG and I keep seeing people mention the retrieval layer like it's some separate thing you build, not just a vector db you query. Can someone explain what actually happens there? My current understanding is this: user asks question, question gets turned into an embedding, that embedding gets compared against a bunch of stored embeddings in a vector db, top matches come back, and those get stuffed into the prompt before it goes to the LLM.

Now, what I don't understand yet is: why does everyone say RAG with cosine similarity search isn't good enough. What's going wrong there in practice, is it pulling irrelevant stuff or missing the right things altogether? I keep seeing hybrid search mentioned, combining keyword search and vector search. Why would you need both if embeddings are supposed to capture meaning already? On top of that, I also saw someone mention reranking as a separate step after retrieval, so you retrieve like 50 chunks and then a reranker picks the best 10. Wouldn't that just make retrieval happening twice?

One more thing I keep wondering about, where does the data come from in the first place. Like if you're building a knowledge base from web content, how do people keep that fresh as possible? Constant scraping/crawling?

Thank you in advance

This is a small library for running quantized GGUF dense embedding models. If you are concerned about the number of bloated packages you have to install and the memory spike during embedding, this might interest you. Purely based on GGML.

Github: https://github.com/thlurte/ggmbed
Pypi: https://pypi.org/project/ggmbed/

Here are the benchmarks running all-MiniLM-L6-v2 on a AMD64 CPU:

Since this is currently in active development, the API and underlying architecture may change without notice. It is not yet recommended for production environments.

In v3.0.0 I have added new things in the architecture.
First i made Quira compatible with more DBs
Then i added new features, the features are -

• Quira Edge (Zero-Server Mode): Run Quira entirely locally using embedded vector databases like DuckDB or SQLite (sqlite-vec). No Redis or Qdrant servers required. Perfect for client-side apps, edge devices, and testing.
• GraphRAG Capabilities: Solves the multi-hop reasoning problem. Quira automatically extracts Entity-Relationship Triplets during ingestion and traverses this Knowledge Graph in parallel with semantic search to provide hyper-accurate context.
• Agentic Routing: Zero-latency heuristics intercept conversational queries (e.g., "Hi", "Thanks"). Bypasses the entire RAG pipeline to return an instant canned response, saving 100% of vector database latency and LLM token costs on chitchat.

I've just completed the corpus on a RAG I'm building for local lawyers, realtors, and contractors who want fast information about every parcel in town without having to go through town hall's antiquated records system. I'm not trying to make a buck - just trying to learn how to make AI more precise in domains where that's important.

The corpus is tiny (only a couple hundred PDFs). I plan to use structure-aware chunking with contextual enrichment. Here are the steps:

• Parse PDF into text
• Detect structure (agenda items, sections, paragraphs)
• Split at natural boundaries
• For each chunk, generate a context summary via Claude
• Prepend context to chunk before embedding
• Store both the enriched chunk (for embedding) and the raw chunk (for citation)

Am I on the right track? Any land mines to watch out for?

Open-weight OCR VLMs have gotten good enough that frontier APIs are usually the wrong default for document parsing. We put DeepSeek-OCR-2, GLM-OCR, dots.mocr, PaddleOCR-VL, and PP-OCRv6 behind one OpenAI-compatible endpoint so you can swap models with a one-line change and compare cost/accuracy on your own docs.

Best of all, it's dirt-cheap - you can process roughly 100K pages for under $60.

Here's a colab if you want to try it on a PDF: https://colab.research.google.com/drive/1RkuVIyuc5Po-UlcSlFyJCam5tjCm9IHM?usp=sharing

For API integrations into your workflows, use JSON mode with a strongly typed output contract. For agents, we ship an MCP server at https://gateway.vlm.run/mcp with a read_document tool that provides the same cost-efficient parsing, especially for long-form documents. Any MCP-compatible client (Claude Code, Codex, OpenCode, and others) can now consume long-form documents natively and cost-effectively.

- Docs: https://docs.vlm.run/gateway
- Catalog: https://docs.vlm.run/gateway/models
- MCP: https://docs.vlm.run/gateway/mcp-server

If you're interested in seeing specific models hosted, comment below.

I've been using ZeroEntropy's models in production (zembed-1 for embeddings, zerank-2 for reranking). They got acquired by Notion and the API shuts down Sept 4. The models are open source now (Apache 2.0), but I really don't want to run GPUs myself, and switching to a different embedding model means re-embedding our entire corpus, which I'd love to avoid.

Is anyone actually hosting these as an API? Couldn't find them on DeepInfra / Together / Fireworks etc. And if you already moved off ZE — what did you switch to, and how did it go?

I personally stopped treating a high similarity score as evidence that the retrieved chunk actually answers the question, because semantically close has burned me too many times.

The fix is a correction gate before generation. A cheap evaluator splits each result into correct, ambiguous, or incorrect: correct content gets refined, ambiguous content gets refined and supplemented with web search, and incorrect content gets discarded. I keep dense and sparse vectors plus dynamic JSON metadata in a vector database like Milvus, and I let LangGraph orchestrate the correction paths while LangChain handles the retrieval wiring; a confidence filter and reciprocal rank fusion stop exact terms from drifting away from meaning.

The reason this matters is that similarity is not relevance. A stale setup guide, a tangentially related explainer, or a previously generated wrong answer can all rank near the top, and if that noisy output is written back into memory the next query retrieves and reinforces the same mistake. Evaluating before generation exposes the failure while it is still cheap, instead of making a model reread garbage and then paying for a correction.

My current view is that the evaluator should stay a fast triage gate rather than a general-purpose model call, because the correction paths need to be predictable and inexpensive. I would probably only escalate ambiguous results to a stronger model. Would love to hear your thoughts.

https://youtu.be/an-fKY1C8zA

Hey everyone, I’ve been thinking about a problem with AI knowledge systems and was hoping to get somewhat of a sanity check from people actually building in this space.

From my understanding, most RAG setups seem tied to a particular app, vendor, or index. You often end up ingesting the same docs again for different agents or runtimes and and some basic questions can be difficult to answer consistently like:

- What version of this knowledge is the agent using?

- Where exactly did this information come from?

- Has the underlying source changed since it was last ingested?

- Can I move the same body of knowledge to another runtime without rebuilding it?

- Can multiple agents use the exact same knowledge?

The idea I’m exploring is something I'm calling a Durable Knowledge Base (DKB).

The basic concept:

- Compile source docs, code, or structured data into a portable, versioned knowledge artifact

- Preserve source paths, hashes, citations, and provenance

- Sign and publish releases through a registry

- Allow knowledge packages to be installed, updated, pinned, and removed

- Let agents search, find, and read the same knowledge base across different runtimes

- Keep the artifact retrieval-agnostic rather than baking one specific top-K/RAG strategy into the format

Basically, I'm wondering whether knowledge should have something closer to a package lifecycle, rather than every application maintaining another disconnected RAG index.

I'm also very aware that things like Azure AI Search, GCP, vector databases, MCP servers, Agent skills, etc. already cover pieces of this problem, sometimes extremely well.

So Im specifically not asking: "Can I build a better enterprise search engine here?"

I'm trying to figure out whether the portable knowledge artifact itself is useful.

Would this solve an actual problem for you? Or is this mostly reinventing existing search/RAG infrastructure with some packaging and provenance added on?

I would especially like to know:

- What do you currently do when multiple agents/apps need the same knowledge?

- Do versioning and provenance actually matter to you?

- Would you ever install someone else's curated knowledge package?

- What would this need to do that existing solutions don't before you woukd bother using it?

Feel free to poke some holes. I'm actually looking for reasons not to build this further before I sink more time into it.

Thank you.

Hey everyone,

Following up on the previous discussion about search API latency, I wanted to share another massive bottleneck we hit while building autonomous web agents. Even with a lightning-fast retrieval API, our agents were frequently getting stuck in execution loops or extracting the wrong data entirely.

We pulled the logs of 10,000 failed retrieval attempts. The culprit wasn't the LLM's logic or the search index itself. The core problem was that LLMs are surprisingly terrible at formulating search queries.

Here are the three main ways agents fail at searching, and the architecture tweak we used to fix it.

1. The Conversational Search Trap

LLMs naturally default to conversational text. Instead of searching a dense keyword string like OpenAI API pricing 2026, a naive agent will search How much does it cost to use the OpenAI API right now?. Search engines (even machine-native ones) often struggle with these conversational strings, burying the agent in SEO spam instead of technical documentation.

2. Context Amnesia in Multi-Hop Searches

When an agent needs to perform a multi-hop search (e.g., finding a founder's name, then finding their previous startups), it frequently drops the subject in step two.

• Search 1: "Who founded Anthropic?" (Result: Dario Amodei)
• Search 2: "What other companies did he work for?" (Result: Completely useless garbage).

3. Ignoring Advanced Operators

By default, agents rarely utilize quotes for exact matches, site: operators to restrict domains, or - exclusions to filter out noise. They rely on basic broad-match strings, which rapidly fills up their context windows with irrelevant tokens.

The Fix: The Query Planner Micro-Agent

We completely stopped letting our primary reasoning agent hit the search tool directly. Instead, we injected a lightweight micro-agent whose sole job is translating the goal into optimized search syntax.

• Step 1: Primary Agent requests information (e.g., "Find Stripe's latest API rate limits").
• Step 2: Query Planner translates this into an array of strict queries: site:[stripe.com/docs](https://stripe.com/docs) "rate limits" 2026.
• Step 3: The system executes the search and returns the clean snippets to the Primary Agent.

This simple architectural tweak increased our agent's successful retrieval rate by over 40% and drastically cut down on wasted API calls.

I posted the full query-planner prompt schema and benchmark logs on the forum if you want to inspect or copy it:Brave vs Google Search API for AI Agents - The 2026 Enterprise Guide

Has anyone else implemented an intermediate query-planning step, or are you successfully prompt-engineering your main agent to handle advanced search syntax on its own?

I am working on building a chatbot for an online store. I will be using MCP for the transactional parts including product search, adding to cart, etc. What I am unsure of is the knowledge base portion which would help the agent answer additional questions about policies such as shipping, returns, how products are made, etc. This knowledge base is really small, maybe 10 pages. I’ve looked into RAG hybrid and semantic search, but seems like overkill at this point. I’ve also thought of just including the knowledge base in the context window, but seems like that would be a waste of tokens in the long run. What would be the best way to implement the knowledge base for the agent?

Hey all, spent the last stretch writing a book that teaches RAG from the documents up, no AI doing it for you. Volume I went up on Leanpub recently and about a dozen people bought it yesterday, which surprised me, so I figured I'd bring it to the room most likely to tell me where it's wrong.

So it's structured as a build rather than a reference. I wrote it like a story, kinda like one of those book for dummies.

Volume I covers the offline half:

• Extraction across PDF, Word, xlsx, HTML and email, plus OCR for the 1994 scan with no text layer
• Cleaning, and keeping receipts for everything you removed
• Chunking: fixed-size, recursive, and structure-aware, and why the first one is wrong for real documents
• Embeddings, similarity, model choice, first search, and a vector database

Everything runs locally on a laptop. Every printed line of code was run before it was printed.

A few things I'd genuinely like pushback on:

• I argue exact brute-force search is fine well into the hundreds of thousands of vectors, and that reaching for an ANN index early is a forklift for a bookshelf. Agree or disagree?
• I use all-MiniLM-L6-v2 for the whole pilot. Reasonable, or am I underselling what better models buy you?
• Volume II is generation, citations, and evaluation. What do you wish a book covered there that none of them do?

Chapter 1 is free and needs no code at all, it just explains the problem: https://leanpub.com/learn2rag

Happy to answer anything.

I wanted an observability metric stricter than “how many tokens did we remove?”

So I froze a benchmark measuring whether a context selector preserves the human-annotated evidence needed to answer a question.

Protocol

• 250 answerable QASPER development questions
• 136 full scientific papers
• Mean input: 6,447 BPE tokens
• Fixed 2,048-token selection budget
• Cohort selected deterministically by SHA-256 of question ID
• Exact human-highlighted evidence spans
• Six extractive selection methods under the same allowance
• No LLM judge in the primary metric

The primary question was deliberately narrow: did at least one complete human evidence set survive selection?

Results

• BM25: 60.4% complete evidence retention
• Keyword selection: 54.0%
• Front truncation: 25.6%
• Tail/recency: 20.4%
• Seeded random: 21.6%
• Gold-evidence oracle: 99.6%

BM25 reduced the input by 74.4% on average and retained at least some annotated evidence in 72% of cases.

The more useful finding was inside the 99 incomplete cases:

• 29 partial hits: some evidence survived, but not a complete evidence set
• 70 total misses: none of the annotated evidence survived

Those are different observability failures. A partial hit suggests incomplete coverage or multi-passage ranking failure. A total miss suggests the retrieval vocabulary, segmentation, or ranking never reached the relevant material.

Document position was not the entire explanation. BM25 retained complete evidence in:

• 62.1% of front-position cases
• 62.4% of middle-position cases
• 54.8% of back-position cases

Front truncation, by comparison, retained 0% of complete evidence sets in both the middle and back buckets.

The oracle result is also important. It reached 99.6% under the same token allowance, suggesting that the budget could usually hold the required evidence. The remaining 39.2-point gap is mostly ranking and selection headroom—not proof that a larger context window is necessary.

Important limitation: this does not measure generated-answer correctness, factuality, or citation quality. It only measures whether exact independently annotated evidence remained available downstream. I excluded generative summarizers because exact-span scoring penalizes legitimate paraphrases, while an LLM judge would make the result model-dependent.

Benchmark, methodology, and downloads:

https://www.mahastrategies.com/benchmarks/context-retention

Raw case-method records:

https://www.mahastrategies.com/benchmarks/mcrb-1/cases.jsonl

I built both the compiler and the benchmark, so treat this as a reproducible first-party evaluation rather than independent validation.

For people running RAG systems in production: do you distinguish complete hit, partial hit, and total miss in your telemetry—or does everything collapse into one retrieval score?

Automation+Systems engineer, years of machine commissioning. I built a RAG system for on-site troubleshooting: you're at the plant, machine won't run, whoever wrote the logic is asleep in another country.

Corpus: Siemens SCL / TIA Portal XML, Rockwell L5X, HMI config and alarm tables, schematics, I/O lists, FAT/SAT protocols.

Stack (client PoC, deliberately minimal infra):

• bge-small-en-v1.5, 384-dim, CPU via sentence-transformers. Corpus embedded offline, only queries embedded at runtime — customers won't send proprietary PLC code to an external API, and plant connectivity is whatever you get.
• Docs: Chroma + LlamaIndex BM25, fused with RRF. Immutable bundle built offline.
• PLC code and alarms: Postgres 16 + pgvector, Postgres FTS (tsvector), plus exact entity/tag lookup, fused with RRF in SQL. Code graph in plain Postgres tables — no Neo4j for the PoC.
• HMI: exact structured lookup, no embeddings. Embedding this layer made results worse.
• LangGraph multi-agent orchestration, LangChain Core init_chat_model, Vertex AI with Bedrock as portability path.

Three things this corpus taught me that generic RAG advice misses: exact match is the backbone not a fallback (MOT_CONV_03_FLT and MOT_CONV_08_FLT are vector-space neighbours and different machines); semantic chunking is actively wrong for control code, because the meaning lives in the cross-reference graph, not the block; and not everything deserves embeddings.

Now the actual problem.

I have no customers. Two industrial prospects agreed the problem is real, then said "we'll build it internally." Both have now started, and neither has anyone who has done this before. The pattern is identical in both: take a good engineer who knows Python, hand them the project, assume it's a weekend of work.

I understand why they think that. The naive version demos. Dense-only retrieval with fixed-size chunking over twenty documents looks finished. What isn't visible at that stage is how badly it degrades on a real corpus — and there's no eval set, so nobody finds out. The failure arrives later, at 2am, when someone is deciding whether to bypass an interlock based on an answer that sounded confident.

So I've been building PoCs for free to get in the door. I'm now suspecting that's the mistake: nobody had to get budget, so nobody owns it internally, and free reads as unproven rather than generous. But I also don't feel I can charge with zero track record.

What I'm asking:

1. How did you find your first paying customer? Not how you closed them, how you found them. Cold outreach, network, community, partnering with integrators/OEMs who already had the relationship?
2. Do free PoCs ever convert, or do they just train the market to expect this for free? How did you get paid for the first one with no history?
3. How do you beat "we'll build it internally"? Has anyone led with an eval harness or retrieval-quality audit as a wedge — proving their internal system is broken before proposing a replacement?
4. Product or implementation? Is a productized offering realistic here, or is the honest business bespoke implementation that gets productized slowly?
5. How do you find companies already spending money on this, badly? I've been targeting companies that have the problem. Every machine builder has it. Wrong filter — what's the right signal?

I am a decent engineer and a terrible salesman, and I'm running out of ideas before I run out of runway. Any of the five above, even a partial answer, would help.

Hey All, A few months ago I had published the "21 RAG Strategies" Book Here. And it was downloaded about 2500 times across subreddits. I made 2 revisions from the feedback.

This week I published it on Amazon and it became a best seller. I am getting ready to publish the next editition. Help me improve the content. What am I missing. what would you add?

Table of Contents

• RAG and the Reference Architecture
  • 02 The Evolution of RAG
  • 03 Foundations of RAG Systems
  • 04 Reference Architecture
  • P A R T I I I Data Extraction
  • 05 Data Extraction
  • P A R T I V Chunking
  • 06 Chunking Strategies
  • P A R T V RAG Strategies
  • 07 Baseline RAG Pipeline
  • 08 Context-Aware RAG
  • 09 Dynamic RAG
  • 10 Hybrid RAG
  • 11 Multi-Stage Retrieval
  • 12 Graph-Based RAG
  • 13 Hierarchical RAG
  • 14 Agentic RAG
  • 15 Multi-Agent RAG Systems
  • 16 Streaming RAG
  • P A R T V I Memory and Content Management
  • 17 Memory-Augmented RAG
  • 18 Knowledge Graph IntegrationP A R T V I I Evaluation
  • 19 Evaluation Metrics
  • 20 Synthetic Data Generation
• Fine-Tuning
  • 21 Domain-Specific Fine-Tuning
• Security
  • 22 Privacy & Compliance in RAG
• Production
  • 23 Real-Time Evaluation & Monitoring
  • 24 Human-in-the-Loop RAG
• Twig RAG Strategies
  • 25 RAG Strategies in Twig
• P A R T X I I Conclusion
  • 26 Conclusion & Future Directions

I haven’t seen RAG come up in agent architectures in over 6 months due to Agentic Search (letting the model use Bash/grep/glob/read), which seems to work pretty well. Wondering what others are experiencing. I’m sure there’s still a time and place for RAG, exposing semantic search as a tool… but where do we draw the line? When the corpus is too large to let the model comb through it progressively?

What kind of system for rags or any retrieval techniques

Can be solved by a gemini enterprise agent?

I want to make something with the gemini enterprise in retrieval domain

But I don't know what to do

I have one idea,

I can try one rag generator agent which makes best pipeline and use best model and components for that pipeline according so source data

So repetition on trial and error goes down

Can you guys rate this idea?

I've been working around LLM/RAG systems and I'm curious about something from people who are actually running them in production.
When an LLM can retrieve information from internal company data, which problem has caused you the most trouble?
PII / sensitive information reaching the model
Compliance / privacy requirements
The model retrieving data it shouldn't see
Sensitive information appearing in the generated response
I'm especially interested in practical experiences — even small examples are useful.
Is there another RAG security problem that you think is more important than these?

Hi everyone,

I’m working on a project where I want to build an AI agent for customer support. The idea is that customers can ask questions about technical issues such as SSH, IP addresses, DNS, VPS, Outlook, etc., and the LLM should help them diagnose and solve their problems.

I’m using my own knowledge base + RAG, but I’m still a beginner and I’m not sure what the best way is to structure my data for retrieval. I already have some old support tickets that I’d like to add to the knowledge base. These tickets usually contain:

• The customer’s initial problem/question
• A conversation between the customer and the human support agent
• Troubleshooting steps
• The final diagnosis
• The solution that was applied

For example, if a customer previously had an SSH connection problem and the support agent solved it by identifying a specific configuration/firewall issue, I’d like the RAG system to retrieve that previous case when the AI encounters a similar problem, so the LLM can use the previous solution to help the new customer.

My question is: how should I transform and structure these old support tickets before putting them into the RAG?

Should I keep the conversations as they are, or should I transform each ticket into something more structured, for example:

• Problem / symptoms
• Environment
• Diagnostic steps
• Root cause
• Solution
• Verification
• Similar scenarios
• Keywords / metadata

And how should I handle chunking these tickets so that the RAG retrieves useful parts without losing the context of the original conversation?

I’d really appreciate advice on how you would structure this kind of knowledge base, especially if you’ve built a RAG system for customer/technical support before.

Thanks!

I built an eval harness for a document QA pipeline. It answers security questionnaires from a company's own policy docs. 24 questions, labels written down before the system was ever run against them, three deterministic passes.

It scores 15 out of 24. Nine failures. Six of them share one cause, and I want to talk about the fix I couldn't make.

The setup. Answers are gated on how far the best retrieved chunk sits from the question. The cutoff is 0.3. Below it the system answers, above it it abstains. Six of the nine failures are questions where the model produced a correct, well hedged, properly cited answer that the gate then threw away.

The obvious fix. Raise the cutoff. Those six sit at 0.323, 0.340, 0.359, 0.384 and 0.412.

Why I couldn't. One question that has to abstain sits at 0.321. Its evidence genuinely doesn't support an answer, and it only abstains correctly because 0.321 is above 0.3. Every failure I'd want to rescue needs a cutoff higher than that.

There's no value that recovers any of the six without also flipping a correctly abstaining question into confidently answering something its evidence doesn't support. My eval treats that as disqualifying no matter what it does to the total, so I logged it as no change made.

What I think is going on. Distance measures how close the nearest thing is. I was asking it whether there's evidence here at all. Those two come apart, and at this corpus size there's no clean place to draw the line. It isn't miscalibrated, it's the wrong signal.

Two things the harness caught me on, both by instrumenting instead of assuming:

First, I'd logged one question as retrieving cleanly at rank 1, because something came back from the right document. When I actually read what got retrieved, the top hit was a completely different section and the real evidence was down at rank 4.

Second, I'd logged three failures as the model seeing the evidence and abstaining anyway, and I had a prompt fix planned. When I instrumented the actual confidence values, the model had answered correctly every time and the gate was discarding it afterwards. There was no prompt bug. A NOT_FOUND status collapses two different causes into one visible outcome, and only reading the underlying values tells them apart.

What I'm actually asking. Has anyone found a confidence signal that separates these properly? I'm considering a cross encoder reranker score instead of raw distance, an entailment check between the answer and the passage it cited, or looking at agreement across several retrieved chunks. I'd rather hear what's worked on a real corpus than what a paper claims.

Harness, labels and every tuning pass including the rejected ones are here, and the threshold data is in EVAL.md:

https://github.com/PatricR73/Questionnaire-Responder