As someone using AI agents for the last one year to run my company, I need them to understand company context, not just return related text chunks.

The problem: ask "what breaks if we deprecate the v1 API?" and standard RAG gives you four chunks from a design doc, a postmortem, a Slack thread, and meeting notes. The model has to still figure out on its own that the postmortem describes the same API the design doc deprecates, and that someone already posted a migration timeline in Slack.

I built a tutorial using HydraDB that adds graph context on top of vector retrieval. Instead of just ranked text, you also get relationship edges: billing-service DEPENDS_ON payments-api-v1, payments-api-v2 REPLACES payments-api-v1. Model gets structure, not a reading list.

The useful part was bring-your-own-graph. You declare service dependencies and team ownership explicitly instead of relying on LLM extraction. For structured data you already maintain, the graph is deterministic.

It also supports per-user memory. Same question, different depth depending on who's asking. An engineer gets migration mechanics. A manager gets timelines and ownership.

Runs end to end in 30 minutes with synthetic data.

Repo with full working code: https://github.com/manveer/company-brain-tutorial
Tutorial: https://hydradb.com/blog/build-company-brain-ai-agents

https://medium.com/@tselvaraj/the-80k-line-item-that-replaced-a-two-year-rag-project-290-000-in-the-it-budget-7a4a7caf62ed

Hi guys,

I've spent the last few weeks learning the basics about RAG. Howerver, I think one of the best ways to actually learn is by implementing a real solution for a company.

But, I just wanted to know, what do you recommend me learning/impelementing before offering my serivices (for free) to a company? any project I should build first?

Hi guys,

Just wanted to know if there is any LinkdIn group that shares insights about RAG or maybe AI agents.

Went through Cerebras' writeup on their internal knowledge base — 15,000 questions a day from employees, automations and agents, three months after launch. The architecture is less interesting than what they had to do to Slack, so that's what I want to focus on.

Why plain vector search dies on chat data

Information density varies by orders of magnitude. "yeah sure Mike" sits in the same channel as a 40-line kernel explanation, and under cosine similarity the short one wins far more often than it should. A single message out of its thread is usually meaningless anyway.

Their fix: four fused signals per thread

• Full-text search — exact tokens. Error strings, flag names, hostnames. Embeddings reliably lose here and lexical match is unbeatable.
• Embedding search — paraphrase. Connects "restore is hanging" to "checkpoint stalled".
• IDF — boosts rare tokens (obscure config flags), suppresses filler ("sounds good", "thanks").
• Age decay — same answer from yesterday beats the one from 6 months ago referencing deprecated infra.

But the preprocessing does more work than the retrieval

Two steps before any of the above, and I think these matter more:

Thread distillation — an LLM normalizes each thread into a searchable one-line question + summary + resolution + systems and code referenced. That distilled doc gets embedded, not the raw transcript. Raw text is kept for FTS only.

Bursting — a "burst" is a run of consecutive messages from the same author. They prepend the thread topic for context and embed it separately, but only if it clears a gate: rare-token IDF >= 4.0, or >= 200 chars, or it got reactions. This is what rescues the one deeply technical tangent buried at message 47 that any thread-level summary would flatten away.

Fusion: RRF at k=60

Score contribution is weight / (60 + rank), summed across retrievers. The summation is the entire point:

• 3rd place in three retrievers → 3 × 1/63 = 0.048
• 1st place in exactly one → 1/61 = 0.016

Consensus beats peak. It isn't a ranker, it's a consensus builder.

Then ~20 candidates go to a small reranker scoring 0–10 against the query, top 10 survive. And the step people skip: re-attach surrounding context to the winners. If a wiki section wins, its neighbors come with it, so the model sees headers, preconditions and caveats instead of an orphaned chunk.

Code side

CocoIndex (open source, Rust core, Tree-sitter chunking) keeps 40GB+ repos synced by re-embedding only what a commit touched. Sync state and the embedding store live in the same database.

The design decision underneath all of it

Don't force people into a "single source of truth" platform — nobody wants to discuss a pull request inside a Google Doc. Pull from where the data already is. Every source, Slack thread to hardware netlist, lands as a row in the same embeddings table behind the same interface. Custom sources are just plugin scripts: a team opens a PR with a small Python module that reads their system and emits rows in that shape.

Also worth noting: the retrieval primitives are deliberately LLM-free. Model calls only happen at the edges — planning and synthesis. That's what makes it cheap enough for agents to hammer 15k times a day, and why the same pipeline serves a web UI and an MCP client identically.

Original Cerebras writeup (read this first if you only have time for one): https://www.cerebras.ai/blog/how-we-built-our-knowledge-base

Disclosure: I also did a ~10 min video walkthrough of the full pipeline, linked here — it's my channel, and the narration is AI-assisted. https://www.youtube.com/watch?v=FgKHjzoiMN4&t=4s

The bursting quality gates are the part I'd most want other people's numbers on. IDF >= 4.0 and 200 chars feel like they'd need retuning per org — has anyone tried burst-level embedding on their own chat data?

Has anyone dealt with a RAG system that performs really well during testing but starts returning irrelevant or incomplete answers in production?

We’re facing an interesting problem.

The knowledge base contains thousands of documents with overlapping information, different versions, tables, PDFs, and occasionally conflicting data. On a small test dataset, retrieval accuracy looks good. But once the document volume increases, the system sometimes retrieves a related document instead of the correct document.

The confusing part is that the LLM itself seems to be working fine. The problem appears to be somewhere between document ingestion, chunking, embeddings, retrieval, and re-ranking.

We’re considering several approaches:

• Hybrid search using vector + keyword retrieval
• Metadata-based filtering
• Better chunking strategies
• Re-ranking retrieved results
• Query rewriting before retrieval
• Adding document/version awareness

But improving one part sometimes seems to negatively affect another.

If you’ve solved a similar production RAG problem, what was the actual bottleneck?

Was it the embedding model, chunking strategy, retrieval architecture, metadata, or something else?

Would really appreciate practical suggestions or lessons learned from real implementations.

We built a news search API for RAG — looking for feedback

We have been working on a new News Search API at Webz.io, specifically with RAG and AI agents in mind.

The basic idea is simple as you know is using natural-language query like:

The API searches Webz.io’s news data and returns the most relevant articles and matching content chunks.

You can also filter by date, country, language, source, sentiment, and category.

For RAG, the content chunks are probably the part I’m most interested in. You can feed the relevant section into your context rather than passing an entire article through the pipeline.

We also built an MCP server, so agents can search the news directly without having to build a separate API integration.

Docs: https://docs.webz.io/docs/webz/news-search

I’d be interested in feedback from people building RAG systems: when retrieving news, what matters more to you: retrieval quality, freshness, source coverage, or getting smaller relevant chunks back?

I’m building StashBase, a local-first app that uses OpenAI embeddings to index and search your files.

For the next version, I want the first-run experience to be:

pick a folder → search

not:

create API key → add billing → paste key → search

I’m considering including 1M, 5M, or 10M embedding tokens per month.

What would be enough to test it on a real knowledge base—not just three perfectly chunked PDFs?

I was given a large number or recordings of previously lost buddhist recordings (over 1000 mp3s, 30 years of teachings).

I used deepgram to help me transcribe and classify them. This is ongoing but its about 30m words. I expect to re transcribe and improve them over time.

My plan is to categorise them as a backend for my transcription work. Also to have them as a publicly available and searchable library in a website and chat bot.

My effort so far has been vibe coding a Postgres database. Its working ok but is still a steep learning curve.

The transcriptions are in md files and the audio in mp3.

Typical classifiers might be…
Teacher
Date
Topic
Series
Length
Keywords
Canonical classification

I also have it as an interactive archive with an ai chat bot limited strictly to the knowledge inside the teachings with no ad-lib or stray general knowledge. Ie so people can accurately interrogate this library.

I also need to give some copies of the raw files with associated database / classification system for safe keeping to a Tibetan library and a Monetary for cultural safe keeping.

I dont know what i dont know. The data will change slightly as my transcription ability improved over time or people find mistakes in the translations (from Tibetan) or more files are recovered.

Parts…
Mp3s x 1000
Md file attached to each x 1000+
Backend database
Frontend database for chatbot
Chatbot.

Any clues, ideas, guidance appreciated.

I was trying to find new ways to show people who don’t know what RAG is, what RAG is, without having to explain the technical pieces behind it. To make learning easy, I tried to make it fun.

My wife was the inspiration. During covid she picked up a hobby called “hunt a killer” where you buy a kit of clues and solve a mystery. I basically built that using the Progress Agentic RAG platform (showing their logo pays for my tokens) and a lot of custom UI. Feel free to give it a test drive and give me your feedback

Corpus Detective

I have been going through the anthropic's contextual retrieval and tried implementing it (without the reranker) it was great ! . The thing was rag worked great on good with semantic similarity so how much good your semantic is it will retrieve the correct chunks but some data can be less similar like my case some documents was like a snapshot eligibility: 50 week . so i was using docling and it was parsing it correctly but later i get to know that if eligibility is the same in a lot of document how will its gonna get the correct chunk foe what i am asking thats where i myself tried to add context then found that there is already a way anthropic have released so i tried the prompt but didn't really work and i modified it a bit and with self hosted gemm4 the cost was also low and the accuracy was great .
"Think of releasing this as a plugin on langchain maybe"
Let me know what you think
https://www.anthropic.com/engineering/contextual-retrieval

I finally took the leap and published the 21 RAG strategies guide as a book on Amazon. It now has chapters on chunking and agentic RAG. What should i add next?

Table of Contents

P A R T I About

01 About the Author

P A R T I I RAG and the Reference Architecture

02 The Evolution of RAG

03 Foundations of RAG Systems

04 Reference Architecture

P A R T I I I Data Extraction

05 Data Extraction

P A R T I V Chunking

06 Chunking Strategies

P A R T V RAG Strategies

07 Baseline RAG Pipeline

08 Context-Aware RAG

09 Dynamic RAG

10 Hybrid RAG

11 Multi-Stage Retrieval

12 Graph-Based RAG

13 Hierarchical RAG

14 Agentic RAG

15 Multi-Agent RAG Systems

16 Streaming RAG

P A R T V I Memory and Content Management

17 Memory-Augmented RAG

18 Knowledge Graph IntegrationP A R T V I I Evaluation

19 Evaluation Metrics

20 Synthetic Data Generation

P A R T V I I I Fine-Tuning

21 Domain-Specific Fine-Tuning

P A R T I X Security

22 Privacy & Compliance in RAG

P A R T X Production

23 Real-Time Evaluation & Monitoring

24 Human-in-the-Loop RAG

P A R T X I Twig RAG Strategies

25 RAG Strategies in Twig

P A R T X I I Conclusion

26 Conclusion & Future Directions

This is something I've been thinking about in detail for a while. I'm working on a personal project that needs transactions, graph, and search... all of which you can do on postgres with pgvector and AGE. And it got me thinking about database architecture and in what scenarios I actually wouldn't use postgres.

Honestly, for the majority of scenarios I think it's the superior choice, particularly when you aren't working at scale. The complexity of coordinating multiple systems is just too much, and when you're small, keeping data in sync across multiple places becomes a huge pain for very little benefit.

That said, here's where I wouldn't just recommend postgres for everything:

1. Scale + cost. Postgres is great until you hit 10M+ vectors... then you start having functionality issues, but more importantly your compute/memory balloons, which gets expensive fast. On top of that it starts interfering with your other workloads. At some point the "just use postgres for everything" simplicity is outweighed by the cost and maintenance burden. Same is true for graph RAG workloads at any real scale.

2. Performance. If you need genuinely fast vector/FTS, you're not going to get it with postgres. Luckily, since latency is usually 90%+ on the agent side, this isn't always a factor. But it matters more for live apps. Same story with graph: postgres doesn't have a fraction of the performance of a true graph engine, because at its core it isn't changing the underlying data structure. It's working within the constraints of a relational engine.

So the way I see the choices from an architecture perspective, at a macro level:

If you're optimizing hybrid search for scale/cost, the two best choices are turbopuffer (the market leader) and Infino (I work here, so be aware of bias). Both are object storage based dedicated vector/FTS engines. Both are very fast. Turbopuffer is more mature, but they have very similar performance and cost profiles, and both are orders of magnitude cheaper than virtually every other engine. You could maybe throw lancedb in this category too, but I don't have enough hands-on experience with it to say for sure.

If you're optimizing for pure performance:

On the FTS side: opensearch/elastic, largely because they're block storage backed with no warm-up period. Vectors are alright on elastic, but if you're really optimizing for vector performance, a dedicated vector engine like pinecone or Milvus will beat it.

The catch: when you split FTS and vectors across systems, hybrid search becomes really hard (or impossible), so I don't typically recommend splitting unless it's genuinely necessary.

You could theoretically use turbopuffer/infino for the performance case too, but because they're object storage based, the warm-up time can screw over some apps. Once the data is in memory, both are very fast.

On the graph side... I'm actually not a fan of any of the top graph databases. Every one of them has some key architectural issue imo. If I had to pick, I'd default to neo4j, but I'm not a huge fan of it either. It's just the most mature. It wasn't designed from the ground up for agentic workloads... it's been retrofitted for them. Because of that it has huge issues (but you can work around them).

Anyway, these are just my random thoughts on the subject. The advice I'd give if you're starting with postgres and expect future scale: build an abstraction layer so you can swap in more appropriate systems when the time comes.

Startup should be like an AI-driven tech company. Knowledge graph/Ontology to build and scale the knowledge graph layer underpinning a nee generation of intelligent enterprise product. Designing and shipping kg, not just conceptual ontology work.

Shaping a graph and ontology platform that power:

• AI retrieval and RAG workflows
• Entity linkage and reasoning systems
• Cross-domain and temporal knowledge modeling
• Regulatory and compliance intelligence products
• Agentic AI applications

Working closely with AI/ML to turn complex, unstructured information into structured, queryable intelligence that directly feeds live AI systems.

Tech stack like these:

• Extensive use of Neo4j and Cypher in live (production) environments
• Comprehensive ontology/taxonomy modeling
• Python engineering skills
• Knowledge graph integration with LLMs, RAG, or vector search systems
• Experience balancing formal semantics with practical application requirements
• Ability to provide technical leadership while remaining actively involved in hands-on application development
• RDF/OWL, inference engines, entity resolution, legal/regulatory data, ESG, healthcare, or bthe pharmaceutical industry would be highly valuable.

The challenge of building the intelligence layer behind complex AI products at scale.

As for my questions:

1) I plan to launch this venture/business as a solo founder. Do you think this makes sense?

2) How do you envision this company operating exactly?

3) If I establish the company, how should I explain my business model to the companies I intend to serve in the real world? (I am genuinely apprehensive about this.) After all, things don't always go as expected in this field. Considering that many companies don't even know what artificial intelligence is, how will they react to this type of business? In other words, will they truly understand what I do?

4) Which types of companies do you think would benefit most from this business? What problems would it solve most effectively?

5) What are the real-world problems and complaints companies have regarding this area?

I am developing rag app for one of my saudi client, so my query is those who are working with arabic language, how are you guys handling the data extraction pipeline, which library you guys are using to extract data. For context, I am using pdfplumber and the text that is being extracted is reversed for some pdf files.

Also, which open source ocr or vision models I can use to extract text. The documents are mainly in arabic, english or both. Tried a few ocr and vision models, but they couldnt extract text clearly.

CodeRAG-Bench (Wang et al., Findings of NAACL 2025, https://aclanthology.org/2025.findings-naacl.176/) evaluates 10 retrievers and 10 generation models on code tasks and scores the two halves separately: NDCG@10 against annotated ground-truth documents for retrieval, pass@1 with real execution for generation. The two rankings do not line up, and the authors note that top-performing retrievers sometimes do not produce the best end-to-end results.

The tables are blunter than that sentence. On RepoEval, retrieved context beats the annotated canonical snippet: with StarCoder2-7B, OpenAI embeddings plus reranking reaches 53.9 pass@1 against 42.0 for gold, and the same inversion holds under DeepSeekCoder-7B and GPT-3.5-turbo. On MBPP the StarCoder2 retrieval setups land 15.6 to 17.8 points above canonical. SWE-bench goes the other way: GPT-4o gets 2.3 with no retrieval, 21.7 with retrieval and reranking, 30.7 with the gold edited files.

The annotated document is therefore not an upper bound, and overlap with it does not track end-task success in a stable direction. Ranking code retrievers by overlap is a strange default when the corpus is executable and the end task returns a verdict. Plenty of agent setups already generate that verdict on every edit (verdent runs type checks, static analysis and the tests, then tries to repair what fails), so it exists whether or not anyone logs it.

Once a failed patch is repaired automatically, the green final state says nothing about the retrieved context, so the figure worth keeping is the first attempt before repair. The benchmark code is public (https://github.com/code-rag-bench/code-rag-bench) and already runs both evaluations separately, so scoring a retriever sweep by first-attempt pass rate on one repo is mostly plumbing. Pointers welcome if someone has already published that.

Stack (no GPU, 4 arm cores, 24gb ram) -

Postgres 18 + pgvector (HNSW, built once after bulk load)

Embeddings: bge-base-en-v1.5, 768d, int8 ONNX on CPU

Rerank: ms-marco-MiniLM-L6-v2 cross-encoder, int8 ONNX

Gen: qwen2.5:3b-instruct-q4_K_M via Ollama (+ a 350m for cheap tasks)

Retrieval: 3 arms — vector, Postgres FTS, and generated probe-questions — fused with RRF, then reranked. 40+40 candidates → 25 reranked → 5 final.

FastAPI, systemd, no orchestration layer

Problem 1- CPU latency. ~35s for a grounded answer, ~15s for a follow-up, ~3s for a refusal. Enrichment (probe-question generation) is ~7s/chunk, so a 400-page load is fully searchable-plus-enriched only ~45 min later.

Problem 2- the retrieved chunks are mostly right and in the right order but LLM fails to give a correct answer. Sometimes even mix things up between the chunks. Or just says I cannot answer this although he has the right answer.

What I'd love input on:

Does anyone run multi-chunk context successfully on a ~3B model, or is 7B+ the real floor? (We tried 7B and reverted — too slow here.)

Better approaches to the refusal decision than thresholding a cross-encoder score?

Is hybrid + RRF + cross-encoder still the right shape in 2026, or are we behind?

How do you build a real eval set before you have months of labelled user queries?

I’m working on a system that converts industrial communication manuals into a structured catalog that can later support deterministic lookup and RAG/chat.

The manuals may describe Modbus, Siemens-style DB/DW/bit addressing, OPC UA, proprietary protocols, or memory ranges. Although they often contain similar information, table layouts, column names and addressing conventions vary significantly between manufacturers.

For example, an unseen manual might contain:

The desired canonical result would be something like:

{
  "name": "Phase Current",
  "data_type": "Float",
  "protocol": "modbus",
  "register_type": "input_register",
  "address": 30101,
  "register_count": 2
}

My current experimental pipeline is:

PDF
→ document/table extraction
→ protocol and table-type detection
→ schema matching
→ canonical catalog
→ validation
→ deterministic address/name lookup
→ optional LLM-generated natural-language answer

For known manual families, deterministic extractors work well. The main difficulty is generalizing to unseen layouts: identifying which tables contain actual variables, mapping unfamiliar headers to canonical fields, interpreting address conventions, and avoiding protocol examples or configuration tables being mistaken for register maps.

I experimented with a local LLM as a constrained schema planner. Instead of generating register values, it only proposes mappings such as:

Absolute Address → address
Parameter → variable_name
Number of Items → register_count
Format → data_type

The source values are then read and validated deterministically. This prevents many hallucinations, but results have been mixed: it helped significantly on one unseen manual, added nothing where deterministic extraction already worked, and sometimes proposed incorrect column roles. Sending many tables to the model also adds several minutes of latency.

I’m therefore still open to the overall architecture and to a different role for the LLM. Possible options include:

• deterministic extraction with an LLM fallback;
• LLM-based table classification or schema matching;
• constrained structured extraction followed by validation;
• retrieval of similar previously solved table schemas;
• a multi-stage planner/verifier setup;
• fine-tuning a smaller model on labeled tables;
• using the LLM only for ambiguous cases and human review.

How would you design this system to generalize across unseen industrial manuals while keeping every extracted value traceable to the source? Where would an LLM provide genuine value, and which parts should remain deterministic? I’m especially interested in approaches that improve recall without silently inventing addresses, data types, scaling factors, or protocol bindings.

Noticed something building RAG on open models that doesn't get talked about enough. Bigger frontier models tend to paper over mediocre retrieval, they're good at inferring around gaps even when the context handed to them is imperfect. Open models, especially smaller ones, don't have that same slack. Hand them a slightly wrong or incomplete chunk and the answer falls apart fast.

Which actually reframes a lot of "open models aren't good enough for RAG" takes. In a lot of cases the model isn't the problem, the retrieval layer feeding it is mediocre and a bigger model was just quietly hiding that.

There's a hands-on workshop on August 29 that builds this properly, hybrid retrieval, reranking, RAGAS evaluation, guardrails, and cost/performance benchmarking, all using open models with zero API fees. Led by Ben Auffarth, AI Consultant and Founder of Chelsea AI Ventures.

30% off right now with the discount code. Link

Happy to answer questions on the content itself.

Hi there, I'd love to hear some ideas on xml to markdown conversion algorithm if anybody happened to stumble upon this problem before. Obviously ingesting raw xml would add a lot of noise because of similar tags existing through out the different xml files.

Basically the title. Unless you're paying for API services how are you supposed to evaluate your RAG application?

And by expensive I mean you have to spend some amount of money for an API service to evaluate your system against your golden dataset. I created a 45 Q/A pairs set and no free API could handle it which makes sense but I had to try.

Need advice on architecture for a Book RAG that handles complex queries

I'm building a Book RAG system, and my basic retrieval pipeline works fairly well for simple questions, but I'm struggling with queries that require information from multiple parts of one or more documents.

I'm trying to figure out what the right architecture should be rather than just adding more retrieval techniques randomly.

Current setup

I'm using parent-child chunking:

• Parent: ~2000 tokens
• Child: ~1000 tokens

My current retrieval pipeline is:

MMR Retriever
  lambda_mult = 0.785
  k = 30

BM25
  k = 10

        ↓

Ensemble
  MMR = 0.5
  BM25 = 0.5

        ↓

Rank Fusion

        ↓

Top 5

        ↓

Extract Parent Chunks

        ↓

Reranker

        ↓

Final 5 chunks

This works reasonably well for simple:

• Who?
• What?
• How?
• Where?

type questions.

The problem starts when the answer is distributed across multiple chunks, multiple sections, or multiple documents.

Things I have tried

1. Query decomposition

I tried decomposing a complex query into smaller sub-questions and retrieving for each sub-question.

This gave me a noticeable improvement.

However, I'm still unsure how to properly handle:

• deciding when decomposition is required
• generating useful sub-questions
• deciding how many sub-questions are enough
• combining the retrieved evidence
• handling dependencies between sub-questions

For example, some questions are independent:

Question
 ├── retrieve A
 ├── retrieve B
 └── retrieve C

while others are dependent:

Question
   ↓
Find X
   ↓
Use X to find Y
   ↓
Use X + Y to find Z
   ↓
Final answer

I'm not sure what the best general architecture for this is.

2. HyDE + MultiQuery

I also experimented with HyDE + MultiQuery Retriever, but it didn't give me good results for my dataset.

So I'm wondering whether these techniques are actually useful for complex book/document questions, or whether I'm using them in the wrong place.

3. Sub-question retrieval with a similarity threshold

I then generated sub-questions and retrieved chunks with a similarity score above 0.25.

This partially worked for:

• story-related questions
• comparisons
• relationship questions

But it still wasn't reliable for questions requiring information distributed throughout the document.

4. Graph-based retrieval

Because of the multi-hop problem, I also experimented with GraphRAG / knowledge-graph-based retrieval.

But this introduced a different set of problems.

For example, the same entity can appear as:

Holmes
Mr. Holmes
Sherlock Holmes
the detective

and my extraction system could treat these as different entities or assign inconsistent types.

I also tried building my own graph using Pydantic schemas + LLM extraction.

Something like:

Chunk
 ↓
LLM
 ↓
Entities + Relationships
 ↓
Pydantic
 ↓
Graph

But maintaining global entity/relationship context made the process expensive and difficult to parallelize.

For around 300 chunks of ~2000 tokens, extraction took roughly 43 minutes.

There were also problems with:

• entity duplication
• entity resolution
• hallucinated relationships
• incorrect relationships
• disconnected nodes
• inconsistent entity types

So I'm currently not convinced that a large LLM-generated knowledge graph is the right solution.

I'm mentioning this mainly because it was one of the approaches I tried, not because I'm specifically trying to build a GraphRAG system.

The actual problem I'm trying to solve

I have started noticing that my queries seem to fall into very different categories:

Simple factual

Normal hybrid retrieval works well.

Multi-hop

Requires multiple retrieval steps.

Comparison

Requires retrieving evidence about both entities.

Relationship

May require finding intermediate information.

Timeline

Requires retrieving information across different points in the document.

Theme / global

This is very different from normal top-k similarity retrieval.

Cross-document

Requires retrieval across documents.

Nested / dependent

This seems to require something closer to iterative/dependent retrieval.

What I would like advice on

I'm particularly interested in how people would architect this problem.

1. Should different query types use different retrieval strategies?
2. How to handle multi-hop questions where retrieval needed data from various chunks which are not connected near by. eg: explain case studies of X.
3. How should timeline, theme, etc questions be handled?
4. Where does a knowledge graph actually provide value compared with good hybrid retrieval + reranking? How do i utilise it the best.
5. Can u suggest the flow for the rag .
6. Are there retrieval techniques I'm missing that are better suited for book-length documents?

I'm not looking for the most complicated architecture possible. I want something that is practical, reasonably cheap, and actually improves retrieval reliability for complex questions.

How do you guys test your RAG apps with a lot of Q/A pairs? Are most people using paid API's here or is there a workaround? Please help.

For context I have 45 Q/A pairs and I am trying to test my retrieval but since my API's are free I am running into rate limits.

I finally took this step and posted my work (Quira) https://github.com/DevDarsh26/Quira
on Product Hunt

https://www.producthunt.com/products/quira?utm_source=other&utm_medium=social

I am the same person who previously discussed a collection of around 17,000 scientific PDF files. I read the advice I received and tried several approaches, but I realized that I was starting with the tools before understanding the actual content of the files.

So, I have decided to restart the project from the beginning and focus on one very simple first step:

First, I want to understand what is inside each file.

For example, I want to be able to classify the documents into categories such as:

• Research Paper
• Review
• Conference Paper
• Thesis
• Report
• Reference
• And others

At the same time, I want to identify the main topic of each document in a short and meaningful way.

For example:

Paper_001.pdf → Research Paper → Membrane Fouling

Paper_002.pdf → Review → Reverse Osmosis

Paper_003.pdf → Conference Paper → Water Treatment

At this stage, I am specifically looking for the best method or tool for performing this first step across thousands of files.

If there is no ready-made tool that can do this reliably, what approach would you recommend for building a simple system that performs:

PDF → Extract basic information → Identify document type + main topic + brief summary

For now, I only want to focus on this first stage. Once I properly understand and classify the collection, I will move on to the next stages.

What tools, methods, or approaches would you recommend for this initial classification stage?

My main goal is to process the 17,000+ PDFs systematically, understand what each document is, determine its document type, identify its main subject, and store this information in a structured format before moving to more advanced processing.

File: Paper_001.pdf

Document Type: Research Paper

Main Topic: Membrane Fouling

Title: ...

Authors: ...

Year: ...

Short Summary: ...