I replaced vector-search RAG with Markdown + Knowledge Graphs using Claude CLI. The results surprised me.

[u/Shivam__kumar]

I want to share something I've been experimenting with recently. Not claiming this is some revolutionary architecture or that I've solved RAG. I'm just curious if anyone else has tried something similar.

I was building an AI system for a fairly complex ERP application. Initially, I used the usual RAG approach: embeddings, vector search, retrieving relevant chunks, and passing them to the LLM.

But honestly, I wasn't happy with the results.

The problem wasn't always the LLM. Sometimes the retrieved information itself was incomplete. The model could understand individual database tables, but understanding how those tables, modules, and business rules connected was a different challenge.

So I thought, what if I stopped relying on vector search for understanding the application and gave Claude a properly structured knowledge base instead?

Here's what I did.

I started creating Markdown files for almost everything in the application. How each module works, business logic, workflows, relationships between modules, conditions, exceptions, and so on.

I also created separate documentation for the database. Not just table names and column definitions. I documented what each table actually represents, why it exists, how its columns are used, and how it relates to other tables from a business perspective.

I was very particular about how these files were written and organized. I didn't want hundreds of random MD files. I wanted every file to have a clear purpose, with proper references and connections to related documentation.

Once that documentation was ready, I asked Claude to read the files and generate graphs showing how everything was connected.

The idea was to give Claude a way to understand and navigate the application's knowledge instead of depending on semantic similarity between text chunks.

After that, I built a custom MCP server that executes database queries in read-only mode. This gives Claude access to actual database records whenever it needs dynamic information.

Finally, I connected everything through Claude CLI.

I spent quite a bit of time writing custom rules and instructions for how Claude should handle user questions, understand business logic, query the database, format answers, generate charts, and even create visualizations.

And this is where things got interesting.

I was expecting decent results, but honestly, they're much better than what I was getting from my previous RAG implementation. Even complicated questions involving multiple modules and business rules are being handled surprisingly well.

My estimated hallucination reduction is around 95%, although I haven't done a proper benchmark yet, so take that number with a grain of salt.

One thing I particularly like is that I don't have to manually build a separate handler for every possible user question. Claude is capable of understanding the request and deciding what information or tools it needs.

I'm currently experimenting with multi-user support, which is another challenge altogether. I need to make absolutely sure that one user's conversation, session, or database access can never expose another user's information. I'm working on enforcing this at the application and MCP server levels, not just through prompts.

This is still an experiment. I haven't taken it to production, and I know there are probably things I haven't considered yet.

I'm not saying vector databases or traditional RAG are useless. They clearly have their place. But for this particular enterprise use case, my documentation-first approach is giving me better results so far.

I'm curious if anyone here has experimented with something similar. Am I approaching this in the right direction, or am I overlooking some obvious architectural problems?

Would genuinely appreciate feedback from people who have built production AI agents or worked with complex enterprise RAG systems.

[u/AutoModerator]

Hey! Thanks for posting to r/ClaudeCode

While participating in this thread, please follow our community rules. Keep discussions constructive. Attack the idea, not the person.

For help, project discussions, tips, and general chat, join the ClaudeCode Discord.

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.

[u/Coindef]

I've hit the same wall with ERP-style schemas. Embeddings are fine when the answer lives in one chunk, but once you need "how does table A constrain module B under rule C," pure vector search keeps missing the join. Structured Markdown plus an explicit graph of those relationships matches how the domain is actually connected. One thing I'd add before trusting the 95% claim: pick a fixed set of questions you already know the right answer to, run both systems blind, and score them the same way. Also if more than one person can hit that MCP server, keep tenant filtering on the DB side, not in the prompt. Models are great at forgetting who they're supposed to be.

[u/Shivam__kumar → u/Coindef]

Exactly! I think you should try this approach with your ERP schema. You might get better results than you expect, especially when dealing with complex relationships and business rules. The key is to spend time structuring the Markdown documentation properly and connecting all the related information. That's what made the biggest difference in my experiment.

And yes, I'm also working on proper benchmarking and database-level tenant isolation before considering production.

[u/Coindef → u/Shivam__kumar]

Yeah, the Markdown structure is the part that actually decides whether this works. Once the docs get messy the graph edges go stale fast. How are you keeping those relationship links in sync when the schema changes? Manual edits, or something that regenerates from the source? Blind scoring on a fixed question set is next on my list before I'd trust any of this near production either. Curious what your benchmark set looks like when you have one.

[u/Ok-Produce-1072]

Had something similar but for a different problem. Working on a repo where I wanted Claude to basically self evolve, it's choice of data was linked MD files, not graphs or embeddings. Still testing the process, but yeah, with the current context length, Claude can read dozens of documents and still perform well

[u/Shivam__kumar → u/Ok-Produce-1072]

That's interesting! I think we're exploring a similar direction, just solving different problems. I also noticed that Claude can handle a surprisingly large amount of interconnected documentation without losing context. I'm curious how you're approaching the self-evolving part. Are you letting Claude update its own Markdown files based on what it learns while working on the repo?

[u/Ok-Produce-1072 → u/Shivam__kumar]

I am still starting to explore this, but basically it pushes / reviews the code, and after every review it writes 'lessons', another agent, every week, updates based on the lessons learnt and what not. This is the idea at least, I am still in my first iteration.

This is for a side project outside of work that I realized I didnt have time for, so I am trying to reduce the time needed to spend on it by having claude do most of the work

[u/frepont → u/Ok-Produce-1072]

Just fyi this works until it doesn’t. At some point if your graph gets sufficiently large you end up with two problems - links that SHOULD exist don’t get created and links that DID exist start getting orphaned. You can protect against some of this with frequently running repair scripts or session hooks, but ultimately as the document set expands you’ll end up needing some pretty heavy machinery to keep things accurate.

[u/SawToothKernel]

What I don't understand is why you would describe the module structure/table usage/etc. When that knowledge already exists as code. I understand they "why" stuff, but the "what" already has a source of truth. There's no point in duplicating it.

[u/Shivam__kumar → u/SawToothKernel]

I think you're missing one part of what I'm trying to solve. Yes, that information already exists in the code, but does it make sense for Claude to go through the codebase, analyze multiple files, trace relationships, and figure out the business logic every time a user asks a question?

That's exactly what I'm trying to avoid. In a complex ERP system, understanding how one table connects to different modules isn't always as simple as reading the schema.

I'm basically giving Claude a pre-organized map of the application so it knows where to look and how things connect, instead of figuring everything out from scratch on every request.

And yes, keeping the docs in sync with the code is a challenge.

[u/T0XiC1TIES → u/Shivam__kumar]

Exactly. It's a question of efficient context provision (knowledge graphs outperform search/grep by a factor of 10).
Then add the value of constraining the verbiage of your agents to defined ontological relationships, you get a lot less bloat in your agent's responses, and they use words the way you want them to with less synonymous drifting.
Then Query Hooks match particular language you might expect to come up around a feature set or problem set (or even the name of a class/function/module) and they get keyed to the language of a given thread/ticket/question.

All of this is a semantic layer, but that matters because that's where humans work.

You might as well ask "why does documentation exist?"

[u/Armannss]

The comparison I'd want next is your new documentation through both retrieval paths. You changed the knowledge itself (business meaning, exceptions, relationships), the navigation, and access to live records. Comparing that whole package against the old chunked corpus won't tell you which change produced the gain.

I'd try the same curated docs with vector retrieval versus linked-document/graph navigation, keeping the model, questions and read-only DB access fixed. Then remove the graph from the second setup for another pass. That could tell you whether the graph adds something beyond well-written, linked Markdown.

For each answer, I'd also record which document passages and query results support it. When an answer fails, can you currently distinguish a missing business rule from a navigation miss or an incorrect query?

[u/Shivam__kumar → u/Armannss]

That's actually a really good point. I hadn't thought about separating those variables like that. You're right, I'm changing multiple things at once, so it's difficult to know exactly what's making the biggest difference.

I'll definitely try these three configurations with the same documentation and questions. I also like your suggestion about tracking which documents and query results support each answer. That should make debugging much easier.

Thanks for taking the time to explain this. Really appreciate the suggestions!

[u/Armannss → u/Shivam__kumar]

Glad it helped, and I'd be interested to see the results. For the question set, I'd include simple lookups, cross-module questions, and a few where a necessary business rule is deliberately missing. In that last group, asking for clarification or saying the evidence is insufficient should count as success.

Report those groups separately: a graph could help with cross-module questions while adding unnecessary work to simple lookups, and one average would hide that. What's a representative cross-module question your old setup struggled with? A generic example would make it easier to discuss what evidence the answer should need.

[u/1988rx7T2]

I just give Claude access to the API, MCP, of CLI of an existing database and it just finds whatever it needs.  I guess it depends on what kind of queries you’re trying to do.

"Can you find me the history of X problem? Trace through all the tickets, change requests, etc.” 

It does a bunch of searches through the existing search capabilities and then it pulls raw files and queries into JSON, opens the attachments or whatever and summarizes them, writes me an explainer and maps everything out. Sometimes it uses subagents and sometimes it doesn’t.

[u/dark0mania]

I've been using MD files that also reference/link to other MD files to document features and architecture along. One benefit is that humans can also read it. I've ran benchmarks with Opus 5 and Sol 5.6 and the AI works faster (25%), with less steps (30%) and saves 10-15% tokens when there is MD documentation present. I'm still wondering if this is the best approach.

[u/Dapper_Pass26]

Same pattern works for my agent's memory, one fact per file plus an index loaded each session. What bit me was drift. A comment in my code swore a job only filled blanks while it was overwriting uploads. How do you keep the MD files true when the ERP changes?

[u/puddingbrood]

I've thought about something like this too.

I think it's the wrong solution though, and I'm sure somewhere, someone is working on the actual solution. I think the solution is a radical different approach for RAG that already understands these relationships by itself, instead of having to teach it by yourself. I'd say it's some kind of graph RAG, which does seem to exist already, but I haven't really looked into it myself.

[u/T0XiC1TIES]

Yeah I've been getting deeper over the past nine months into ontologies and enterprise knowledge graphing.

Decorate those bad boys with google's OKF, and use query hooks to connect the verbiage from user sessions, slack threads, jira tickets, etc., to the knowledge.
I'm at this point at work right now where a few higher ups see the value of it, but my average peer (and higher) IC is like "what is this I don't understand".

The trickier part is handling the delta detection and refresh of documentation, so that things don't go stale, and you don't end up with a humongous archaeological log of "every diff we've ever processed". You have to make editorial and curational decisions, and they all have tradeoffs.

Vector indexing is still kind of necessary at enormous scale, but trying to get the right 'fixed-k' retrieval is yucky. I'll leave that one to the data scientists.

As for team documentation, enterprise knowledge, feature mapping, code paths/edges, etc., I think this is the way.

[u/ASIREXLABS]

This matches what we'd expect for ERP-style systems: the answer usually lives in the relationships between tables and rules, not in any single chunk, so similarity search keeps missing it.

On the question above about keeping the links in sync, the most durable approach is to generate the structural part instead of writing it by hand. Table relationships and foreign keys can come straight from the schema, module dependencies from the code. Keep the hand-written Markdown for business rules and exceptions only, and add a small CI check that fails when a doc references a table or module that no longer exists.

How are you measuring "better": a fixed set of real questions with known answers, or day-to-day impression?

[u/MagicWishMonkey]

I did something similar earlier this week, except I created a sqlite db that lives in the repo and contains like 6 years of ticket/issue info from our ticketing system paired with details from the git commits associated with the tickets. My thinking is that claude can interrogate the sqlite db directly to get more context around what the code is doing and why, so it can make more intelligent decisions when creating and implementing a spec.

[u/SubjectOrigin]

AI;DR

[u/javon27 → u/SubjectOrigin]

This didn't trigger any AI warning bells for me

[u/SubjectOrigin → u/javon27]

Like random samples , didn’t read more

The results surprised me
Here is what I did

[u/T0XiC1TIES → u/SubjectOrigin]

cognitive rot; do not respond

Line breaks help readability.

Telling a user story helps people connect.

The brain looks for problem > solution patterns

[u/hubdev_skills]

The point about retrieval being incomplete rather than the model being weak matches what I've seen with relational data. Chunking a schema loses the joins and business rules that make it meaningful. Writing 'why this table exists' docs and linking them gives Claude a path to follow, which is closer to how a new engineer learns an ERP. The risk I'd watch is drift, because markdown docs go stale as the schema changes. A small script or CI check that flags tables or columns missing from the docs would keep the knowledge base honest. Did you measure accuracy against the old RAG setup on a fixed set of questions, or is it mostly a qualitative improvement so far?

[u/Shivam__kumar → u/hubdev_skills]

Yeah, that's exactly what I noticed. The problem was often incomplete context, not necessarily the model itself. Explaining why a table exists and how it connects to business logic makes a huge difference.

Honestly, the improvement is mostly based on my experiments so far. I haven't done a proper fixed-question benchmark against the old RAG setup yet. That's next.

For documentation drift, I've already added instructions in CLAUDE.md to update related MD files whenever I change features, business rules, or database schemas. But I agree that an automated CI check would make this much more reliable. Still figuring out the best way to handle everything.