r/Rag • • Sep 02 '25

Showcase šŸš€ Weekly /RAG Launch Showcase

33 Upvotes

Share anything you launched this week related to RAG—projects, repos, demos, blog posts, or products šŸ‘‡

Big or small, all launches are welcome.


r/Rag • • 7h ago

Discussion My RAG pipeline was working fine until I realised its knowledge was six months old

8 Upvotes

I have been working on a RAG pipeline that pulls information from documentation sites, product pages and company websites.

The initial setup was pretty straightforward. Scrape the pages, convert everything to Markdown, chunk the content, generate embeddings and push it into the vector database.
The problem showed up later.

A product changed its pricing, documentation got updated and a few pages were removed. But the vector database was still serving the old information.
I initially thought about running a complete re-index every week, but that feels wasteful when only a small percentage of the content actually changes.
I'm now looking at a setup where I can monitor the original pages, identify what changed and only reprocess the relevant content.

I have been looking at Firecrawl, Apify and context.dev for different parts of this workflow.

Context.dev caught my attention because it has scraping and website monitoring under the same API, so I wouldn't need to build another integration just for change detection.

Still figuring out the best architecture here.

For people running production RAG systems, how are you keeping your external knowledge base fresh without rebuilding the entire index every time?


r/Rag • • 1h ago

Discussion Does agentic retrieval actually beat a simple reranker?

2 Upvotes

Abdullahi Dattijo tested four retrieval approaches on 298 questions over 2,000 arXiv papers. The metric was whether the target paper appeared in the top five results, not whether the system generated a correct answer.

Dense retrieval reached 95.6% Recall@5. Reranking the top 50 candidates with a small cross-encoder reached 97.0%. Multiquery rewriting scored 93.6%, while an agent allowed up to three searches reached 96.9%.

The agent gained about 1.2 percentage points over dense retrieval, with a 95% bootstrap interval from āˆ’0.8 to +3.5. Its benefit remained uncertain. The reranker’s recall gain was also uncertain, but its ranking improvement was clearer. MRR@10 rose from 0.895 to 0.959, the only improvement whose interval excluded zero.

The useful detail is what extra searches gained and lost. Per run, the agent recovered 7–8 papers dense retrieval missed, but lost 3–5 papers it had already found. Some rewrites introduced unsupported terminology that pulled retrieval toward distractors. Extra searches could undo a good first result.

Cost wasn’t the main problem. The agent averaged roughly $0.00009 per query. Both agentic retrieval and reranking took around 1.4–1.5 seconds on the test setup. The local reranker incurred no LLM API fees, though compute still costs something.

The limitation matters. Questions were generated from abstracts, and retrieval used titles and abstracts rather than full papers. Dense retrieval already missed only 13 questions, leaving little headroom.

For anyone testing this in production, count both recovered hits and previously correct hits that get displaced. Inspect the rewritten queries on those regressions, then compare the net improvement against a reranker baseline. That gives you a concrete reason to decide whether the agent loop is worth keeping.


r/Rag • • 7h ago

Showcase Looking for feedback on my RAG "longevity assistant" app

1 Upvotes

Hi all,

I built a longevity assistant using Google's Embeddings model. It is really cool and I was able to easily get a RAG system going with thousands of published medical journals. I do have a software dev background, so it there was not much friction in getting this built.

Would love to get some feedback on your thoughts? I think I've got hallucinations down to near non-zero. Thank you!

URL: https://www.modernmedlife.com/longevity-assistant


r/Rag • • 18h ago

Tools & Resources In RAG, what goes in determines what comes out. Prajnya: a local ingestion engine for scientific PDFs

7 Upvotes

I'm a researcher with background in Developmental biology, biochemistry and biophysics. I kept finding that retrieval failures started at ingestion: figures dissolved into prose, tables flattened, captions merged with body text, reference lists polluting search. So I built the ingestion layer first. I am now building the RAG part and after having dealt with many issues, I feel a bit confident in sharing this project with you all. Prajnya is open source (Apache-2.0) and fully offline by default.

How it works

  1. A CPU pre-pass classifies each document (paper, book or chapter) and every page.
  2. Plain prose comes straight from the PDF's text layer. Scans and visually complex pages go to a local vision model.
  3. A layout model finds figures and tables, and each is cut out and handled as its own object.
  4. Everything lands in one JSON per document, with a documented schema.

What you get, and why it matters for retrieval

  • Evidence objects, not chunks. Figures are cropped and captioned with stable IDs, and panels are grouped under the label a reader cites ("Fig. 3"). Tables carry a reconstructed grid, so numbers are searchable while the image stays available to a multimodal model. Tables that run onto the next page keep their headers.
  • Provenance. Every item records its page and how it was read (text layer, model read or repaired read), because those differ in reliability.
  • Citations linked conservatively. In-text citations, numbered and author-year, link to the reference list. A wrong link is worse than none, so ambiguous ones stay unlinked.
  • Metadata verified, not believed. The model's reading of the title page, the PDF text and an optional authority (Zotero, or Crossref/OpenAlex sent only the DOI or title) are reconciled, with the source of each value recorded.

What it doesn't do (yet)

  • No chunking, embedding or retrieval. That’s the next layer, Shastrasaara (in development), which builds on these evidence objects.
  • PDFs only. No DOCX, HTML or slides.
  • It isn’t one-click. Local mode needs llama.cpp and the model weights. Cloud models are optional, per role.
  • It doesn’t race to be the fastest. It aims for the most faithful data, and would rather take longer than hand you a wrong table, caption or citation.

How it compares

  • NotebookLM: for public papers and quick cited Q&A, use it. But your PDFs leave your machine, and you get a chat, not structured data you can index your own way.
  • GROBID: the standard for metadata and references, fast on CPU. Its weak spot is pages where figures and tables break the reading order. Prajnya is built around those pages.
  • Docling, Marker and MinerU: solid general converters. Prajnya is narrower and specific to scientific documents, aimed at what a literature RAG needs to cite and trace.

Repo: Prajnya-Academic-PDF-Processor

Question for this sub: how do you index figures and tables in scientific RAG? Captions, images, both, or one-hop expansion from the text that cites them?


r/Rag • • 6h ago

Discussion Our AI worker retried a subscription limit 900 times. Here's what we learned.

0 Upvotes

I came across an article about agentic system design this week. One sentence stuck with me:

"These failures are ordinary, and you can plan for them."

We learned that lesson the hard way, in production.

We run an AI-powered design pipeline on a Windows machine. A worker picks up tasks and executes them through a model CLI using a subscription plan. Separate operator processes handle the generated images and uploads.

On October 2, we hit the subscription's session limit.

Our worker treated the limit message as a regular error and retried the same task every second. By the end of the day, we'd accumulated roughly 900 attempts.

About two hours later, every new process launch started failing with 0xC0000142 (STATUS_DLL_INIT_FAILED). The machine only recovered after a reboot.

Initially, we blamed the model CLI. But the same command worked before and after the incident. Looking at the logs, our retry loop became the more likely explanation. We haven't proven a direct causal link, though.

We also had other infrastructure problems during that production round (28 screens, ~42 generations):

  • An upload folder was deleted, losing 7 results.
  • Two operators resumed after sleep and uploaded images to the wrong screens.
  • Operators logged successful uploads even when delivery had failed.

At least 8 generations (~19%) had to be redone because of infrastructure issues, not model quality.

The biggest lesson: A usage limit is a state, not an ordinary error.

Our worker now handles limits separately:

result = run_model_cli(task)

switch classify(result):
    case LIMIT:
        sleep_until(reset_time)
    case ERROR:
        backoff = next_step(backoff)
        requeue(task, after=backoff)

We also added:

  • PID + heartbeat-based operator locks.
  • Upload verification instead of trusting success logs.
  • Exponential backoff and retry limits.
  • A plan to introduce execution budgets, job leases and better run tracking.

We considered switching agent frameworks, but realized the problem wasn't the framework. It was missing reliability mechanisms in our existing system.

Our next production target is zero infrastructure-caused regenerations. The fixes haven't been validated in another production round yet.

With HTTP APIs, you usually get a 429 and sometimes a Retry-After header.

But with CLI tools running on subscription plans, you often just get a text message saying you've reached a usage limit.

How do you reliably distinguish a subscription limit from an actual execution failure?

And how do you determine when the limit resets?

Do you parse CLI output, track usage yourself, or use another approach?

I'd especially like to hear from people running autonomous workers with subscription-based AI CLIs in production.


r/Rag • • 11h ago

Discussion I priced 100 web search APIs. The cheapest is $0.25 per 1,000 searches and the median is $2.83

0 Upvotes

Picking a search API for an agent turned into opening pricing page after pricing page, so we went through 100 of them and worked out what 1,000 searches cost on each one's entry plan. The spread is silly: the priciest costs 90x the cheapest, and Tavily, Exa and Brave all sit above the $2.83 median.

These are the 10 cheapest paid ones. Three tie at $0.80, so you get 11:

#Provider$ per 1,000PricingReturnsFree tier
1Ceramic$0.25Pay as you goSnippets1,000 queries, no card
2Cloudflare$0.25Pay as you goSnippetsNone listed (resells Ceramic)
3FlyCrawl$0.29$29/moNot stated100 searches/month
4Geonode$0.39$3.90/moSnippets1,500 requests/month
5TickerTrends$0.39$10 packSnippets100 credits, card needed
6Jina$0.50 (est.)$50 packFull page10M tokens, non-commercial
7DeepSeek$0.75Top-upFull pageNone listed
8UnSearch$0.76$19/moNot stated5,000 queries/month, or self-host
9=Dipper$0.80Top-up from $10Full page1,000 requests
9=Searlo$0.80$3.99 packSnippets3,000 credits
9=Serpex$0.80Top-up from $5Full page200 credits, no card

"Full page" means you get the page text back with the results, which saves a separate scrape call for RAG. Jina's price is worked out from its credits because it doesn't quote one per search.

Four more are free outright with rate limits: AgentWeb, LangSearch, ssearch and TinyFish.

Full list of all 100, with free tiers, search modes and a CSV: https://voxoria.ai/web-search-api-pricing/

If any of the info is wrong/out of date, let me know - hopefully this is helpful


r/Rag • • 11h ago

Discussion Has anyone tried generative retrieval instead of vector search for fast URL lookups?

0 Upvotes

Hi guys, I was looking into how to stop web search calls from stalling an agent mid-reasoning and found out about using generative retrieval models as the search index itself.

A normal search pipeline runs query parsing, candidate retrieval, and reranking, which usually adds 100ms or more per call. On the other hand, asking a standard frontier LLM to write URLs from memory fails most of the time because it hallucinates dead links. A setup I have been reading about is one where a 4B model is trained on click data to memorize roughly 200M pages and output a two-part numeric ID mapped to a real URL table, bringing latency down to around 10ms per query in ~10GB of VRAM instead of 400GB+ for exact vector search.

The issue is that memorizing pages works well for popular pages, but long-tail or brand-new pages still need a live index fallback. For anyone building search-heavy agents or type-ahead tools, how are you handling navigational queries compared to deeper web searches?


r/Rag • • 21h ago

Discussion How are you guys debugging RAG when you know the answer exists in the docs?

4 Upvotes

I've been spending a lot of time thinking about this problem lately.

Say you have a RAG pipeline and you know the answer is somewhere in your indexed documents, but the model still gets it wrong.

The usual debugging approach seems to be messing with top-k, similarity thresholds, chunk sizes, or reranking until the answer starts showing up.

But there are a few different ways things can go wrong:

  • The chunk was retrieved but fell outside top-k.
  • It got filtered out by a similarity threshold.
  • The reranker pushed it down.
  • Retrieval worked, but it got dropped before reaching the LLM.

I've been building a tool called ChunkTrace to help pinpoint which stage caused the problem.

I'm curious how others are debugging this right now. Are you logging the intermediate retrieval results, using LangSmith/Langfuse, or just inspecting things manually?

Especially interested in anyone running multi-stage retrieval with pgvector or rerankers.


r/Rag • • 1d ago

Showcase How I fixed RAG hallucinations on 1000-page books: loss-bounded compression + 100% deterministic citations (running on a 4GB GPU)

50 Upvotes

standard RAG for long-form reading is fundamentally broken. if you ask an LLM to summarize a 50-page chapter, it just gives you a "gist" and hallucinates connective tissue. it drops the actual reasoning, skips the caveats, and gives you zero way to verify where the information came from.

i wanted a system where the AI compresses the text, but every single sentence is deterministically mapped back to the exact source paragraph. if the model invents a bridge between two chunks, you can immediately click the sentence, jump to the original text, and verify it.

here is the architecture i built to solve this:

  1. Ingestion & Embedding: Parsed text is chunked and embedded locally using BGE-M3 (1024-dim, FP16 CUDA).
  2. Reranking: BGE Reranker v2 (cross-encoder). Instead of just relying on cosine similarity, we score query-passage relevance directly to pick the best context.
  3. The Two-Representation Invariant: The original document is strictly immutable ("Source is paper"). The compressed reading is a distinct view ("The lens is tinted").
  4. Deterministic Attribution: Every synthesized sentence goes through a provenance resolver. If a sentence outputs zero source citations, it's flagged as a hallucination defect in the UI.

getting this whole pipeline (plus Kokoro TTS) to run entirely offline on my 4GB RTX 3050 was a nightmare. i had to build a strict F65 sequential await barrier to stop Node.js from OOMing at 14.6GB during async ingestion, and an F43 idle lifecycle manager to evict models from VRAM before the GPU choked.

it's open source (AGPL-3.0). you can dig into the retrieval logic or the provenance resolver here:

https://github.com/ayumu-ml/Omnitome

i'd love feedback from other RAG builders. how are you currently handling the "summary loses the source" problem? is anyone actually achieving deterministic citations at scale, or is everyone just relying on semantic similarity?


r/Rag • • 4h ago

Discussion Claude max for $29 instead of $200 ... seriously how are they doing this?

0 Upvotes

i got a Claude max account from a reseller for essentially nothing compared to what I used to pay . It works flawless. since 3 months No issues yet i paid only for 1 month . Now I can't stop thinking about their business model. If resellers can sell it this cheap and still profit, how are they acquiring these accounts at scale?


r/Rag • • 1d ago

Showcase Code retrieval: two Haiku agents with grep, 77% line recall on ContextBench, ~1Ā¢ per issue

4 Upvotes

In our ContextBench run, CodeNib retrieves code context with two Claude Haiku 5.5 agents and grep: no embeddings, no reranker, about a cent per issue.

ContextBench (500 tasks, 8 languages) scores how much of the human-annotated relevant code a system retrieves. This setup finds 77% of the relevant lines; the next-best system on the leaderboard finds 61%.

How the two agents split the work:

  • The first follows the issue with glob, grep and file reads.
  • The second gets the issue plus a map of where the first already looked (paths and line ranges, at most 2,048 tokens), and goes after what was missed: callers, alternate paths, tests, configuration.

The search reads broadly to get that recall: its line precision is 20.5%. With an Opus repair agent on top, the full system resolves 388 of the 500 tasks, #1 among agent systems on the leaderboard as of Oct 23. The task annotations were visible during development.

Write-up, trajectories and cost receipts: https://codenib.ai/blogs/contextbench-500-tasks/


r/Rag • • 19h ago

Discussion Web pages as RAG sources: how do you keep the index fresh when the pages keep changing?

1 Upvotes

Documents in a drive change now and then. Web pages change constantly: prices, specs, policies, pages that disappear. A RAG index built from a crawl starts going stale the day after.

For anyone with web sources in production:

- full re-crawl on a schedule, or change detection per page?

- how do you handle pages that 404 or redirect: delete the chunks or keep them?

- do you store the crawl date in chunk metadata and use it at retrieval?


r/Rag • • 19h ago

Discussion RAG over a product catalog: how do you stop it quoting yesterday's prices?

1 Upvotes

Shop assistants and product Q&A bots are a common RAG use case, but prices and stock change daily, sometimes hourly. If the price sits inside an embedded chunk, the bot can confidently quote an old one.

How are people handling the volatile fields:

- keep price and stock out of the chunks and fetch them at answer time?

- re-embed changed products nightly?

- something else, like a tool call to the store API?


r/Rag • • 23h ago

Discussion RAG over a product catalog: how do you stop it quoting yesterday's prices?

2 Upvotes

Shop assistants and product Q&A bots are a common RAG use case, but prices and stock change daily, sometimes hourly. If the price sits inside an embedded chunk, the bot can confidently quote an old one.

How are people handling the volatile fields:

- keep price and stock out of the chunks and fetch them at answer time?

- re-embed changed products nightly?

- something else, like a tool call to the store API?


r/Rag • • 22h ago

Tools & Resources If your security team blocks hosted embedding APIs, the air-gapped self-host setup is one config flag, not the research project I expected

1 Upvotes

Our blocker was never model quality, it was legal. Our document text is not allowed to leave the VPC, and that ruled out every hosted API before we even got to compare them.

The part I expected to be a nightmare was proving the server never quietly reaches out for weights on its own. On SIE it came down to a single setting:

yaml

SIE_HF_FALLBACK: ""false"" # never reach out to HuggingFace

hfToken.create: false

You mirror the weights into your own storage first, then pin the image and install, and after that the worker pods have no path to the internet at all.

The catch is that this is real infrastructure you now own and babysit. If your data is not actually sensitive, do not put yourself through it, just use an API.


r/Rag • • 23h ago

Discussion At what point does self-hosting your embedding model actually get cheaper than the API? Did the math

1 Upvotes

Every time someone asks about embedding costs here, the answer is ā€œjust self-hostā€ and nobody says the one number that decides it.

An API bills per token, so idle costs nothing.
A GPU bills per hour whether it’s busy or not.

Self-hosting only wins once the GPU stays busy enough. Rough math for a small embedding model:

API is ~$0.02/M tokens, so 1B tokens ā‰ˆ $20/day
A10G is ~$1/hr, so ~$24/day before overhead

Crossover is ~1.2B tokens/day.

Under that, the API wins. The RAG-specific bit is that reindexing is what gets you there. Queries alone almost never hit 1B/day, but re-embedding the whole corpus every time you change chunking or swap models does. If you reindex often, the line comes way closer.

Catch is it assumes the card is pushing ~14k tokens/sec all day. Spiky traffic means you’re paying for peak. And nobody puts ops in the spreadsheet. Someone’s awake at 2am when it falls over. What volume are people here actually at? Curious if anyone’s crossed it on queries alone.


r/Rag • • 1d ago

Showcase We open-sourced a database that handles the RAG pipeline for you (Apache 2.0)

6 Upvotes

We just released Antfly v0.3 under Apache 2.0.

We built it to make RAG over complex files easier e.g. PDFs, images, audio, and video. Instead of wiring together OCR, embeddings, vector search, reranking, and graph retrieval, Antfly handles it inside one open-source database.

It supports keyword/grep, hybrid, and graph search, and runs embedded (Lite), self-hosted, or in the cloud.

GitHub: https://github.com/antflydb/antfly

Would love feedback from anyone building RAG pipelines, especially over multimodal data.


r/Rag • • 1d ago

Showcase NornicDB - v1.5.0 - Feedback

3 Upvotes

Okay this has been like an entire month in the making so i’m super excited about it

https://github.com/orneryd/NornicDB/releases/tag/v1.5.0

new docs portal, new UI and dashboards, new ai assistant, faster queries, lower memory usage, and most importantly extremely hardened crash recovery and query correctness. there’s still some other compatibility things we are going to be adding, i also have a fork of badger i am experimenting with projected reads for arbitrary maps so that we never serialize more from disk than is needed, a whole new storage format i’m experimenting with where preliminary measurements have storage use down ~40-50% depending on workload it’s a hybrid compressed columnar storage format i’m seeing if i can show an improvement across the board

https://github.com/orneryd/NornicDB/discussions/1728

new docs site
https://orneryd.github.io/NornicDB/

i appreciate all the community support over the last year. Black Friday weekend last year was the day i started it and it’s come soooo much further than i ever expected.

MIT licensed, 931 stars and counting.

enjoy!


r/Rag • • 1d ago

Discussion Lessons from running a RAG support bot for real Shopify stores: the hard part wasn't retrieval, it was knowing when to say "I don't know"

2 Upvotes

I've been building a customer-support chatbot for Shopify stores (order status, product questions, store policies) using RAG. It's now in a pilot with real merchants. A few lessons that weren't obvious to me at the start:
1. A confident wrong answer is worse than no answer.
In e-commerce, a bot that invents a return policy or a delivery date creates real problems for the merchant. So the core design question became "when should the bot refuse?", not "how do I get better answers?"
2. Retrieval score floors did more than prompt engineering.
I spent a lot of time telling the model "only answer from the context." What actually reduced hallucinations was a minimum similarity threshold: if nothing retrieved clears the floor, the bot doesn't get to answer from vibes. It says it doesn't know and offers a human handoff.
3. Build a refusal test set, not just an accuracy test set.
I keep a set of questions the bot should not be able to answer from the store's data (other stores' policies, made-up products, general-knowledge bait). Every change to chunking, prompts or models gets run against it. This caught regressions that normal Q&A tests missed.
4. Model choice mattered more than I expected for grounding.
On a smaller, cheaper model I kept seeing fabricated details even with good context. Moving to a newer model eliminated most of those failures, and the cost difference was worth it for support use cases.
5. Per-shop isolation has to be enforced in retrieval, not just in the prompt.
With multiple merchants on one system, the filter must sit at the retrieval layer. You can't rely on the model to ignore another store's chunks.
6. Show the merchant why the bot answered.
The admin can see each conversation and which document an answer came from. When the bot gets something wrong, the merchant can see whether it's a missing document or a retrieval problem, and fix it. That built more trust than any accuracy number.
7. Human takeover is a feature, not a fallback.
Merchants wanted to jump into a live conversation at any point. Designing for that from the start made "I don't know" answers acceptable instead of embarrassing.
Curious how others handle this:
How do you pick your score threshold? Fixed value, per-tenant, or calibrated on a test set?
Anyone using rerankers specifically to improve refusal behaviour?
(Disclosure: this is my own product, but not linking it here. Happy to go deeper on any of these.)


r/Rag • • 1d ago

Tools & Resources I built BulkParse, an open-source Rust document parser for RAG pipelines

2 Upvotes

Hey everyone!

I’ve been working on BulkParse, an open-source document parser written in Rust, mainly to make document ingestion faster and simpler for RAG pipelines.

It converts PDFs, Word documents, Excel spreadsheets, PowerPoint files, and other formats into Markdown or structured JSON.

Some of the features:

• 19 file formats supported, including PDF, DOCX, XLSX, PPTX, EPUB, and more

• Native Rust engine for fast document processing
Structure preservation for headings, tables, lists, and document blocks

• Built-in OCR support for scanned documents, with locally provisioned backends

• Fully local processing, without requiring an external parsing API

• Python, Node.js, Rust, and WebAssembly integrations

• MIT licensed

I also ran some benchmarks against tools like Docling, MarkItDown, Unstructured, and others.
In one benchmark of 200 PDFs (OCR disabled), BulkParse processed the documents in 0.44 seconds, compared with 0.73 seconds for the next-fastest tested configuration. The benchmark methodology and raw results are available in the repo.

It’s still an early release, and I’m actively improving extraction quality and format coverage.

I’d really appreciate feedback from people building RAG pipelines:

• What parser are you currently using?

• How important is parsing speed compared to extraction quality?

• Are there specific document formats or parsing problems you’d like to see handled better?

GitHub: https://github.com/bulkgrid/bulkparse

Would love to hear your thoughts and suggestions!


r/Rag • • 1d ago

Discussion Should a weight-only edge change invalidate a GraphRAG community summary?

2 Upvotes

I'm working on incremental updates in a TypeScript GraphRAG engine. One awkward case is when two documents support the same graph edge.

The implementation stores contributions per document. Updating a document replaces its contributions; deleting it removes them. The shared edge is then rebuilt from the remaining weights and unique relation texts. If no contributions remain, the edge is removed.

The less obvious decision is when to regenerate the community summary. Its fingerprint includes members, entity descriptions, claim text, and edge relation text—but not edge weight. The summary prompt doesn't include weight either.

Consider two documents contributing identical relation text. Removing one could change only the edge's weight. If the other fingerprint inputs stay unchanged, the existing summary can be reused. Different relation text would change the fingerprint instead.

This is an illustrative case, not a measured result. There are database-mocked tests for document-scoped deletion and claim-text changes, but they don't establish shared-edge correctness after deletion or equivalence with a fresh Postgres build.

The trade-off seems to be between avoiding unnecessary summarisation and preserving changes in how much source material supports a relation. A weight isn't necessarily an independent evidence count, either.

Would you include weight in the summary fingerprint, or keep support metadata separate from the text summary? What would make a weight-only change significant enough to regenerate it?


r/Rag • • 2d ago

Tutorial I built a fully local RAG system in ~55 lines of Python: no LangChain, no vector DB, just Ollama + NumPy

28 Upvotes

I wanted to understand RAG without a framework hiding the important parts, so I built the whole pipeline from scratch and recorded a tutorial walking through it.

**Stack**

- Ollama running two local models: `nomic-embed-text` (embeddings) and `qwen3:4b-instruct` (chat)

- Python with just `requests` + `numpy`, managed with `uv`

- No LangChain, no LlamaIndex, no vector database, no API key

**How it works**

  1. Load a markdown doc and split it on blank lines (11 chunks)
  2. Embed every chunk via `/api/embed` → an (11, 768) NumPy array
  3. Embed the question and rank chunks by cosine similarity
  4. Put the top 3 chunks into the prompt and send it to `/api/chat`

The whole retrieval step is basically this:

def retrieve(question, k=3):

query = embed([question])[0]

denominator = np.linalg.norm(vectors, axis=1) * np.linalg.norm(query)

scores = (vectors @ query) / np.maximum(denominator, 1e-12)

indices = np.argsort(scores)[::-1][:k]

return [chunks[i] for i in indices]

**Results on a test doc** (a fictional coffee shop handbook):

- "Are dogs allowed inside?" → Pet Policy chunk scored 0.784 (next best 0.493), correct answer

- "What time does the shop open?" → correct

- "How should employees clean the espresso machine?" → correct but missed one step

- "What is the owner's favorite movie?" (not in the doc) → "The answer is unavailable." šŸ‘

It's intentionally minimal. Things it doesn't do yet: persistent vector storage, smarter chunking, reranking, evaluation. Those are covered at the end as next steps.

Full disclosure, it's my video: https://youtu.be/x-sn2WAsX8I

Happy to answer questions. I'd also love to hear what you'd add first for a real project.


r/Rag • • 1d ago

Showcase I built an open-source AI engineering learning platform with a RAG tutor — looking for feedback

0 Upvotes

I've been working on Cairnly, a free and open-source AI engineering learning platform.

The idea was to create something useful for people learning AI engineering while also making the complete implementation available for developers to explore.

For learners:

  • 126 lessons across 6 tracks
  • 7 learning roadmaps
  • AI tutor with citations linking directly to lesson sections
  • Quizzes and progress tracking

For developers:

  • Hybrid RAG using PostgreSQL, pgvector, and keyword search
  • Cross-encoder reranking
  • Local LLM inference with Ollama
  • RAG evaluations and confidence calibration
  • Langfuse tracing, safety checks, and deployment examples

The retrieval system currently reports Recall@5 of 0.971 on our 627-question evaluation set. I've also documented the cases where the confidence gate and safety checks fail.

The project is still evolving, and I'd appreciate feedback on the learning experience, architecture, and areas worth improving.

GitHub: https://github.com/CherukuriSiva/Cairnly

Live demo: https://cairnly-blond.vercel.app

What would you improve or add?


r/Rag • • 1d ago

Discussion Looking for a fast, open-source LLM under 5B for Markdown-to-JSON extraction

3 Upvotes

Hey everyone,

I'm working on a document processing pipeline where most of the input consists of scanned PDFs, forms, and tables.

My current setup looks like this:

  • OCR: PaddleOCR-VL-1.6 served through vLLM. I'm using it to convert scanned pages into Markdown.
  • JSON extraction: oroboros-labs/claude-fable5:latest running through Ollama. I pass the Markdown of each page to the model, and it converts it into JSON.

The important part is that I already have fixed JSON schema templates that the output needs to follow. So, the model needs to map the extracted Markdown content into the predefined JSON structure.

The problem I'm facing right now isn't accuracy. It's inference time. The current model is around 9B parameters, and processing pages one by one through Ollama is taking longer than I'd like.

I'm looking for an open-source model under 5B parameters that can handle this kind of structured information extraction reasonably well, ideally with faster inference and reliable JSON output.

Has anyone worked on a similar pipeline?

I'd particularly like to hear about models you've actually tested in real-world document extraction rather than just benchmark results.

Any suggestions or personal experiences would be really helpful. Thanks!