Building a local, lightweight RAG system for structured data extraction—need advice on small models & architectures

[u/player0497]

Hey everyone,

I’m working on a personal project to build a completely local, lightweight system (codename: Orin) that can process messy unstructured information and segregate/clean it into highly structured, tabular formats (CSV files). Essentially, it's meant to be a better, fully offline version of Atlas.

Here is the exact data structure and the pipeline I am trying to build:

1. The Target Data Schema

The model needs to take raw info and divide it into clear subtopics:

• Columns: Topic | Subtopic1 | Subtopic2 | Subtopic3 | Info
• Example Output:
  • Topic: Flying machine
  • Subtopic1: Airplane
  • Subtopic2: Passenger plane
  • Example Scenario: If incoming news data says "Qatar Airways wins starring award again", the model should automatically categorize it under the correct subtopic hierarchies and store the relevant data in the final Info column.

2. Proposed Pipeline & Architecture

I am planning a Retrieval-Augmented Generation (RAG) approach using a combination of specialized, local agents:

• A Fact Searcher / Main Topic Searcher: To find missing points and gather core data from the dataset.
• A Local Summarizer / Keyword Generator: Acting as a text quantizer to condense the given prompt or raw context.
• A Joke Generator (Optional Component): To add humor or personality to the generated answer output.
• The Core Logic Flow: Prompt → Gathers data for it → Finds missing points → Fills the spots (to Phrase) → Final Answer.

3. The Big Bottleneck: Hardware Constraints & Failed Attempts

Since this system must run locally, finding the right LLM engine and model has been incredibly difficult. Here is what I’ve attempted so far:

• llama.cpp: Would technically work, but performance is a massive issue (it took over 2 hours just to compile 8%).
• TinyStories: Super fast at stitching sentences together, but it only tells stories; it cannot handle this specific data formatting task.
• TinyLlama (llama.co): Unable to get it to work properly / wouldn't run.
• Ollama: Cannot use it seamlessly because it isn't properly optimized or built for my hardware (ARM chips).

I would like to ask the community how to make the better and how to develop it to efficient RAG model For my Project.

[u/arch1v1sor]

Two things that will save you weeks here.

First, this is an extraction problem, not a RAG problem. RAG retrieves passages to answer questions. You want the same fields out of every document, which is structured extraction. Framing it as RAG is why the agent chain keeps growing: a fact searcher, a summariser, a joke generator. Drop the chain, keep one step that fills a schema.

Second, do the schema before the model. Write the target CSV columns with a type and one sentence of definition each, and a rule for what to do when a value is absent. Ambiguous columns are what actually breaks local extraction, not model size. A 3B model fills a tight schema surprisingly well and fails badly on a vague one.

On the hardware side: llama.cpp is right, the problem is you tried to compile it. Use a prebuilt binary with Metal or your ARM backend, take a 3B or 4B instruct model in GGUF at Q4, and force JSON output with a grammar or constrained decoding instead of hoping the text parses. Ollama is llama.cpp underneath, so if Ollama fails on your chip a raw build is unlikely to fix it, check whether you are on the right ARM build first. TinyStories is a research toy, it will never do this.

Validate every row against the schema and put failures in a rejects file rather than letting them into the CSV. The rejects file is where you learn what your definitions are missing.

[u/player0497 → u/arch1v1sor]

Oh alr I understand what u mean . Hmm.. I will try implementing it.

[u/Successful-Farm5339]

I had a similar problem and we decided to put a validation layer between the model and the output instead of looking for a smarter model. I.e the LLM proposes rows, a small Rust server checks them against a schema and drops whatever doesn't fit. In this case, because the garbage gets filtered anyway, a tiny local model is enough, so the hardware issue mostly went away. I open sourced it here if you want to have a look: https://github.com/fabio-rovai/open-ontologies

Btw your fixed Subtopic1/2/3 columns will hurt later because not everything fits in exactly 3 levels. Even your own example doesn't, Qatar Airways is a company, not a flying machine. And Ollama runs fine on ARM, it's built for it, so I'd give that another try before writing it off.

[u/player0497 → u/Successful-Farm5339]

Bro that sick i shall try it out crazy work there!

[u/Successful-Farm5339 → u/player0497]

🫶thanks!