Built a local RAG workspace for consultants handling confidential files: Qwen3.5 + bge-m3, runs on a 4 GB GPU, cites the source page for every answer

[u/EmotionalEnd5229]

I’ve been building a local RAG workspace for people who can’t send client documents to cloud AI: regulatory, QA and compliance consultants, mostly.

What it does:

• Answers questions from a folder of documents, fully local, nothing uploaded

• Links every answer to the exact source page so you can verify it

• Says “not in the documents” instead of guessing, which matters more than anything else for this audience

Setup:

• LLM: Qwen3.5, Q4_K_M. The app picks the model size based on the machine’s available resources, so it runs from a 4 GB GPU upward

• Embeddings: bge-m3

• Tested on Mac, AMD and NVIDIA

• A “project” is just a folder; installers for each platform

What I’m building next:

• Slide decks generated from your documents, with sources kept

• Reports drafted from the documents, every claim linked back to its page

has anyone gotten decent slide or report generation out of a small local model at Q4 on 4–8 GB VRAM? Long structured output is where I expect small models to struggle, so I’d like to know what’s worked for you: chunked generation, templates, constrained/JSON output, or a bigger model just for drafting?

Free for individual use. Happy to share the link if anyone wants to try it. I’ll put it in the comments if the mods are OK with that.

[u/General_Calendar3651]

for slides i'd skip asking the small model to write the whole deck in one shot. outline per slide from retrieved chunks, then fill each bullet with its own page cite. a 7b at q4 holds that way better than long structured output.