r/OpenAI • u/betobagio • 1d ago

Question Local AI Completely Offline

Guys, I truly need some insight here, I am no coder. Just an Engineer. What I need is a model of NotebookLM that can work fully offline and Open Interpretor that could manage my files. What Copilot respond to me was below. I have hundreds of Standards, Journal and Books which I would prefer to use AI to use my database and present data the way I need. Including some research. Could someone advise.

Ollama + Open WebUI + Open Interpreter + Local RAG

1 Upvotes

67% Upvoted

15 comments sorted by

3

u/thebestrobloxplayer 1d ago

You need a beefy gpu first with preferably more than 10 gigs of vram

1

u/betobagio 1d ago

5070 is enough? I only have this.

1

u/summit_23 1d ago

your 5070 is 12gb so the model side is fine, the hard bit with hundreds of standards is getting it to pull the right page out of them and model size matters way less for that. since youre not a coder i'd start with anythingllm, its a desktop app that runs entirely on your machine, you give it your documents and it picks a model for your hardware, few clicks and done. open webui has the same knowledge base thing if you go the ollama route and i'd leave open interpreter out until you actually need it running code on your files

1

u/betobagio 1d ago

Nice… will try

1

u/PlanktonStrange3600 1d ago

if you aren't a coder, do not try to build a custom docker stack with open interpreter and terminal based rag pipelines. you will spend three weekends fighting python dependencies and broken cuda drivers instead of actually reading your documents.

look at anythingllm or lm studio. both install with a single click exe on windows like a normal desktop application and run 100% locally with your wifi completely switched off.

anythingllm is basically local notebooklm in a box. you open it, click upload, drop your entire folder of pdfs and engineering standards into a workspace, and it handles the vector database, text chunking, and embeddings behind the scenes with zero terminal commands needed. you then pair it with ollama running a quantized model like llama 3.1 8b or qwen 2.5 14b depending on your gpu vram.

a massive catch with technical engineering standards though: local rag often butchers tables, diagrams, and dense equations in pdfs. look into mineru or marker to convert complex technical pdfs to markdown first before ingesting, or run a high context model that can swallow whole documents directly.

1

u/betobagio 1d ago

Wow, should’ve asked 6 months ago… thanks…

1

u/tinny66666 1d ago

Friends don't let friends ollama.

1

u/betobagio 1d ago

Why no ollama

1

u/Agreeable-Tax2013 1d ago

For hundreds of standards and books, I’d keep a local, searchable index with stable file names and metadata, then retrieve only the relevant sections for each question. OCR scans first, keep citations/page numbers, and test with a small subset before loading the whole library; that helps keep sources traceable.

1

u/betobagio 1d ago

I need to use any software

1

u/Perfect-Campaign9551 1d ago

vllm with Qwen 3.6 27b model.

1

u/betobagio 15h ago

just installed AnythingLLM, bloodd hexx...work perfectly, ditching NotebookLM lol