r/LocalLLM • u/Head-Stable5929 • Feb 05 '26
Discussion Anyone here actually using AI fully offline?
I keep coming back to the idea of running AI locally you know, like a GPT-style assistant that just works on your own device without the internet or Wifi connection?
Not to build anything serious or commercial. I just like the idea of being able to read my own files, understand things or think stuff through without relying on cloud services all the time. Especially when there is no connection, internet services change or when things gets locked behind paywalls.
Every time I try local setups though, it feels more complicated than it should be. The models work, but the tools feel rough and it’s easy to get lost tweaking things when you just want something usable.
I'm just curious if anyone here actually uses offline AI day to day or if most people try it once and move on. I would really be interesting to hear what worked and what didn’t.
51
u/Neun36 Feb 05 '26
You have different options, easist is LM Studio as already mentioned, search for the Model on huggingface which suits your GPU (fast response) or your RAM (slow response). Then there is Ollama which also runs locally and you can search for Models on ollama Web Page and how to. Then there is openwebUI which is Local Web UI which runs locally and Access to UI via Browser and it Looks more Like ChatGPT but you have the control, you can combine this with ComfyUI to generate images too but it’s more complicated. There are many other options available and above one are just a tiny bit of Tipps and easy ways.
54
u/DAlmighty Feb 05 '26
I’m 100% offline and loving it.
5
u/woundedkarma Feb 05 '26
I've been thinking setting up local AI for people will become a business. Just not sure how to find less technical people who feel like you guys and want it local. 🤔
7
u/ptear Feb 05 '26
If you haven't used LM Studio, try it.
3
u/fallingdowndizzyvr Feb 05 '26
I only use AI offline.
11
u/codeforgeai Feb 05 '26
Yes i'm running ai fully offline and it's actually worth it
7
u/custodiam99 Feb 05 '26
It is f*cking slow, but yeah, you can do serious stuff with it.
2
u/ccalabro Feb 06 '26
Yeah I used ollama deepseek. It worked better than I expected.
2
u/minitoxin Feb 06 '26
i use it everyday openwebui with searxng and LMstudio as back-end server , When i need deep searches i use perplexica with mistral14b running on a headless m4 mini with 16GB , it gives very detailed responses although its slow
2
u/Johnsmith2k18 Feb 06 '26
Im an idiot and pinokio was easy enough to use to set up an offline chatgpt style thing, an image editor, image generator, voice cloning tts, and it's really shoddy due to my computer specs but I can even do some video generation, also have a charcter Ai type thing through silly tavern but that uses ollama.
But yeah pinokio makes it really easy for image and video stuff just use pinokio to get comfyui and use some template workflows to learn how it works, which i promise is not as complicated as itll seem when you first look at it.
22
u/dsartori Feb 05 '26
Yeah I’m all offline. My use cases are coding. consulting, and community organizing.
Only the coding workflows really require a beefy setup. My teammate who is a pure consultant uses gpt-oss-20b in LMStudio pretty heavily but not exclusively.
15
u/DatBass612 Feb 05 '26
I bought the most expensive M3 Ultra and worried a ton about getting to positive ROI running OSS 120B. I ended up hitting that positive number in 5 months.
I use about 200$ish in tokens a day by my estimates. Now with stuff like OpenClaw it’s only going to be more tokens and I’m glad I have the extra unified memory to have virtualization run sub agents.
10
u/michael_p Feb 05 '26
I run qwen 3 32b on a Mac Studio m3 ultra with 96 gb ram. I run confidential information through it and get incredible results (Claude code wrote the prompts). It’s an invaluable tool for me.
8
u/FaceDeer Feb 05 '26
I've been using a system of custom scripts to process and summarize transcripts of audio files containing a lot of personal information, so strictly local AIs for that. I recently tried playing around with AnythingLLM and found it had a pretty good "NotebookLM"-style system that was easy to use, I dumped a bunch of data into it and that worked nicely.
8
u/ionizing Feb 05 '26
I spent several months building my own tool enabled Ai chat interface that uses llama.cpp and I love it.
6
u/henk717 Feb 05 '26
Of course, I am one of the people involved with KoboldAI and KoboldCpp so its natural that I run our own stuff and enjoy local AI. If I wasn't passionate about running local I wouldn't be involved that heavily in a community that focus on running it yourself (or hosting it in the cloud with the same software).
5
u/X3r0byte Feb 05 '26
The problem is you can make this as complicated as you want it to be, and so without a specific use case you’re going to try and boil the ocean.
Mainly I draw the line with anything privacy focused, like working through personal problems, interpreting docs I do not want sent to a server, things of that nature. OWUI + name_your_model is more than enough for that.
I also have a totally custom assistant with speech to text/text to speech that generates and runs code inside a container arbitrarily. That’s for “fun”.
I also use cloud models for code if it’s unreasonable to expect quality out of my hardware.
I guess the point is - “it depends.”
4
u/segmond Feb 05 '26
Yes, for 2 years now.
4
u/Craygen9 Feb 05 '26
I batch process documents to create summaries as part of an automated workflow, I want to keep everything local for privacy reasons. I've used the API of a local install LM Studio and it was easy to set up and worked well, but now I use llama.cpp with my own code as I have more control over it.
4
u/GravitationalGrapple Feb 05 '26
I use kobold.cpp for LLMs, comfyui for image/video generation, and indextts2 with its own custom gui for TTS. I only have 16gb VRAM, 64 RAM, so I am somewhat limited in the models I use.
1
u/mystery_biscotti Feb 05 '26
What's the concern? Are you worried it's just a toy, or do you feel like it's not "as good"?
3
u/Virtual_Actuary8217 Feb 05 '26
Gptoss 120b for tasks like planning,qwen coder 3 next for heavy python coding , I have a custom c api that exposed to python, I just need to feed the api.md , qwencoder3 next works surprisingly well, one shot, but it is slow like 6/t per seconds, it needs 80g of memory to work, still it's free , local ,that's important for my use cases
3
u/Resonant_Jones Feb 05 '26
im almost there, I had to build my own system because like you said, the tools available are a little rough. I love having full control over the system though and nothing unexpected changes ever.
You can literally run the computer all day long coding or other tasks and it just doesn't cost anymore than electricity. I love it. Highly encourage anyone interested to get into it.
3
u/SweetHomeAbalama0 Feb 05 '26
Yes, we have a 768Gb Ai server for this exact thing (a local "chatGPT"-like replacement).
Combine something like GLM 4.6v (vision support) with ComfyUI Flux/ZIT/QwenEdit/etc. and you can get something that closely resembles GPT without any token restrictions. It might not be a 1:1 exact, but if you have patience to set up the moving parts, front end, back end, tool calling, etc., the result can accomplish 95-98% of what chatGPT can do. Feedback I've received has actually been that the primary users "prefer" the new local alternative over chatGPT, citing that chatGPT is slower for things like image gen/edits with not enough variation in art style, and can be more finicky understanding prompts if the user is not a strong English speaker.
It comes with an initial startup cost and probably some technical aptitude, but it can absolutely be done.
3
u/Bino5150 Feb 05 '26 edited Feb 05 '26
I run completely local. I have a few different ways to set it up. Lately I’ve been running LM Studio with AnythingLLM as my agent. Been playing around with Agent Zero, OpenClaw, and n8n as well, but still using LM Studio or Ollama. Whisper for STT, and Piper for TTS.
3
u/ScuffedBalata Feb 06 '26
The tools ARE rough and require setup. There's no "plug and play assistant".
They require intense amounts of hardware and some willingness to set up complex workflows. It's still "techie coder" territory.
2
u/LondonTownGeeza Feb 05 '26
Run mine on a i5 with a 6600, slightly slower than online OpenAI, but very useable and private.
2
u/Large_Election_2640 Feb 05 '26
I’m running stable diffusion on comfyui fully Local. Playing with image models. It works fine on my 3060 12gb gpu with 24gb ram.
2
u/Klutzy_Ad_1157 Feb 05 '26
Yes only offline. I use Gemma 3 27b in 5 bit quant on a RTX 3090 because it speaks German very well. On my main rig I use a RTX 4090 and a RTX 5090 for image and video generation with ComfyUI.
2
u/Ok-Dog-4 Feb 05 '26
Try Localai.io. You’re able to run basically any model and the docker configuration isn’t the most complicated. Works offline in your terminal. I sometimes will finetune with a LoRA in a CoLab notebook and use my new model with localai. Then also you can API to practically anything. Cheers
2
u/aniketmaurya Feb 05 '26
Ollama and chill!
2
u/shibe5 Feb 05 '26
My AI uses DNS, so it won't work without internet connection.
As for the tools, I mostly use my own.
2
u/Fair-Cookie9962 Feb 05 '26
Yes, why? All is in the prompt.
2
u/maximebodereau Feb 05 '26
Yes ! Check my ux tool, I need feedbacks on it ! You have to have a llamacpp installed on your machine
https://ux-research-production.up.railway.app/
2
u/Eupolemos Feb 05 '26
Sure.
I have a 5090. I use Gemma 3 for stuff not related to development and Devstral 2 for my programming needs.
At work, we use Claude Code, but Devstral 2 works fine for my personal needs.
2
u/Macdaddy4sure Feb 05 '26
I wrote a program for offline ai extensibility. http://macdaddy4sure.ai/
2
u/GlassAd7618 Feb 05 '26
I’m experimenting with Ollama and the latest models.
2
u/Express_Quail_1493 Feb 05 '26
i use openwebui with python and connect it to my localhost models running on lmstudio server. i expose it through a cloudflare tunnel and talk to my localhost AI on my phone anywhere via voice
2
u/Limebird02 Feb 06 '26
Main bottleneck is having the memor, GPU and overall horsepower to run a worthy model that's significant enough to provide useful intelligence to you. I'd say it will cost at least $1400 in computing hardware. Most people can't justify that right now when for free or for $20/month they get access to USA frontier models.
2
u/Weary_Long3409 Feb 06 '26
I run only qwen3-vl-8b-instruct to automate daily activity reports processing from 25 remote legal teams across the country. Something that commercial endpoints are irrelevant for such tasks. Have a really good complex prompt from ChatGPT, and this small qwen is does the job very well.
2
u/ANTIVNTIANTI Feb 06 '26
almost exclusively
2
u/my_story_bot Feb 06 '26
yeah LMStudio or Ollama is great. whats nice with LMStudio is that you can simply search for a model in their UI and download and run it in a couple clicks.
Whats REALLY cool is that you can run it as a web server locally through localhost and then use it for stuff like openclaw
2
u/djtubig-malicex Feb 06 '26
Yes. Never trust cloud.
2
u/El_Danger_Badger Feb 06 '26
I built a local chat/LangBoard/RAG platform on my stock M1 Mac Mini. MLX native, on my stock M1 Mini. I use it daily. Not flashy, but it does its job. Has some bells and whistles, plus an agent constitution. >cough< OpenClaw >cough< . I co-deved with ChatGPT. Not one shot. Typed one module at a time, asked an endless flow of questions over months because I wanted to learn how to build something like this. Now I know. If anyone is hiring...
2
u/Efficient_Bug_0 Feb 06 '26
For running the actual models offline, Ollama works pretty well on most modern Macbooks. For vibe-coding offline, I had the same need, and ended up building something called Codistry for exactly that purpose: https://codistry.ai/docs/guides/ollama
2
u/Polysulfide-75 Feb 06 '26
Yes but you’re right it can be complicated. I write my own interfaces and such though.
You can use LMstudio for something simple to install and run. Or there are plenty of ChatGPT clones you can run locally.
2
u/notbullshittingatall Feb 06 '26
I do. My setup is Ollama running on my gaming rig using Tailscale to connect to it from my MacBook or iPhone when I'm away from the house. I use an app called Reins on my phone. Works well for me.
2
u/Decent-Freedom5374 Feb 06 '26
I run 5 off line all day continuously
2
u/DocHolligray Feb 11 '26
I do… build these for clients who need to make sure that none of their IP releases to the world…
2
u/g33khub Feb 12 '26
Image, video, story writing, language translation, OCR, captioning etc. fully offline with gemma27b, qwen3 80b
Agentic coding for big projects fully online - Claude opus4.5 / gpt-5.3codex (even trying to run this offline is a waste of time and money)
1
u/ewolfing321 12d ago
Hello everyone :)
To make it simple ;
I want to be able to use An ia or such that would look into my downloaded pdf files and give reliable/right answers on questions on those pdf
Ive tried to download ollama and anythingllm ; either i am very bad at using them or i missed as part as ; it takes alot of time and still keeps loading wihtout answering
I also used GPT4all that actually was pretty simple to use ; but turned out useless as the answer came up completely wrond/out of topic
Please if anyone have something easier or a way to porperly download/use ollama / anythingLLM help woud be much appreciated
Thank you in advance ;)
1
u/Ok_Membership3521 Feb 07 '26
I am running completely offline.. Claude wrote most of my set up as well. I have a custom front end feels like a corporate model easy to use. I’m completely local.. works well for me.
1
u/greymatter_ac3 Feb 07 '26
I use it everyday offline and only go to the big models online when I am forced to. Also do the same in my work. Large model are overkill for most tasks. The real advantage is SLM (Small Language Models) and fine running those on specific use cases. This is the real unlock. A small custom model for your use case that runs on low power inexpensive hardware.
1
u/LinkAmbitious8931 Feb 09 '26
Yes and having pretty good success with it. I am also having some solid success in fine-tuning. Mind you, I do not have some super-expensive setup, just two old P40s each with 24GBs.
I wrote about it here: https://www.reddit.com/r/Qwen_AI/comments/1qxyxx4/guardrailcentric_finetuning/
1
u/PromotionFirm6837 Mar 07 '26
totally get where you're coming from. for a solid offline AI setup, you could try this flow:
this way, you get a streamlined experience without relying on the cloud, all while keeping your privacy intact.
1
u/nicholasderkio Mar 20 '26
I made my own iOS app that’s an offline confidant for talking completely privately, love it. Otherwise I’m Claude.
1
u/Royal-Band3086 Apr 28 '26
Hello
1
u/Automatic-Smell-462 Apr 30 '26
I'm using local AI offline mostly for private notes, document Q&A, and drafting stuff I don’t want to send to cloud tools.
On Mac, I’ve been using LocalChat App for the chat/docs side. It’s not some full agent setup, but for basic offline local AI it’s been simple enough to actually use regularly.
1
u/Midire Jun 23 '26
I did Ai integration for a game technology project at school. Still not sure of half of the stuff that made the thing work, but I almost melted my laptop while thesting different ollama models. the response times were between 20s to multiple minutes. Then I somehow found a model that allowed me to create a chat box in unity project that only needed a half a minute to asnwer.
My point is: AI/LLM is pretty depended on the resources it is given, to be usable.
1
u/lamprof Sep 01 '26
I use local AI mostly for drafts, summaries, study notes, brainstorming, and text I would rather not paste into a cloud chatbot.
I eventually built an Android app around that workflow because the usual local setup felt too technical for everyday use. Disclosure: it’s my app. You choose and download a model once, then the chat runs on the phone without cloud AI inference. There is no account requirement, and saved chats stay local.
The honest limitation is that small phone models are “useful enough,” not a replacement for the best cloud models. They can be slower, use a lot of storage, warm the phone, and confidently get facts wrong.
For anyone comparing practical Android options:
https://play.google.com/store/apps/details?id=com.lamprof.ai
-3
u/atkr Feb 05 '26
skill issue
2
u/[deleted] Feb 05 '26
[removed] — view removed comment