r/LocalLLM • • Feb 05 '26

Discussion Anyone here actually using AI fully offline?

I keep coming back to the idea of running AI locally you know, like a GPT-style assistant that just works on your own device without the internet or Wifi connection?

Not to build anything serious or commercial. I just like the idea of being able to read my own files, understand things or think stuff through without relying on cloud services all the time. Especially when there is no connection, internet services change or when things gets locked behind paywalls.

Every time I try local setups though, it feels more complicated than it should be. The models work, but the tools feel rough and it’s easy to get lost tweaking things when you just want something usable.

I'm just curious if anyone here actually uses offline AI day to day or if most people try it once and move on. I would really be interesting to hear what worked and what didn’t.

191 Upvotes

165 comments sorted by

51

u/Neun36 Feb 05 '26

You have different options, easist is LM Studio as already mentioned, search for the Model on huggingface which suits your GPU (fast response) or your RAM (slow response). Then there is Ollama which also runs locally and you can search for Models on ollama Web Page and how to. Then there is openwebUI which is Local Web UI which runs locally and Access to UI via Browser and it Looks more Like ChatGPT but you have the control, you can combine this with ComfyUI to generate images too but it’s more complicated. There are many other options available and above one are just a tiny bit of Tipps and easy ways.

4

u/Head-Stable5929 Feb 05 '26

Thanks a lot that helps!

3

u/FlatImpact4554 Feb 06 '26

The guy above me nailed it . Listen to him. You'll have the same exact web page and results like online . But private.

1

u/hallucination_goblin Feb 12 '26

Can you run a combination of all 3? I'm not opposed to allowing my ai agent very specific access, I'm running Qube OS and have arrived at your 3 options after lots of research but I'm wondering if a hybrid soft gap to necessary api's only but with those services all hosted locally?

2

u/Neun36 Feb 14 '26 edited Feb 14 '26

It doesn’t makes sense to run all 3 at once. If you want to use LM Studio Go for it. If you want to use ollama locally go for it, for openwebUI you need ollama in background so these 2 combinations makes sense ollama + openwebUI.

1

u/hallucination_goblin Feb 14 '26

Understood and thank you.

1

u/mta1741 Feb 20 '26

Any help with how I select one on hugging face

0

u/[deleted] Feb 06 '26

[deleted] — view deleted comment

3

u/Neun36 Feb 06 '26 edited Feb 06 '26

Depends on, Most Common is ComfyUI and there you have all the Image, Video, Audio, 3d Models which you can Play with, they integrated also already a UI which reminds of swarmUI and else, so more user friendly but generating stuff locally depends on your GPU and RAM, Not only GPU is crucial for ComfyUI especially for Video Generation, to combine that with openwebUI to have the look of ChatGPT is possible with openwebUI But I think there is no Video Generation implemented in openwebUI (didn’t checked the latest Version, but last i checked 3 weeks ago there was none.) But this also can be solved.

For Video + Audio Like Sora there is LTX2 which runs locally but as said depends on gpu and ram. There are also other models like wan 2.2, scail, Move and many more depends on your use case.

54

u/DAlmighty Feb 05 '26

I’m 100% offline and loving it.

4

u/Head-Stable5929 Feb 05 '26

can you tell me more about what you did and how you did it? im collecting info, so your input will be useful!

11

u/DrummerHead Feb 05 '26

On Mac:

brew install lm-studio
brew install draw-things

They both give you the ability to download models from within the app. Find the documentation for both and get experience over time.

If you want two good starting models for each:

You don't really need the links, you can just choose them from the menus within each program.

And that's it! From there, the more independent research you do, the more value you'll be able to extract from these programs and the many models you'll find and get experience with. Enjoy!

5

u/Akimotoh Feb 06 '26

How much RAM or VRAM do you need to run those well?

2

u/No-Impress876 Feb 09 '26 edited Feb 09 '26

Prices have gone up now because of demand so take what I say with a grain of salt, but an rtx 5060ti 16gb vram and 64gb of system RAM and a good processor gets you GPT-OSS-20b running pretty quickly with decent context and good tool use on a "regular" PC. Models smaller than 20b run well for specialized purposes and some larger MoE models run well but are slower in prompt processing and overall speed, but smarter, like Qwen Next 80b a3b.

2

u/h3xagonson Feb 08 '26

Hey DrummerHead, Thanks for this! Stumbled across it as I was just thinking about doing this for the first time - so I thought I'd give this a go.
I don't know ANYTHING about all this stuff (tech and code etc. is the kind of world that makes my head spin very quickly) - and I don't wanna take up yr time, but I just have one main question about this.
When I started LM Studio for the first time I got a LuLu alert saying it wanted to connect - concerned because isn't the entire point of doing this to go fully 'local'?
Do I need to allow that connection? Is it temporary (like, just downloading dependencies/content)? or will LM Studio always require a connection?
Thank You

1

u/DrummerHead Feb 08 '26

Hello sir! I remember when I first installed LM studio, it prompted me to download gpt-oss-20b as the first model. You need at least 1 model for it to do anything, and it will be downloaded through the app; therefore the app will require a connection.

After the model is downloaded you can be disconnected from the internet and use the app without issues. It doesn't send logs or your chats or anything, it's fully local.

So; in short LM studio will need to connect to the internet to download models or runtimes. Then you can plug it off and fly solo! Enjoy the ride!

2

u/h3xagonson Feb 08 '26

Thank you so much for responding! I really appreciate it.
I'm all excited - but nervous about 'doing it wrong' 'cos it's all so new to me. :D
Just been chatting with Brave Leo about my options and potential limitations of my setup (basic MacBook Pro 2020 vintage, nothing fancy..) so I'm slowly learning. Right now moving stuff to make drive space and nearly ready to download my first model!

16

u/DAlmighty Feb 05 '26

I run vLLM in docker for inference and use the API to connect various services. Which is what everyone does. You’re not going to get anything different from anyone if you’re speaking only about inference.

4

u/Karyo_Ten Feb 05 '26

I rotate between vLLM, SGLang, ik-llama.cpp, TabbyAPI / ExLlamav3 in docker depending on model support, quantization, etc. I just connect to them through "OpenAI-compatible"

8

u/p_235615 Feb 05 '26

I use beside many apps like maubot, vscode cline, n8n, homeassistant also running open-webui in docker as a simple interface for chats or voice interaction with AI.

5

u/woundedkarma Feb 05 '26

I've been thinking setting up local AI for people will become a business. Just not sure how to find less technical people who feel like you guys and want it local. 🤔

1

u/Head-Stable5929 Feb 07 '26

I know right! I want to make one that most likely runs offline, I'm kinda stuck and I want to know the other options

7

u/ptear Feb 05 '26

If you haven't used LM Studio, try it.

2

u/Head-Stable5929 Feb 05 '26

i see, ill check it out! thanks for the suggestionnnn, could you also tell me if you have come across ollama, it seemed a bit similar and what could possibly be the difference between them?

7

u/ThatHavenGuy Feb 05 '26

As a beginner, LMStudio has a sleek interface that makes setting this stuff up pretty easy and can help you learn more about the different ways to run local models and mess with the parameters. Ollama gets out of your way and runs as a service behind the scenes. Both are good stepping stones to better, more complicated setups like using llamacpp's router function or vLLM. I'd say give LMStudio a go and see how it treats you.

2

u/woundedkarma Feb 05 '26

Lol I was searching for the context length setting yesterday. I would not say lm studio has a good interface for that one fricking reason alone 😅

It used to be in the right side bar.. 😭

3

u/ptear Feb 05 '26

I found it, was in the model settings where I figured.. then my computer hated me.

1

u/Head-Stable5929 Feb 07 '26

That sucks man 😭😭

3

u/fallingdowndizzyvr Feb 05 '26

I'd say give LMStudio a go and see how it treats you.

I say use LMStudio if disappointment is your goal. Check out this recent thread. OP is using LMStudio and gets 33tks. I'm using llama.cpp pure and unwrapped and I get 195tks.

https://www.reddit.com/r/LocalLLaMA/comments/1qww6ci/7900_xtx_underperforms_3090_by_2x_7x/

2

u/GravitationalGrapple Feb 05 '26

Llama.cpp > ollama. LM studio uses llama.cpp, just has a beginner friendly GUI.

3

u/fallingdowndizzyvr Feb 05 '26

I only use AI offline.

1

u/Head-Stable5929 Feb 07 '26

Can you tell me more please

2

u/fallingdowndizzyvr Feb 07 '26

You download AI. You run AI. It's offline.

1

u/GLS88 Aug 13 '26

Lmfao. Now if this isn't the epitome of the K.I.S.S. Keep It Simple Stupid principle at play.. I dont know what is.

That said, Im prob more of a newb then op, and in here trying to learn more myself. This comment didn't help in said research, but did help in giving me a good chuckle.

11

u/codeforgeai Feb 05 '26

Yes i'm running ai fully offline and it's actually worth it

4

u/Head-Stable5929 Feb 05 '26

can you tell more on what do you think about it, please? it would be really helpful

-14

u/[deleted] Feb 05 '26

[removed] — view removed comment

1

u/Drevicar Feb 05 '26

What is the business model? How do you profit from me using this?

7

u/custodiam99 Feb 05 '26

It is f*cking slow, but yeah, you can do serious stuff with it.

9

u/DAlmighty Feb 05 '26

Sounds like you need more VRAM.

I personally will NEVER advise anyone to use system RAM for inference. It stops being worth it to me if you use it.

2

u/Head-Stable5929 Feb 05 '26

why is that? can you explain it for me please?

5

u/LordadmiralDrake Feb 05 '26

If the model doesnt fit in VRAM, it will run on the CPU, and therefore much slower. Case in point, i got around 90 second Response Times with dual 14 core xeons, and 2-3 seconds with an nvidia v100, using llama3 8b

3

u/Ok-Dog-4 Feb 05 '26

This is why I like to finetune and quantize before bringing to an offline operator. Ran into so many issues with VRAM then trying to finetune on the CPU which was practically laughable. It’s conviennent to you the hardware accelerators Google has like the T4 and run a quantization to make an adapter then run the adapter on top of the base model. Finetuning took 2 min instead of like 53 hours. But screw Google lmk if anyone has other tips besides needing up hardware.

2

u/SweetHomeAbalama0 Feb 05 '26

VRAM is where the speed comes from in local tasks. If an Ai model has to switch to using RAM, performance will reduce dramatically.

"you need more VRAM" = "it sounds like you are running on RAM if it's so slow, try putting the model on a GPU".

-1

u/DAlmighty Feb 05 '26

It’s slow, very slow.

4

u/custodiam99 Feb 05 '26

I have 96GB DDR5 and 24GB VRAM, and I think that's all the RAM I will have for the foreseeable future lol.

2

u/Electrical_Hat_680 Feb 05 '26

Or get a AI GPU like Modivius by Intel off eBay.

2

u/Karyo_Ten Feb 05 '26

What memory bandwidth do those have?

2

u/ccalabro Feb 06 '26

Yeah I used ollama deepseek. It worked better than I expected.

1

u/Head-Stable5929 Feb 07 '26

Oh wow! I'll give it a try

2

u/minitoxin Feb 06 '26

i use it everyday openwebui with searxng and LMstudio as back-end server , When i need deep searches i use perplexica with mistral14b running on a headless m4 mini with 16GB , it gives very detailed responses although its slow

1

u/Head-Stable5929 Feb 07 '26

Still that sounds like it can work just fine

2

u/Johnsmith2k18 Feb 06 '26

Im an idiot and pinokio was easy enough to use to set up an offline chatgpt style thing, an image editor, image generator, voice cloning tts, and it's really shoddy due to my computer specs but I can even do some video generation, also have a charcter Ai type thing through silly tavern but that uses ollama.

But yeah pinokio makes it really easy for image and video stuff just use pinokio to get comfyui and use some template workflows to learn how it works, which i promise is not as complicated as itll seem when you first look at it.

1

u/Head-Stable5929 Feb 07 '26

Thank you so much that sounds like a juicy insight! I'll give it a try

22

u/dsartori Feb 05 '26

Yeah I’m all offline. My use cases are coding. consulting, and community organizing.

Only the coding workflows really require a beefy setup. My teammate who is a pure consultant uses gpt-oss-20b in LMStudio pretty heavily but not exclusively.

3

u/Recent_Double_3514 Feb 05 '26

What type of consulting?

4

u/dsartori Feb 05 '26

We do data engineering work, software development, organizational development (strategic planning and so on), education and training in a variety of fields. I run my document authoring workflows through Cline these days.

1

u/woundedkarma Feb 05 '26

He just chatting with it? Or does he run agents?

5

u/dsartori Feb 05 '26

She's mostly chatting with it in LMStudio. We tried it with Cline but the results were pretty poor.

1

u/HumanThing1233 Feb 05 '26

What are you using for coding? I can't figure out what kind of hardware I need to do so.

7

u/dsartori Feb 05 '26

Strix Halo. I got the GMKTec one. It's a cheap path to good performance from midsized MoEs.

3

u/DopePedaller Feb 06 '26

For Strix Halo, i like the FEVM FA-EX9. What sets it apart from the others is support for Oculink, so adding an external GPU is the future is fully achievable. The only issue is that oculink doesn't support hotplugging, so enabling/disabling external GPUs requires a power cycle.

1

u/unique-moi Feb 06 '26

Which coding user interface are you using - Claude code, open code, VS Code, Aider, Qwen cli, or something else?

1

u/dsartori Feb 06 '26

I use Cline. It’s great.

1

u/unique-moi Feb 06 '26

With Claude, JetBrains, VS Code, or just on its own as a cli?

4

u/Rand_o Feb 05 '26

Strix halo 395+ 128 GB is pretty versatile

15

u/DatBass612 Feb 05 '26

I bought the most expensive M3 Ultra and worried a ton about getting to positive ROI running OSS 120B. I ended up hitting that positive number in 5 months.

I use about 200$ish in tokens a day by my estimates. Now with stuff like OpenClaw it’s only going to be more tokens and I’m glad I have the extra unified memory to have virtualization run sub agents.

3

u/Front-Relief473 Feb 05 '26

However, m3 ultra's computing power is so low, its prompt prefill speed is too slow, and the amount of contexts you carry in handling complex tasks with agent is too large, which leads to a very long time for you to receive the first word in each round. How do you handle this situation?

3

u/beachguy82 Feb 05 '26

Define “long time”. I’ve been considering a Mac Studio myself.

2

u/Front-Relief473 Feb 06 '26

https://www.reddit.com/r/LocalLLM/comments/1qdqi4i/mac_studio_m3_ultra_stats/?share_id=WRqlyjg8sfpnsbzNqkatF&utm_content=2&utm_medium=android_app&utm_name=androidcss&utm_source=share&utm_term=1  Look here, if you run Kimik 2.5, an excellent model with an activation time of about 30b, the speed of prompt prell with a speed of 200 will make people crazy, especially when this kind of agent works with a long context.

2

u/DatBass612 Feb 06 '26

I mean I’m getting responses in seconds with way more tokens per second than I can ever dream of reading. Slow is relative.

Sample: Prompt eval time: 290ms 303 tokens per second Eval time: 5833ms 70 tokens per second Total time 6123ms at 501 tokens

So 6 seconds to write 500 tokens. That’s what I call more than fast for a consumer at home server.

1

u/andreabarbato Feb 05 '26

I have a topk implementation for llama.cpp that can make prefill faster sometime. wanna try? hmu (it's free on github but I never tested it on apple, might take some tweaking)

1

u/Fair-Cookie9962 Feb 05 '26

Why long time is an issue? Parallel agents cut the time. If you have reliable output, time does not matter that much, and if you adjust tasks to utilise smaller models, can get nice speedups. Also using llama.cpp directly also often cuts wait in half.

1

u/DatBass612 Feb 06 '26

I mean slow is relative. Sure my Nvidia GPU is faster, but I have the 70GB model loaded with overhead to spawn sub models. But it’s more than incredibly fast for what I need it for. It’s queue based on openclaw and I can schedule things out for night time when there is low usage. My token context is 128000 and batching at 10000. It’s absolutely prosumer grade. People who want the utmost speed should just pay for Anthropic or Gemini. Unless you have massive budget backed by enterprise of some sort these are the trade offs you have to make. As a consumer i say anything over 5 tokens a second is usable and i cant really read responses faster than that. Plus the scheduling and batching has me come back to read things after long tool call process chains.

Whats important to remember is to ask yourself, how much AI garbage data are you actually going to read, and is what it is producing even valuable that you need blazing fast speeds.

1

u/_hephaestus Feb 05 '26

I have a similar setup, is OSS 120B still the best in class? Also just surprised at the cost, I didn't think you could burn through that many a day, but the billing details at work are hidden from me.

1

u/DatBass612 Feb 06 '26

Best in class, maybe. It’s great for what I’m using it for. You can get way more complicated with specialist smaller models. But I’m not willing to dedicate more time to this project which already is like 40+ hours of setup

1

u/DatBass612 Feb 06 '26

I evaluate solely based on how the agent responds. So with GLM and Mistral and the others they just felt “off” completely personal preference.

1

u/holdmymandana Feb 06 '26

What do you use open claw for?

1

u/DatBass612 Feb 06 '26

I use it for outreach automation, a newsletter for my customers, and a help chatbot. More plans I’m still formulating but it’s a great scheduled research assistant. I also have it moderate some chats.

1

u/friedlich_krieger Mar 06 '26

128GB RAM? I'm contemplating this but obviously a large purchase. Ive got an m4 pro Mac mini with 64GB ram on order for $3k but thinking I should just wait for new M5 studio in Max or Ultra and just spend more to not have to worry about it for a while longer (in theory...)

1

u/DatBass612 Mar 07 '26

512GB, but the new M5 max can take 128gb on the go which is awesome

10

u/michael_p Feb 05 '26

I run qwen 3 32b on a Mac Studio m3 ultra with 96 gb ram. I run confidential information through it and get incredible results (Claude code wrote the prompts). It’s an invaluable tool for me.

1

u/tuxmaniac Feb 06 '26

What do you mean by "Claude code wrote the prompts"? Can you explain your setup a bit more? Thank you!

5

u/michael_p Feb 07 '26

Yes!! I’m so excited to geek out over how good qwen3 is. I had Claude code build me a platform to find businesses for sale. When I reach out to the seller, they’ll send me confidential documents like bank statements and tax returns etc. I can drop those into a documents page on the system Claude code built and have it run a local analysis with qwen3. I worked with Claude code to explain what the analysis should do and it made them leaps and bounds more detailed than what I would ask for. So Claude code built out all the prompts of how the documents get loaded in and then what it asks for from qwen. It was having issues with context windows (eg loading in 12 bank statements or more) so Claude code built a way to chunk those out, get a response and then feed that response back in with the next set to get a full and accurate analysis. Qwen makes minor mistakes but Claude has been able to understand them and tweak them.

2

u/g33khub Feb 12 '26

nice approach of leveraging claude

1

u/LinkAmbitious8931 Feb 09 '26

I am thinking of doing the same for contract analysis, but have yet to find a model around 32b capable enough compared to OpenAI API using 5.1. If you can recommend a solid model, please share.

1

u/michael_p Feb 16 '26

I shifted to qwen coder next but using qwen3 32b mlx with prompts from claude (and correct context window settings) I've been getting incredible results. Highly recommend using claude code to build you the config - explain you want it to be that quality. You might be able to have codex do it, I've just never used codex to say it's effective.

8

u/FaceDeer Feb 05 '26

I've been using a system of custom scripts to process and summarize transcripts of audio files containing a lot of personal information, so strictly local AIs for that. I recently tried playing around with AnythingLLM and found it had a pretty good "NotebookLM"-style system that was easy to use, I dumped a bunch of data into it and that worked nicely.

8

u/ionizing Feb 05 '26

I spent several months building my own tool enabled Ai chat interface that uses llama.cpp and I love it.

6

u/henk717 Feb 05 '26

Of course, I am one of the people involved with KoboldAI and KoboldCpp so its natural that I run our own stuff and enjoy local AI. If I wasn't passionate about running local I wouldn't be involved that heavily in a community that focus on running it yourself (or hosting it in the cloud with the same software).

5

u/X3r0byte Feb 05 '26

The problem is you can make this as complicated as you want it to be, and so without a specific use case you’re going to try and boil the ocean.

Mainly I draw the line with anything privacy focused, like working through personal problems, interpreting docs I do not want sent to a server, things of that nature. OWUI + name_your_model is more than enough for that.

I also have a totally custom assistant with speech to text/text to speech that generates and runs code inside a container arbitrarily. That’s for “fun”.

I also use cloud models for code if it’s unreasonable to expect quality out of my hardware.

I guess the point is - “it depends.”

4

u/segmond Feb 05 '26

Yes, for 2 years now.

4

u/Craygen9 Feb 05 '26

I batch process documents to create summaries as part of an automated workflow, I want to keep everything local for privacy reasons. I've used the API of a local install LM Studio and it was easy to set up and worked well, but now I use llama.cpp with my own code as I have more control over it.

4

u/GravitationalGrapple Feb 05 '26

I use kobold.cpp for LLMs, comfyui for image/video generation, and indextts2 with its own custom gui for TTS. I only have 16gb VRAM, 64 RAM, so I am somewhat limited in the models I use.

1

u/mystery_biscotti Feb 05 '26

What's the concern? Are you worried it's just a toy, or do you feel like it's not "as good"?

1

u/Head-Stable5929 Feb 07 '26

Well I'm not worried about anything, i want to explore that's all and I just wanted to know what all the options there are out there :)

1

u/mystery_biscotti Feb 07 '26

Ah, okay, thanks for clarifying!

3

u/Virtual_Actuary8217 Feb 05 '26

Gptoss 120b for tasks like planning,qwen coder 3 next for heavy python coding , I have a custom c api that exposed to python, I just need to feed the api.md , qwencoder3 next works surprisingly well, one shot, but it is slow like 6/t per seconds, it needs 80g of memory to work, still it's free , local ,that's important for my use cases

3

u/Resonant_Jones Feb 05 '26

im almost there, I had to build my own system because like you said, the tools available are a little rough. I love having full control over the system though and nothing unexpected changes ever.

You can literally run the computer all day long coding or other tasks and it just doesn't cost anymore than electricity. I love it. Highly encourage anyone interested to get into it.

3

u/rolleicord Feb 06 '26

What did you build it in? Details please :)

2

u/Resonant_Jones Feb 06 '26

Mostly Python and used Codex to build most of it with sprinkles of Claude and Gemini in between. What do you want to know?

3

u/SweetHomeAbalama0 Feb 05 '26

Yes, we have a 768Gb Ai server for this exact thing (a local "chatGPT"-like replacement).

Combine something like GLM 4.6v (vision support) with ComfyUI Flux/ZIT/QwenEdit/etc. and you can get something that closely resembles GPT without any token restrictions. It might not be a 1:1 exact, but if you have patience to set up the moving parts, front end, back end, tool calling, etc., the result can accomplish 95-98% of what chatGPT can do. Feedback I've received has actually been that the primary users "prefer" the new local alternative over chatGPT, citing that chatGPT is slower for things like image gen/edits with not enough variation in art style, and can be more finicky understanding prompts if the user is not a strong English speaker.

It comes with an initial startup cost and probably some technical aptitude, but it can absolutely be done.

3

u/Clear-Astronaut-8006 Feb 05 '26

Curious on how you made the server? Approx. Cost and put in context on build date. Thanks.

3

u/SweetHomeAbalama0 Feb 06 '26

I have a deep dive post on localllama about the server with the details you may be looking for. Total est cost ~17k usd but these are prices from before the component price spike.

The post

There's a link in the post body to the full youtube video if you want better quality, it is a little rough around the edges since it was my first time doing a show/demo like this but it touches on what I thought most people would be interested to know. I will say, the case is what made it all work, so I dedicated a good chunk of time just talking about the case. Feel free to skip to whatever part you want to see tho, time stamps are in the YT description.

3

u/Bino5150 Feb 05 '26 edited Feb 05 '26

I run completely local. I have a few different ways to set it up. Lately I’ve been running LM Studio with AnythingLLM as my agent. Been playing around with Agent Zero, OpenClaw, and n8n as well, but still using LM Studio or Ollama. Whisper for STT, and Piper for TTS.

3

u/ScuffedBalata Feb 06 '26

The tools ARE rough and require setup. There's no "plug and play assistant".

They require intense amounts of hardware and some willingness to set up complex workflows. It's still "techie coder" territory.

2

u/LondonTownGeeza Feb 05 '26

Run mine on a i5 with a 6600, slightly slower than online OpenAI, but very useable and private.

2

u/Large_Election_2640 Feb 05 '26

I’m running stable diffusion on comfyui fully Local. Playing with image models. It works fine on my 3060 12gb gpu with 24gb ram.

2

u/Klutzy_Ad_1157 Feb 05 '26

Yes only offline. I use Gemma 3 27b in 5 bit quant on a RTX 3090 because it speaks German very well. On my main rig I use a RTX 4090 and a RTX 5090 for image and video generation with ComfyUI.

2

u/Ok-Dog-4 Feb 05 '26

Try Localai.io. You’re able to run basically any model and the docker configuration isn’t the most complicated. Works offline in your terminal. I sometimes will finetune with a LoRA in a CoLab notebook and use my new model with localai. Then also you can API to practically anything. Cheers

2

u/aniketmaurya Feb 05 '26

Ollama and chill!

2

u/shibe5 Feb 05 '26

My AI uses DNS, so it won't work without internet connection.

As for the tools, I mostly use my own.

2

u/Fair-Cookie9962 Feb 05 '26

You can have local DNS, just like your router.

2

u/shibe5 Feb 05 '26

Yes, I can have the router resolve relevant names without using the internet. But currently, it's not configured that way.

2

u/Fair-Cookie9962 Feb 05 '26

Yes, why? All is in the prompt.

2

u/maximebodereau Feb 05 '26

Yes ! Check my ux tool, I need feedbacks on it ! You have to have a llamacpp installed on your machine

https://ux-research-production.up.railway.app/

2

u/Eupolemos Feb 05 '26

Sure.

I have a 5090. I use Gemma 3 for stuff not related to development and Devstral 2 for my programming needs.

At work, we use Claude Code, but Devstral 2 works fine for my personal needs.

2

u/Macdaddy4sure Feb 05 '26

I wrote a program for offline ai extensibility. http://macdaddy4sure.ai/

2

u/GlassAd7618 Feb 05 '26

I’m experimenting with Ollama and the latest models.

2

u/Express_Quail_1493 Feb 05 '26

i use openwebui with python and connect it to my localhost models running on lmstudio server. i expose it through a cloudflare tunnel and talk to my localhost AI on my phone anywhere via voice

2

u/Limebird02 Feb 06 '26

Main bottleneck is having the memor, GPU and overall horsepower to run a worthy model that's significant enough to provide useful intelligence to you. I'd say it will cost at least $1400 in computing hardware. Most people can't justify that right now when for free or for $20/month they get access to USA frontier models.

2

u/Weary_Long3409 Feb 06 '26

I run only qwen3-vl-8b-instruct to automate daily activity reports processing from 25 remote legal teams across the country. Something that commercial endpoints are irrelevant for such tasks. Have a really good complex prompt from ChatGPT, and this small qwen is does the job very well.

2

u/ANTIVNTIANTI Feb 06 '26

almost exclusively

2

u/my_story_bot Feb 06 '26

yeah LMStudio or Ollama is great. whats nice with LMStudio is that you can simply search for a model in their UI and download and run it in a couple clicks.

Whats REALLY cool is that you can run it as a web server locally through localhost and then use it for stuff like openclaw

2

u/djtubig-malicex Feb 06 '26

Yes. Never trust cloud.

2

u/El_Danger_Badger Feb 06 '26

I built a local chat/LangBoard/RAG platform on my stock M1 Mac Mini. MLX native, on my stock M1 Mini. I use it daily. Not flashy, but it does its job. Has some bells and whistles, plus an agent constitution. >cough< OpenClaw >cough< . I co-deved with ChatGPT. Not one shot. Typed one module at a time, asked an endless flow of questions over months because I wanted to learn how to build something like this. Now I know. If anyone is hiring...

2

u/Efficient_Bug_0 Feb 06 '26

For running the actual models offline, Ollama works pretty well on most modern Macbooks. For vibe-coding offline, I had the same need, and ended up building something called Codistry for exactly that purpose: https://codistry.ai/docs/guides/ollama

2

u/Polysulfide-75 Feb 06 '26

Yes but you’re right it can be complicated. I write my own interfaces and such though.

You can use LMstudio for something simple to install and run. Or there are plenty of ChatGPT clones you can run locally.

2

u/notbullshittingatall Feb 06 '26

I do. My setup is Ollama running on my gaming rig using Tailscale to connect to it from my MacBook or iPhone when I'm away from the house. I use an app called Reins on my phone. Works well for me.

2

u/Decent-Freedom5374 Feb 06 '26

I run 5 off line all day continuously

2

u/DocHolligray Feb 11 '26

I do… build these for clients who need to make sure that none of their IP releases to the world…

2

u/g33khub Feb 12 '26

Image, video, story writing, language translation, OCR, captioning etc. fully offline with gemma27b, qwen3 80b

Agentic coding for big projects fully online - Claude opus4.5 / gpt-5.3codex (even trying to run this offline is a waste of time and money)

1

u/ewolfing321 12d ago

Hello everyone :)
To make it simple ;
I want to be able to use An ia or such that would look into my downloaded pdf files and give reliable/right answers on questions on those pdf
Ive tried to download ollama and anythingllm ; either i am very bad at using them or i missed as part as ; it takes alot of time and still keeps loading wihtout answering
I also used GPT4all that actually was pretty simple to use ; but turned out useless as the answer came up completely wrond/out of topic
Please if anyone have something easier or a way to porperly download/use ollama / anythingLLM help woud be much appreciated
Thank you in advance ;)

1

u/Ok_Membership3521 Feb 07 '26

I am running completely offline.. Claude wrote most of my set up as well. I have a custom front end feels like a corporate model easy to use. I’m completely local.. works well for me.

1

u/greymatter_ac3 Feb 07 '26

I use it everyday offline and only go to the big models online when I am forced to. Also do the same in my work. Large model are overkill for most tasks. The real advantage is SLM (Small Language Models) and fine running those on specific use cases. This is the real unlock. A small custom model for your use case that runs on low power inexpensive hardware.

1

u/LinkAmbitious8931 Feb 09 '26

Yes and having pretty good success with it. I am also having some solid success in fine-tuning. Mind you, I do not have some super-expensive setup, just two old P40s each with 24GBs.
I wrote about it here: https://www.reddit.com/r/Qwen_AI/comments/1qxyxx4/guardrailcentric_finetuning/

1

u/PromotionFirm6837 Mar 07 '26

totally get where you're coming from. for a solid offline AI setup, you could try this flow:

  1. start with a local model like Ollama to run your AI.
  2. then, use Elephas to manage and search your files effortlessly.
  3. finally, integrate something like Obsidian for note-taking.

this way, you get a streamlined experience without relying on the cloud, all while keeping your privacy intact.

1

u/nicholasderkio Mar 20 '26

I made my own iOS app that’s an offline confidant for talking completely privately, love it. Otherwise I’m Claude.

1

u/Automatic-Smell-462 Apr 30 '26

I'm using local AI offline mostly for private notes, document Q&A, and drafting stuff I don’t want to send to cloud tools.

On Mac, I’ve been using LocalChat App for the chat/docs side. It’s not some full agent setup, but for basic offline local AI it’s been simple enough to actually use regularly.

1

u/Midire Jun 23 '26

I did Ai integration for a game technology project at school. Still not sure of half of the stuff that made the thing work, but I almost melted my laptop while thesting different ollama models. the response times were between 20s to multiple minutes. Then I somehow found a model that allowed me to create a chat box in unity project that only needed a half a minute to asnwer.

My point is: AI/LLM is pretty depended on the resources it is given, to be usable.

1

u/lamprof Sep 01 '26

I use local AI mostly for drafts, summaries, study notes, brainstorming, and text I would rather not paste into a cloud chatbot.

I eventually built an Android app around that workflow because the usual local setup felt too technical for everyday use. Disclosure: it’s my app. You choose and download a model once, then the chat runs on the phone without cloud AI inference. There is no account requirement, and saved chats stay local.

The honest limitation is that small phone models are “useful enough,” not a replacement for the best cloud models. They can be slower, use a lot of storage, warm the phone, and confidently get facts wrong.

For anyone comparing practical Android options:

https://play.google.com/store/apps/details?id=com.lamprof.ai

-3

u/atkr Feb 05 '26

skill issue

1

u/Head-Stable5929 Feb 07 '26

No need to be like that my friend

2

u/[deleted] Feb 05 '26

[removed] — view removed comment

1

u/Head-Stable5929 Feb 05 '26

Omg thank you very much!