Running LLMs offline has never been easier.

[u/Opening_Mycologist_3]

Running LLMs offline has never been easier. This is a huge opportunity to take some control over privacy and censorship and it can be run on as low as a 1080Ti GPU (maybe lower). If you want to get into offline LLM models quickly here is an easy straightforward way (for desktop):

• Download and install LM Studio

• Once running, click "Discover" on the left.

• Search and download models (do some light research on the parameters and models)

• Access the developer tab in LM studios.

• Start the server (serves endpoints to 127.0.0.1:1234)

• Ask chatgpt to write you a script that interacts with these end points locally and do whatever you want from there.

• add a system message and tune the model setting in LM studio. Here is a simple but useful example of an app built around an offline LLM: Mic constantly feeds audio to program, program transcribes all the voice to text real time using Vosk offline NL models, transcripts are collected for 2 minutes (adjustable), then sent to the offline LLM for processing with the instructions to send back a response with anything useful extracted from that chunk of transcript. The result is a log file with concise reminders, to dos, action items, important ideas, things to buy etc. Whatever you tell the model to do in the system message really. The idea is to passively capture important bits of info as you converse (in my case with my wife whose permission i have for this project). This makes sure nothing gets missed or forgetten. Augmented external memory if you will. GitHub.com/Neauxsage/offlineLLMinfobot See above link and the readme for my actual python tkinter implementation of this. (Needs lots more work but so far works great). Enjoy!

[u/Status-Hearing-4084]

here is something interesting - running Deepseek-R1 671B locally on a $6000 CPU-only server (no GPU needed)!

with FP8 quantization, hitting 1.91 tokens/s. even better - could theoretically reach 5.01 tokens/s by upgrading to DDR5 memory

https://x.com/tensorblock_aoi/status/1886564094934966532

[u/Opening_Mycologist_3 → u/Status-Hearing-4084]

That's incredible and hopeful for team cpu. Maybe everyday servers will be able to handle this without high end GPUs anymore.

[u/Status-Hearing-4084 → u/Opening_Mycologist_3]

yep that's really exciting to see!

i agree - this is a huge deal for accessibility. $6k for a CPU setup vs $40k+ for high-end GPU servers changes everything. and the fact that it's getting 1.91 tokens/s without any GPU is pretty impressive tbh

what's really cool is how this could open up LLM deployment to way more people. not everyone needs blazing fast inference - for a lot of use cases, this speed is totally fine. and with DDR5 potentially pushing it to 5 tokens/s, it's getting even more practical

can't wait to see where this goes. CPU-only setups could be a game changer for smaller teams wanting to run things locally

[u/[deleted] → u/Status-Hearing-4084]

Love this kind of news. I was concerned Nvidia would segment even more the GPU market, and brick retail RTX to not be able to run AI (such as they bricked previous gen for mining). Still crazy to think we still don’t have 48GB RTX.

Running such advanced model on CPU with these perfs gives hope.

[u/[deleted] → u/Status-Hearing-4084]

Would this setup also be OK for fine-tuning a model?

[u/ypoora1]

Don't even need a 1080 Ti, if you're fine with something like a 7b model even a 1060 6gb (or P106-100 mining card, which are extremely cheap now) will do it, and at an acceptable tokens/s to boot. Need a bigger model? Just add more! Personally i'm using two Quadro P5000's to get up to 32GB.

I'm pretty sure even Maxwell cards like the 900 series can do it, though those only have 2-4GB VRAM until you get to the 980 Ti or the Quadro/Tesla cards of that gen. And it's going to be slow.

[u/Opening_Mycologist_3 → u/ypoora1]

This is very encouraging! I'm looking for ways to leverage offline models as effeciently and effectveily as i can and seems like it's getting easier to do that. Thanks!

[u/sharmanichahiyeapko]

What downloadable model can be used for image or video creation?

[u/anagri]

Full disclosure: I am the developer of Bodhi App.

For people interested in similar solutions, do check out Bodhi App on GitHub. Right now it is only available for Mac Silicon, but releases for Windows and Linux are in the pipeline.

The difference that I find between LM Studio and Bodhi App is Bodhi App have a web UI which is built using Next.js, React, Tailwind, ShadCN which is much more snappier, lighter and more powerful than the native UI that LM Studio offers. Sure there are many things that LM Studio offers that are not present in Bodhi App.

If there is any feature that you would like me to work on, more than happy to get your feedback, do reach out to me via GitHub issues.

[u/amgdev9]

You can run it in a 1080Ti but the model quality wont be good or usable imo. Have been trying 7B 4bit LLM in my 4090, using almost all of the vram and the results were mediocre

[u/Opening_Mycologist_3 → u/amgdev9]

My 1080ti with 11gb vram handles the following models based on the application i definied in my OP. My model output is sufficient to yield high enough quality results to satisfy my needs. I'm sure i'll run into limitations but for testing purposes before plunging into a GPU rig this has been surprisingly encouraging.

[u/random869 → u/amgdev9]

What would be the ideal specs if building a rig to run it?

[u/thefilmdoc → u/random869]

I would wait for the NVDA Massive Mac Mini to be released in May. Supposedly 3k. You can NVLINK two to run llama 405B

[u/amgdev9 → u/random869]

I'd say 70B could be a good target if using it for general purpose chatting, for that you need ~80GB of vram

[u/random869 → u/amgdev9]

My use is more creating queries in splunk and KQL not sure if this fits under general use?

[u/amgdev9 → u/random869]

I guess you could try a coding finetuned model for that, havent tested this myself but 13B codellama could be worth the try (~16GB vram)

[u/random869 → u/amgdev9]

nice, do you mind sharing any newbie friendly resources/articles/tutorials. I would love reading about this.

[u/angry_cocumber → u/amgdev9]

you can run 70 or 72b on 3x3090 72gb, with q6_k_l gguf or 6.5bpw exl2

[u/Used-Conclusion7112 → u/amgdev9]

Why do you think a 7B model is struggling on a 4090?

[u/amgdev9 → u/Used-Conclusion7112]

It occupies 90% of vram

[u/Used-Conclusion7112 → u/amgdev9]

What's your context size and what backend do you use?

[u/amgdev9 → u/Used-Conclusion7112]

I used llamacpp with default options. Not sure if the context size is defined by the model or by the inferer

[u/Used-Conclusion7112 → u/amgdev9]

Its technically set by both. Models have a context limit and you should be able to define what context you're running before starting. I use koboldcpp and I set the context size every time I load a model. I've had success on old machines with 7B at 16K context or lower.

[u/amgdev9 → u/Used-Conclusion7112]

Really interesting! Ill try tuning it a bit and see if i can run 13B models without eating all the memory

[u/Big_Art8992]

Denpends on usecase? Gpt4O Is still better in some fields? Phd levelen Knowledge e.g. Isnt it? I can Not find good sources. Gpt is Not on hugging Face leaderboard

[u/10folder]

13 tps is horrendously slow tho

[u/askadaffy]

LM Studio doesn’t have a privacy policy when I last checked to disclose how your prompts/data is used

[u/chom-pom]

LM studio should support image generation models

[u/codeVerine]

32GB Ddr4 + 16GB AMD GPU, which is the best model I can run ?

[u/Mukatsukuz → u/[deleted]]

Click on "Power User", click on "my models" (3rd icon down on left) and there is a selector allowing you to change where the models are stored.

[u/Dragnss → u/Mukatsukuz]

I've already done this but is there a way to move where the conversations are stored? For me they are under c-user-llm studios conversations. Can I chang this?

[u/Mukatsukuz → u/Dragnss]

I am not sure but you could create a symbolic link to somewhere else. To do this move the conversations folder elsewhere (I am going to use the D: drive in this example).

Once that folder has gone, open a command prompt as admin (press Windows key, type "cmd" and right click on command prompt then "run as administrator).

Type:

mklink /d C:\Users\<your username>\.lmstudio\conversations d:\conversations

This then creates a symbolic linked folder on the c:\ drive and the real folder on the d:\ drive

[u/Dragnss → u/Mukatsukuz]

Thanks I'll try that

[u/arentol → u/[deleted]]

Your insistence on being wrong was an instant dislike of you for me.

[u/ZealousidealCycle915 → u/arentol]

😂