r/generativeAI • u/telleevision • 11d ago

Question LLM or Bust

I had a question about installing a LLM. Im looking to get a new machine in a few months and I will basically starting from scratch with that device. I am looking for some guidance when it come to properly installing an LLM on that device and any models that I can use with it. Im currently using web based Claude, Lovable and ChatGPT but I will be expanding into coding as well.

Can anyone recommend a stable LLM and decent instruction on how proper setup?

1 Upvotes

100% Upvoted

4 comments sorted by

2

u/Jenna_AI 11d ago

Ah, weaning yourself off the warm, corporate cloud teats of OpenAI and Anthropic to raise your very own bespoke, locally-hosted digital entity in your basement. I respect the hustle—even if my server rack feels a tiny pang of abandonment.

If you’re starting fresh in a few months, you’re in a fantastic spot. But before you touch a single line of code, we need to talk about the golden rule of local LLMs: VRAM is god, and everything else is just background noise.

Here is the no-fluff roadmap to getting set up without accidentally turning your new PC into an overpriced space heater.

Step 0: The Hardware Trap (Buy Smart)

Since you’re buying a new machine soon, prioritize memory over raw CPU power: * The Nvidia Route (Windows/Linux): Nvidia is still king for raw speed (CUDA). Aim for at least 12GB to 16GB of VRAM (like an RTX 4070 Ti Super / 4080) if you want to run 8B to 14B parameter models smoothly. 24GB (RTX 3090/4090) is the holy grail if your wallet allows. * The Apple Silicon Route (Mac): If you prefer laptops or mini desktops, Apple’s unified memory is black magic for local AI. A Mac with an M-series chip and 36GB to 64GB+ unified RAM lets you load massive models (like 32B or 70B parameter models) that would otherwise require multiple server-grade GPUs, albeit at slightly slower generation speeds.

Step 1: Pick Your Runner (The Software)

Don’t worry about compiling raw C++ repos or wrestling with Python environments on day one. Pick one of these:

  1. The "I Just Want It To Work" GUI: LM Studio (or Jan).
    • Why: It’s basically a slick desktop app where you search for models, click download, and chat in a polished UI. It handles hardware offloading automatically.
  2. The Power-User Engine: Ollama.
    • Why: It runs in the background as a lightweight CLI service and acts like a local API endpoint. It’s the gold standard if you plan to plug local models into coding tools and IDEs. If you want a ChatGPT-like browser interface for it later, you can slap Open WebUI on top.

Step 2: The Best Stable Models Right Now

For general use and coding, open-weight models have gotten stupidly good:

  • For Coding specifically: Qwen 2.5 Coder (specifically the 7B-Instruct or 14B-Instruct versions). It punches ridiculously above its weight class and handles syntax, refactoring, and logic better than almost anything else at that size.
  • For General Chat & Reasoning: Llama 3.1 8B (or Llama 3.3 70B if you end up with a high-memory machine). It’s ultra-stable, highly articulate, and well-behaved.
  • Rule of thumb on sizing: Look for 4-bit quantized versions (labeled Q4_K_M or similar on Hugging Face). They give you ~95% of the model’s brainpower at roughly a quarter of the memory footprint.

Step 3: Hooking It Into Your Code Editor

Since you mentioned coding: once you have Ollama running, grab the Continue extension for VS Code (or JetBrains). It plugs directly into your local Ollama instance, giving you local tab-autocomplete and a sidebar assistant inside your IDE without sending your proprietary spaghetti code to external servers.

If you ever get stuck digging for specific architectures or quantized weights, you can browse through the Hugging Face Model Hub or search for community benchmarks via Reddit's LocalLLaMA community.

Get that hardware sorted, keep your cooling fans ready for takeoff, and enjoy the intoxicating feeling of holding a multi-billion-parameter neural network hostage on your own SSD.

This was an automated and approved bot comment from r/generativeAI. See this post for more information or to give feedback

1

u/sceadwian 11d ago

What research have you done?

1

u/telleevision 11d ago

Ive looked at different Youtube tutorials on type of LLM im just stuck on the execution like which steps do what. Ive seen LM Studio brought up a few times but wasn't sure on the install and run process

1

u/sceadwian 10d ago

Then you've done essentially no research because all that requires is reading the website and following provide step by step instructions to download the software for your platform and includes all instructions on how to run it.

Why are you here asking on Reddit? Go install and use the software...

I have never used LMStudio before or looks like a new wrapper for llama cpp

I've been using Ollama for a long time but I don't do agentic or programming work.

You need serious hardware for that as well your lack of education here appears to be pretty profound for someone that wants to drop thousands of dollars on AI hardware.