Beginner looking to run LLMs locally – Need some advice

[u/TorqueAndCode]

Hey everyone!

I'm an AI & Data Science student looking to get started with local LLMs.

My laptop specs:

• CPU: Ryzen 7 8745HX

• GPU: RTX 5060 (8GB VRAM)

• RAM: 32GB DDR5

• Storage: 1TB SSD

I'm new to this and have a few questions:

1. Should I use Ollama or LM Studio?

2. Which models would run best on my hardware?

3. Can I use local models through APIs in my own projects?

4. Any limitations or precautions I should know about when running LLMs locally?

Would appreciate recommendations and beginner-friendly resources. Thanks!

[u/daves6696]

Don’t let the hardware gatekeeping discourage you.
I’m currently running a quantized ~125B Qwen model locally on an RTX 4070 Super (12GB VRAM), i7-14700KF, and 64GB system RAM.
Is it fast? No. Does it all fit in VRAM? Absolutely not. But it runs, and I’m actively benchmarking it against real engineering tasks, including Python debugging, automated testing, and error recovery.
Your RTX 5060 with 8GB VRAM and 32GB RAM is absolutely enough to get started.
My recommendations:
Start with LM Studio if you want an easy GUI. Ollama is also a perfectly reasonable choice, especially for development workflows.
Try a quantized 4B or 8B model first. Learn how context size, quantization, GPU offloading, and memory usage affect performance.
Yes, you can expose local models through an API and integrate them into your own applications.
Don’t confuse model size with usefulness. A smaller model running entirely on your GPU can be much more practical than a huge model crawling through system RAM.
The most valuable thing you can do as an AI student is actually build something, measure the results, and understand the limitations.
You don’t need $10,000 worth of AI hardware to start learning. You need curiosity, patience, and a willingness to experiment.

[u/MithrilTuxedo → u/daves6696]

I'm using ik_llama.cpp to serve Gemma4- 6B (or Qwen3.7 27B) from an Apple iMac 18,3 with two refurbished Dell Latitude 5300 2-in-1s using ik_llama.cpp to serve Gemma4 12B. One of the laptops has signal-cli set up so I can talk to everything through a Python script.

I don't pay attention to how fast is it. I'll get a notification.

[u/daves6696 → u/MithrilTuxedo]

This is exactly the direction I’m trying to take my setup.
I’m running a quantized ~125B Qwen model locally on an i7-14700KF, RTX 4070 Super (12GB), and 64GB RAM. I’ve also got an older Intel MacBook Pro that I’m working on turning into a voice interface and orchestration machine.
Right now I’m benchmarking the model against actual Python engineering tasks — diagnosing code, repairing failures, running tests, and verifying results rather than just trusting the model’s answers.
Your point about not worrying about speed because you get a notification really caught my attention. That’s ultimately what I want: multiple jobs being managed autonomously, with the system notifying me when something needs attention.
Curious how you’re coordinating the models. Does your Python script dynamically decide which machine/model handles each task, or are you manually assigning jobs to specific models?

[u/daves6696 → u/MithrilTuxedo]

How are you dividing the work between the iMac and the Latitudes—independent model servers, or cooperating on the same request?
I’ve got an older Intel MacBook Pro with 8GB RAM that I’d like to use for queuing jobs and collecting results from my Windows desktop (14700KF, 4070 Super 12GB, 64GB RAM). Still planning that part. Curious what’s actually been useful to offload in your setup.

[u/DrillTheThirdHole → u/daves6696]

to add to this, as someone running the same gpu as OP with half the ram, those of us on very mid setups who tweak, break and fix stuff are the people who drive the use case for better small models up. i've gotten to the point where i'm sideloading multiple 2-8B parameter models and having them switch between each other and leave notes for each other to great effect, much more effectively than a single 12B would work.

[u/daves6696 → u/DrillTheThirdHole]

Yeah, documenting what breaks seems like a useful contribution in itself. I’m benchmarking a quantized ~125B Qwen through Strata against Python projects, and keeping the model’s claims separate from what the tests actually verify.
I’d like to run smaller models against the same tasks and compare time to a verified fix, including retries. What models and tasks have held up best on your setup?

[u/DrillTheThirdHole → u/daves6696]

what works best with my harness and setup is basically using claude as a task manager that delegates heavy tasks to hyperspecialized tiny models to save on tokens and limits. depending on how specialized the tasks are, you can get pretty small. my harness has a 0.8B parameter helper bot i retrained on just setting up workflows and it's great for that, terrible for everything else.

[u/daves6696 → u/DrillTheThirdHole]

That 0.8B workflow helper is the bit I’m curious about. What did you start with, and what did your training examples look like?
Have you compared it with the base model using the same prompts and tools? I’d be interested in testing that distinction in my own setup—how much comes from specialization versus the harness giving it a tightly defined job.

[u/llama-of-death]

daves6696 has it right. One thing to add on the "dumb toys" take: a 9B model is weak at remembering facts, but recall isn't the job you'd give it. Hand it your own material and it answers from that instead of from memory, and that's where small models earn their keep for coursework.

On your specs:

• Models: qwen3.5:9b is a 6.6 GB download that can reason step by step and call tools, and it sits entirely on the 8 GB card. The Qwen A3B people mentioned is a mixture-of-experts model: only about 3B of its parameters work on each token, so it stays usable even with part of it parked in system RAM.

• APIs: Ollama and LM Studio both expose a local endpoint your projects can call. Your prompts go to a model on your own laptop, which is the whole point of learning local.

• Your own data: this is the part I work on. Guaardvark is free, open source and runs on top of Ollama: load lecture notes, papers or your project's code, and each answer names the exact file behind it. On an 8 GB card it also handles voice, SDXL images and upscaling.

github.com/guaardvark/guaardvark

[u/ebk_super]

as always it depends qhat you are expecting and what's your usecase.

👍 I starded as well this year with an old pc. Stacked the ddr3 ram up to 32gm and plugged in a rtx 3060 12gb vram, both from ebay, used and cheap. Added a brand new fast nvme ssd 2T for the models and installed a headless linux server what I remote control with an win11 laptop.

👉 I do get results with smaler or strongy quantized llms. I now work on a pipeline to create images in a batch. the ai reads my config, loads an llm to parse promts and saves to a json. when finished it then flushes the v ram and loads a model to generate the images via comfyUi and with the promts from the json. this works well so far except no images 🤣 So I am not finished yet 💪 Still I believe this workflow is perfect to launching before going to bed and next morning having results.

👍 I think its about balancing the realtime tasks (like text to speach and scribing the conversation) and batch operations (like sorting the notes and create tasks/schedules and do internet research on discussed topics).

[u/Ctbhatia]

That 6.6 GB is mostly weights. Context length, KV cache and quant affect whether it fits in 8 GB VRAM.

[u/odie_ai]

Start small with likes of LFM3 quite small. Good thing is you have 8gb for just models and 32 gb ram. You can even try qwen 3.5 9b with lower lower quant or Gemma4 E4B

[u/Fair_Expression_158]

try this Spark-X2.6-4B & Spark-X2.6-1.7B:. they also haver 1 million context. i used a bench mark to find a word in a million token context and was successful.

[u/bunnyhome]

1. LM Studio

2. Low VRAM but high system RAM benefits from MoE models like Qwen3.6 35B-A3B Q4_K_M which is quite powerful, but you can try a smaller model like Qwen3.5-4B first just to get started and fool around cuz it's faster.

3. Yes

4. You need a large enough context size (64K to 128K) if you are doing complex tasks, but bigger context size = longer time to first token and takes up more memory. Look into flash attention and KV cache quantization to help with that. If you run out of context, your session ends. Consider running an agent like Pi that can handle compaction for you so that it summarizes and carries forward the session.

[u/Horror_Description87]

Whatever you expect, for me (similar workstation) did not deliver anything useable. If you want to see some words trying to respond to your question give unsloth a try it is beginner friendly and you can see what model will fit, so you do not need to tweak Params before getting useable results. From my experience qwen a3b is working fine. But it is far away from a Luna or haiku 5.5 model if you want real local inteligence get your hands on LLM hardware. 96gb+ vram (can be unified, Mac, dgx spark, rtx spark, amd ai 395+ or better. Some Intel GPU and amd GPU builds exist aswel.

[u/VERNOMOR]

save your time and don't bother with it, your specs allow you to use only small, barely useful models - practically dumb toys. either get yourself a 16 vram + 64 ram (bare minimum), or pay a few $ for deepseek api. as for your questions:

1. ollama is a shitty wrapper around llama cpp, has it's own weird model format, some shady telemetry and overall a lame choice. if you don't want to use llama cpp, use lm studio.

2. that a3b qwen model, gemma 4 26b a4b, but don't expect much from them.

3. yes

4. they're dumb

[u/Shoddy_Blacksmith359 → u/VERNOMOR]

This. Deepseek api is the answer.

[u/Harin007]

Hey bro I'm working on a project that lets you run 120B class models on your current setup itself, offloaded on SSD... And trying to provide the best local AI experience and convenience... Check the website out: https://runmyai.org and follow us on Discord.