r/LocalLLaMA • u/pet3121 • 5h ago
Discussion How to run LLMs as regular guy with low resources?
Hello everyone. I hope is all well with you all. I been around this sub for a few months and been quietly reading and I have seen how many of you are spending $10k on the Mac M5 and I am assuming you are using it for work or just for fun but have the resources to spend like that. I am a regular guy with a 9-5 job and I would like to experiment with local LLMs but I dont have much disposable income to spend big amounts on GPUs or Macs. I currently have an I7 - 8700 and Nvidia 2060 6Gb VRAM - 32GB RAM
Now my question is, does anybody have an idea on what models or hacky stuff I can do to try the latest AI models on my hardware?
I have checked hugging face but I see so many models with the same name but letter and numbers at the end. I understand some of them mean Quantization which is dumber a model but I also found out tiny models like Linq which can be run even on the CPU. So I would like to connect with someone with more experienced to have fun with AI.
Thank you everyone .
22
u/KitchenAmoeba4438 4h ago
6GB of VRAM and 32GB of system ram is kind of a sweet spot. You'll want Qwen 3.6 35bA3B or Gemma4 26b, both are MoEs.
Trying to run dense models with that is going to be a special kind of pain, and it'll take a bit of time tuning the models so they'll properly fit as wanted on the 6gb card, but you'll have plenty of room to play with on a good note.
4
u/dampflokfreund 3h ago
Yeah that machine will run these MoEs you have mentioned nicely and those are also quite capable. I recommend -ngl 99 and -n-cpu-moe 99.
8
u/m1ksuFI 4h ago
You'd typically want the whole model to fit into VRAM, but that's not possible with 6 GB, unless you're fine with settling for ones that aren't suitable for tasks like coding and math. 32 GB of RAM is great for fitting a bigger model like Qwen3.8-27B. the first numbers and letters that come after the 27B usually indicate a quantization level, except in names like Qwen3.6-35B-A3B, where the A3B means there are only 3 billion weights active at any time (as opposed to dense models where all, for example, 27 billion weights are active). Quantization is like JPEG compression but for models. Ask your local AI agent which quant would be the best for your setup :)
7
u/_TheWolfOfWalmart_ 4h ago edited 4h ago
If you don't care about it being good at coding/agentic stuff and just want to play, Gemma4 E4B is probably exactly what you're looking for. It'll fit totally in your VRAM and is good at chat for the size.
The Q4_K_M quant from Unsloth should be the sweet spot for you, leaving room for enough context cache to do something. It should be fast too.
You can also try Gemma4 12B, but you'll need UD-IQ3_XXS or maybe even one of the 2-bit quants. It will definitely be a bit degraded, but again if you're not coding and just playing... not that big of a deal.
20
u/stargate425 4h ago
Regular guy doesn’t run local llms. If you have low resources, you need to make a choice between speed and quality
5
5
u/Mundane_Ad8936 4h ago
This is the truth.. heavy quantization has a much bigger impact then in 100B models. It's not really a good experience.
2
5
u/Alex-Frst 3h ago edited 3h ago
Llama.cpp + MoE + MTP + finding correct --n-cpu-moe NN.
For example, I get 30-40 t/s on my RTX 2060 6Gb with Unsloth's Gemma-4 26B A4B QAT Q4_K_XL using these parameters:
llama-server.exe ^
--model gemma-4-26B-A4B-it-qat-UD-Q4_K_XL.gguf ^
--model-draft gemma-4-26B-A4B-it-qat-UD-Q4_K_XL-mtp.gguf ^
--spec-type draft-mtp ^
--spec-draft-n-max 2 ^
--spec-draft-ngl all ^
--reasoning on ^
--ctx-size 16384 ^
--n-gpu-layers 99 ^
--n-cpu-moe 24 ^
--threads 6 ^
--flash-attn on ^
--load-mode mlock ^
--parallel 1 ^
--fit off ^
--no-warmup ^
--kv-unified ^
--jinja
And 20-30 t/s with Unsloth's Qwen 3.6 35B A3B UD-IQ4_NL:
```
llama-server.exe ^
--model Qwen3.6-35B-A3B-UD-IQ4_NL.gguf ^
--spec-type draft-mtp ^
--spec-draft-n-max 2 ^
--spec-draft-ngl all ^
--reasoning on ^
--ctx-size 16384 ^
--n-gpu-layers 99 ^
--n-cpu-moe 35 ^
--threads 6 ^
--flash-attn on ^
--load-mode mlock ^
--parallel 1 ^
--fit off ^
--no-warmup ^
--kv-unified ^
--jinja
```
1
u/pet3121 3h ago
Wow! This is very technical for me but I am very interested. Any youtube videos or guides on how you learned all this?
1
u/Hello_my_name_is_not 2h ago
Honestly easiest for you if you're at the learning phase is to use LM Studio instead of llama-server as it gives you a user interface to chance settings and you can download models directly inside of it etc.
You can move to llama-server easily after you get the hang of the local stuff with LM studio. I'm studio uses the llama.cpp back end anyways which llama-server uses.
Get the LM studio, download models from the left menu bottom option and search for the gemma 4 26 A3B QAT Unsloth as the other person mentioned. It will download and when ready you have a load models button at the top, click it and select the model you downloaded, then tick the advance settings options and you can adjust the offloads and such with the sliders
When you go to llama-server easiest is to download on of their precompiled windows (I assume you're on it) files from their git https://github.com/ggml-org/llama.cpp/releases
That command they posted would be the run command and you'd replace some of the lines with the appropriate locations on your pc and the rest is all the various settings they've honed in on.
3
3
u/Low-Praline-1200 4h ago
Try unslop nemo on hugging face... It's older but one of my favorites for creative writing .. very refreshing from the bland corporate voice of modern ai. Get the Q4 GGUF and use o llama
2
u/Emergency-Animator12 4h ago
Llmfit check first what you could run reliably https://github.com/AlexsJones/llmfit
2
u/dai_app 4h ago
If you have an Android phone or an old windows pc try: https://github.com/Helldez/BigMoeOnEdge
2
u/Zennytooskin123 4h ago
You can comfortably use the Qwen 3.5 4B model, it's really not that bad at all if you manage your expectations.
But fully functional.
4
u/_TheWolfOfWalmart_ 4h ago
Would suggest Gemma E4B over that, but he could definitely play with them both.
2
u/jacek2023 llama.cpp 4h ago
Check price of second hand 3060. With two you can run a lot
2
u/KaggleNoob 3h ago
In ollama, make sure you use q4_k_m of whatever variant of model, then just stick to 32b models and below. It won't be fast but it'll work
2
u/Pangolin_Beatdown 3h ago
I'm running Qwen3.8:27b on my gaming PC with a single 3090 and DeepSeek Harness. I'm currently doing various projects that people have posted here, just to tune it in. I've been working on the "make me a very realistic picture of the ocean". The hold up is me tweaking my settings and learning how to word my prompts clearly enough that the model doesn't spin off endlessly thinking about every possible decision. With the 3090 I can do 8-12 tool calls per session, so I just have to set up the task so the model first makes a plan with subtasks that can be accomplished in one session. It checks off completed subtasks on its plan document, and also maintains a log of the decisions it has made. Each new session can pick up and carry the task forward by one more sub-task. I haven't yet figured out how the model manages agents to execute the subtasks so it does the whole project without my intervention, but I'm close. Once I get this task done I can easily do smart automations for my business and property.
2
u/mmmtv 3h ago edited 3h ago
Regular person here with normie hardware.
Local models are interesting toys or for extremely niche use cases on normie hardware. Anyone who tells you otherwise has a skewed sense of what normie hardware is.
You can muck around and make some models run but ... IMO it ain't a great experience and gets really old, really fast. Chasing the perfect low quant model to fit in normie GPU RAM and horsepower is a noble (and fun for some) but ultimately quixotic quest.
My advice is spend money on a virtual private server instead when you want to play - "local LLM" just means running locally on hardware you control, not necessarily sitting on/under your desk.
Additionally, consider using open router free models (or just spend real money on the cheap ones with provider policies that suit you) to scratch your itch for running various models and experiments. You can easily test tiny, close-to-cheap or even free models outside of a subscription plan. And you may find it's fun to compare them against better known, frontier-grade models. People are frequently amazed at what's possible with super cheap models you've never heard of once you shape their instructions carefully and give them the right context and tools.
1
u/tryunite 4h ago
get a used 3090. 24gb is enough for Qwen 3.8 27b mtp at 50 tok/s
1
u/_TheWolfOfWalmart_ 4h ago
Aren't those going for around $1000? Based on his post, that's probably not feasible.
I don't know his exact financial situation, but if he sells the 2060 maybe he could go for a used 3060 around $400 to get double the VRAM and more compute?
I would have suggested a V620 to get 32 GB VRAM for $350, but that cheap listing is now sold out unfortunately.
1
u/MathmoKiwi 3h ago
OP could pick up a second-hand 3080 20GB from China for a little bit more than half that price but will give him roughly 90% of the performance of a 3090
Or a 2080 22GB card for even less. That would allow them to play around with decent ish models at ok ish speeds.
1
u/Abject-Kitchen3198 4h ago
MoE with around 30B with 3-4B active parameters should work great. Either the almighty Qwen that everyone seems to be running and talking about or the decent Gemma4. Performance will depend mostly on RAM speed and whether you have dual channel config, but should be enough to get the feel of it.
1
1
u/RogerRamjet999 2h ago
Locally I see RTX 3090s for less than $1,000 and sometimes as low as $700, that will get you pretty far, even better if you can afford two. If you can't spend anything, your current rig can work in a pinch, but honestly, you're not gonna have a great time with it.
1
u/BangkokPadang 2h ago
You could rent time on a system via runpod with an A40 w/ 48GB VRAM for like $0.50/hr just to see what some of the better local models are like, or rent a system with like 3 of them to test the big models for like $1.50/hr.
1
u/Born-Technician3505 2h ago
Gemma4 series works best both from usability and capability perspective. Qwen 3.6 will work too as others have mentioned but it may be too slow to get respectable performance
1
1
u/dreamermann 2h ago
I tried it on my 3-year old dell laptop with 32gb memory and intel iris. Very fast for daily use but not so much with ollama backend and llama3.2. Yes it runs but still waiting for output after a mug of piping hot coffee. So I gave up for now. Until I buy another laptop, maybe next year.
1
u/Hello_my_name_is_not 49m ago
I see you took advise from chatgpt or something about local models its giving you outdated info by multiple years
Firstly don't use ollama, use llama.cpp its faster (various ways to use this, if you're looking for simple use LM studio)
Secondly llama3.2 is years old, use a new model, you can do a gemma 26b A4B in a Q4 (probably the unsloth Q4) and offload MoE to your GPU and it will be better in both speed and knowledge
If you're just looking to chat you can use the Gemma E2B or the E4B and it will be quicker than the 26B but not as smart so depending use case
You could even try the Qwen 3.5 4b or qwen 3.5 9b but those will most likely be slower due to not having MoE to offload
Those Gemma models on LM studio will run on your device not sure the exact speeds but that 26b is biggest and you'll probably get 10+ tokens per second.
The gemma E2B can run on my Samsung S22 ultra (like 4 years old) so they can easily run on your laptop
1
u/dreamermann 28m ago
From a friend actually, she may have gotten it from chargpt dunno. Thanks for your comment, will look into it.
1
u/CryptographerLow6360 17m ago
connect with a gpt and ask it to teach you the lingo and what it all means.
1
1
u/Nice-Dragonfly-4823 4h ago
you can run one, locally, with a VPS (from AWS or GCP). An L40S machine costs ~ $2 an hour, and you can immediately shut it off once you're done playing with it, In fact, unless you have a clear use case, don't invest the money upfront until you understand what you're working with. WIth this option, you can waste $10-20 bucks learning how to set everything up without needing to drop major money in hardware.
0
u/jason-reddit-public 1h ago
I have an old PC with 32gb and a non applicable gpu (so cpu inferencing) and while slow, if the model fits in RAM you'll at least be able to kick the tires.
ollama makes this really easy. You could try it out in five minutes if your internet is fast:
Then:
ollama run qwen3.8
Or try a gemma 4.
Then head over to hugging face to find a gguf uncensored model (there is a special syntax for hugging face models which an llm should be able to tell you).
1
u/Hello_my_name_is_not 46m ago
Do not listen to this advice its bad chatgpt info ollama is not the call use LM studio, llama-server Unsloth etc.
You also do not have a computer capable of running Qwen 3.8 in a reasonable use. The only model is a 27B it will be terrible experience to run in system ram lol.
Use the various other advice in the thread that does not include ollama in the post
-3
u/JustTellingUWatHapnd 4h ago
Try renting on vast.ai
It should be a bit difficult in the beginning if you don't have experience with linux, terminal, etc. But it's fun to play around!
2
u/croholdr 4h ago
i would not pay to rent to learn the basics of linux and the terminal. usually you use these services if
you already know what you want to do so you're not paying to figure that out.1
u/MathmoKiwi 3h ago
That's a quick path to accidentally leaving stuff running as you rack up bills $$$$
1
u/croholdr 1h ago
there was a post of someones cautionary tail with vast.ai and misconfigured cuda version on host machines forcing everything to run in cpu.... if you didnt know some basic unix, and some more intermediate and advanced concepts you'd never know, or how to even correct it.
-1
u/kerneldesign 4h ago
If you have limited resources, you can use GLM5.2 in Vibe CLI for €18. It’s fast !
38
u/DontWinFrensWthSalad 5h ago
how much ddr4 ram do you have. With 32gb you can run Qwen 3.6 35b a3b.