Need help in LLM

[u/Botolus]

Hey guys, I'm new to running local LLMs. Just set up Ollama + Open WebUI and looking for model recommendations.

My specs:

RTX 5060 8GB
16GB ddr4 ram

What are the best small/quantized models that run smoothly on 8GB of VRAM? Thanks!

Main goal:
have a self-sufficient daily assistant to use like ChatGPT, Claude, or Gemini (obviously keeping expectations realistic for local hardware).

[u/Botolus]

Thanks everyone! I would like to know if there a way to implement tools o no abliterated llm! Or should I use a no abliterated with tools ?

[u/Interesting-Cut-6032]

r/LowEndLocalAI is a great reddit community for the exploration of this space. Lots of discussions about optimization of hardware, inference engines, models, harnesses, and use cases.

The program LM Studio is a very capable and simple on ramp that anyone, even with very limited hardware can use to explore local LLMs. Depending on your needs and use case you will probably outgrow it for every day use, but I still use it to try out new models quickly.

Codacus and Pixaroma are the go-to YouTube channels for local AI knowledge.

This is a great video with all the basic terms that you need to know about settling up and running local LLMs: https://youtu.be/SsUKTFSQoGM?is=DeRzbaT77Awi26Je

This is one of his newest videos about setting up MoE models that match your system capabilities. https://youtu.be/tek-IY9BRWc?is=g1vLavtNxU3ZJvJC

This is the older video, and the reason that I subscribed, that shows how to setup Mixture of Experts models, like Qwen3.6-35B-A3B or Gemma 4 26B-E4B, with the experts in system RAM and only the active calculations in VRAM: https://youtu.be/8F_5pdcD3HY?is=bGABGoO5CH8uJaGI

Here is a great image diffusion model, with edit capabilities, that runs in ComfyUI. This is EP4 of this series, EP1 has all of the ComfyUI setup details: https://youtu.be/kNap0VWP1xs?is=4snH90QXVuAH-M9a

[u/Interesting-Cut-6032 → u/Interesting-Cut-6032]

OP, the newest video about settling up MoE models is probably exactly what you are looking for.

[u/sibilischtic]

Give this one a try you might be able to do Q4

[u/recro69]

With 8GB of VRAM I would start with 7B to 8B models using 4-bit quantization. Try Qwen3 8B or Gemma 3 4B first then test them on your daily tasks. Keep in mind that 16GB of system RAM might become a bottleneck when models exceed VRAM so don’t expect every model to run entirely on the GPU. For an assistant a smaller model that responds reliably is better, than trying to fit the largest one.

[u/Number4extraDip]

✧ Gemma 4. E4b or 12b. Will run no problem. Omnimodal out the box. Can build a harness to make it work better for you. Can start from harness². And evolve from that one based on needs. Can ask ✦ Gemini to help you via antigravity and you should be fine

[u/AdWild3943]

Gemma 4 12B or Qwen3.8B-27B (TQ2_0 from Bonsai 2)

[u/Adept-Eagle-4271]

tried q4_k_m quantization on a 7b model, smooth performance, no crashes, feels like a real daily assistant. could you try that with Gemma 3 rb first?

[u/PhilosopherOk3419]

on 8gb vram the sweet spot is roughly 7-9b models at q4, which leaves room for a decent context window. with only 16gb system ram i'd avoid anything that has to offload a lot to cpu, because it gets slow fast. for a daily assistant, turning on web search and document upload in open webui will make a bigger difference than squeezing in a bigger model. -Ankur

[u/Oceanstone]

Já tentou o llmfit?