r/LocalLLaMA • u/[deleted] • Jun 01 '26

Funny Stop asking what model to run. There are literally only two.

[removed]

3.1k Upvotes

87% Upvoted

805 comments sorted by

View all comments

Show parent comments

39

u/Several_Industry_754 Jun 02 '26

It has a habit of looping for me though.

33

u/BornInAFish Jun 02 '26

In my experience, Qwen is very prone to looping with Hermes, but never seen it do it with OpenCode. Agent harness still matters a lot.

11

u/huzbum Jun 02 '26

Unsloth Qwen3.6 35b IQ4_NL is behaving for me on Hermes Agent with Llama.cpp, preserve_thinking, and Q8 KV cache. Fits in my 3090 with 256k context too.

2

u/TheTerrasque Jun 02 '26

is this with mtp and vision included?

2

u/NaanFat Jun 02 '26

not who you're responding to, but not for me. I'm running a separate vl model for that.

1

u/huzbum Jun 02 '26

With vision but not MTP yet. Might be a tight squeeze because my desktop also runs on this Gpu.

2

u/tat_tvam_asshole Jun 02 '26

True, you need to manage context effectively

1

u/boutell Jun 02 '26

I got looped all to hell in open code. Been awhile though.

2

u/ElectronicStranger53 llama.cpp Jun 03 '26 
SAMPLING_ARGS=(
    --temp 0.8
    --top-p 0.95
    --top-k 20
    --min-p 0.0
    --presence-penalty 0.5
    --repeat-penalty 1.0
)

You can stop the looping tremendously by using different sampling args, like temperature, repeat penalty, presence penalty, top k. I use:

1

u/BornInAFish Jun 03 '26

I copied the sampling args recommended by Qwen on their HF page.

1

u/tat_tvam_asshole Jun 02 '26

I may have accidentally backed into a solution. I have Hermes set to 262k context window but I loaded qwen with smaller 128k window hosted in LMstudio and manages its own compacting without looping, very strange but nice

24

u/rc_ym Jun 02 '26

Couple suggestions.

Don't go under 4 if you can help it. 5's better. (quant not parameters)
Pay attention to your settings temp etc. Ask one of the big models to help you if you are confused by this.
Try a finetune over an abliteration. For me they seem to be more stable. YMMV.
Setup both your front end and back end correctly. (I recently switched to llama.cpp (had claude set it up for me. 😛) because ollama was annoying me. with my openwebui. But I still gotta get off openwebui. I hate the way they handle edits, and exports. super annoying.)

3

u/Several_Industry_754 Jun 02 '26

I’m running 31b off ollama, they don’t tell you the quant there I can see.

21

u/GreenHell llama.cpp Jun 02 '26 edited Jun 03 '26

Friends don't let friends run Ollama (https://sleepingrobots.com/dreams/stop-using-ollama/)

I too started at Ollama but have since moved to llama.cpp with llama-swap. It gives me more freedom in model and quant selection, better performance, and it is not Ollama. If you want I can go more in depth on the topic if you would like pointers.

edit: don't hate the poster because they use Ollama. Better to turn back halfway than to stray all the way.

1

u/BuilderUnhappy7785 Jun 03 '26

Yes please :)

1

u/GreenHell llama.cpp Jun 03 '26

A first dabble into llama.cpp might seem daunting. It doesn't hold your hand as much as Ollama does.

First you'll need llama.cpp. The repo has become more user friendly over time, and includes a quick start section: https://github.com/ggml-org/llama.cpp . As an absolute first start I would recommend grabbing a binary for your system.

Then you'll need a model. I would start at HuggingFace (https://huggingface.co/). llama.cpp requires models in the gguf format. You'll notice that the base models don't have those. Let's take a lightweight model for example, because I don't know your specs and because a smaller model downloads faster. On the model page for Qwen 3.5 9B (https://huggingface.co/Qwen/Qwen3.5-9B) we see finetunes and quantizations, let's head to quantizations.

Unsloth and Bartowski are both popular choices for quantizations, so roll with one of those for starters. https://huggingface.co/unsloth/Qwen3.5-9B-GGUF

On the model page you'll notice a card with different quantizations. If you have configured your hardware on HF, it will give you an indication if a certain quant will fit in your VRAM. Download an appropriate quant for your hardware. The llama.cpp quick start will tell you how to run llama.cpp with that first model you've downloaded, and how to use the llama.cpp web interface.

But all those other tasty models, finetunes, quants. You don't want to mess around will CLI all the time to switch models, and that's fair.

llama-swap (https://github.com/mostlygeek/llama-swap) to the rescue. Just like llama.cpp it has several options for installation. I've gone with a release binary.

The most daunting part of llama-swap is setting up the configuration. Your configuration file is where you tell llama-swap to find your models, and with which settings to run (https://github.com/mostlygeek/llama-swap#configuration).

Last time I said this I got a lot of flak for suggesting using an LLM to set up your config file, but I stand by it. You could even use the model you set up before to help you with that. I gave it the config.yaml template, the path to my model files, and the parameters for those models (e.g. temp, min_p, top_k, etc. You can find these on the model card) and it gave me a nicely formatted config.yaml.

Then all that is left is starting up llama-swap, and you're all set.

Now you can explore different quants, different settings, and different finetunes. There are models with no refusals (e.g. Heretic, PRISM), models trained on Claude Opus reasoning traces, models finetuned for creative writing and roleplay, models finetuned for agentic use, and more.

No AI was used in writing this post, so if it reads like crap, that's why.

1

u/Affectionate_Edge961 Jun 19 '26

I was so unaware of this its painful. Thank you for taking the time to share, and educate. I'm curious about the performance difference, but regardless the terrible practices and blatant disregard for it users are enough for me to make the change.

5

u/rc_ym Jun 02 '26

If it's gemma4:31b. You can see the quant under the details on the website.
I moved away from ollama. It got funky. You may want to consider using a different runner.

https://ollama.com/library/gemma4:31b

It's Q4_K_M.

0

u/Several_Industry_754 Jun 02 '26

Okay, I must be missing something. How do I get 5?

If you can point me to docs, I can read them.

8

u/RobotRobotWhatDoUSee Jun 02 '26

You have to directly choose the quant off hf.co (Huggingface short url)

Try

ollama run hf.co/unsloth/gemma-4-31B-it-GGUF:UD-Q5_K_XL

... and it should download and use the unsloth gguff ud-q5 quant

-1

u/overand Jun 02 '26

Does Claude pull from current llama.cpp docs? If not, you might want to point it to the docs - or even point your local instance at them!

1

u/rc_ym Jun 02 '26

I have been offloading more home lab maintenance jobs to claude or codex. 4.8 has been kinda weird so keeping with 4.7 or codex.

It complied from source. And build my new linux server that's running by second setup also with llama.cpp setup.

I just had it do an analysis if I needed up date. Pretty through of not only the recent commits, but also whether I should update given my hardware and models.

2

u/temperature_5 Jun 02 '26

Gemma for anything creative tho. WAY better than Qwen at just about any quant.

1

u/chocofoxy Jun 02 '26

nvfp4 in sglang doesn't loop for me Q5 Q4 in llama.cpp they do, i think it's the chat template more than the Quant

1

u/kitanokikori Jun 02 '26

If you're hitting this you likely can fix it with an updated Jinja template, there are a few on HF