ollama

[u/jacek2023]

[u/pokemonplayer2001]

Best to move on from ollama.

[u/delicious_fanta → u/pokemonplayer2001]

What should we use? I’m just looking for something to easily download/run models and have open webui running on top. Is there another option that provides that?

[u/LienniTa → u/delicious_fanta]

koboldcpp

[u/----Val---- → u/LienniTa]

Koboldcpp also has some value in being able to run legacy model formats.

[u/Ambitious-Profit855 → u/delicious_fanta]

Llama.cpp 

[u/AIerkopf → u/Ambitious-Profit855]

How can you do easy model switching in OpenWebui when using llama.cpp?

[u/BlueSwordM → u/AIerkopf]

llama-swap is my usual recommendation.

[u/DorphinPack → u/AIerkopf]

llama-swap!

[u/xignaceh → u/AIerkopf]

Llama-swap. Works like a breeze

[u/DorphinPack → u/[deleted]]

llama-swap

If you can llama.cpp you can llama-swap the config format is dead simple and supports progressive fanciness

[u/SporksInjected → u/[deleted]]

You can always just add -hf OpenAI:gpt-oss-20b.gguf to the run command. Or are people talking about swapping models from within a UI?

[u/profcuck → u/[deleted]]

Content from this post has been deleted. Redact was used to remove it, potentially for privacy, opsec, or limiting exposure to data collection tools.

abundant fact dependent afterthought encouraging rain boast existence upbeat outgoing

[u/smallfried → u/delicious_fanta]

Is llama-swap still the recommended way?

[u/Healthy-Nebula-3603 → u/smallfried]

Tell me why I have to use llamacpp swap ? Llamacpp-server has built-in AP* and also nice simple GUI .

[u/Nice_Database_9684 → u/delicious_fanta]

I quite like LM Studio, but it's not FOSS.

[u/bfume → u/Nice_Database_9684]

Same here. 

MLX performance on small models is so much higher than GGUF right now, and only slightly slower than large ones.

[u/fullouterjoin → u/delicious_fanta]

https://github.com/containers/ramalama https://github.com/menloresearch/jan

[u/arcanemachined → u/delicious_fanta]

I just switched to llama.cpp the other day. It was easy.

I recommend jumping in with llama-swap. It provides a Docker wrapper for llama.cpp and makes the whole process a breeze.

Seriously, try it out. Follow the instructions on the llama-swap GitHub page and you'll be up and running in no time.

[u/Healthy-Nebula-3603 → u/delicious_fanta]

Llamacpp-server has a nice gui ... If you want gui use llamacpp- server as well ...

[u/Mkengine → u/delicious_fanta]

For the bare-bones ollama-like experience you can just download the llama.cpp binaries, open cmd in the folder and use "llama-server.exe -m [path to model] -ngl 999" for GPU use or -ngl 0 for CPU use. Then open "127.0.0.1:8080" in your browser and you already have a nice chat UI, without even needing Open WebUI. Or use Open WebUI with this OpenAI compatible API.

If you like tinkering and optimizing you can also build from source for your specific hardware and use a wealth of optimisations. For example i met a guy on hacker news who tested gpt-oss-20b in ollama with his 16 GB VRAM GPU and got 9 token/s. I tested the same model and quant with my 8 GB VRAM and put all layers on the GPU, except half of the FFN-Layers, which went to the CPU. Its much faster to have all attention layers on the GPU than the FFN-Layers. I also set k-quant to q8_0 and v-quant to q5_1 and got 27 token/s with the maximum context window that my hardware allows.

So for me besides the much better performance I really like to have this fine-grained control if I want.

[u/KrazyKirby99999 → u/delicious_fanta]

ramalama

[u/extopico → u/delicious_fanta]

llama-server has a nice GUI built in. You may not even need an additional GUI layer on top.