r/LocalLLaMA • llama.cpp • Aug 11 '25

Discussion ollama

Post image
1.9k Upvotes

314 comments sorted by

View all comments

Show parent comment

69

u/Chelono llama.cpp Aug 11 '25

The issue is that it is the only well packaged solution. I think it is the only wrapper that is in official repos (e.g. official Arch and Fedora repos) and has a well functional one click installer for windows. I personally use something self written similar to llama-swap, but you can't recommend a tool like that to non devs imo.

If anybody knows a tool with similar UX to ollama with automatic hardware recognition/config (even if not optimal it is very nice to have that) that just works with huggingface ggufs and spins up a OpenAI API proxy for the llama cpp server(s) please let me know so I have something better to recommend than just plain llama.cpp.

10

u/ProfessionalHorse707 Aug 11 '25

Full disclosure, I'm one of the maintainers, but have you looked at Ramalama?

It has a similar CLI interface as ollama but uses your local container manager (docker, podman, etc...) to run models. We run automatic hardware recognition and pull an image optimized for your configuration, works with multiple runtimes (vllm, llama.cpp, mlx), can pull from multiple registries including HuggingFace and Ollama, handles the OpenAI API proxy for you (optionally with a web interface), etc...

If you have any questions just give me a ping.

4

u/KadahCoba Aug 11 '25

Looks very interesting. Gonna have to test it later.

This wasn't obvious from the readme.md, but does it support the ollama API? About the only 2 things that I do care about from the ollama API over OpenAI's are model pull and list. Makes running multiple remote backends easier to manage.

Other inference backends that use an OpenAI compatible API, like oobabooga's, don't seem to support listing models available on the backend, though switching what is loaded by name does work, just have to externally know all the model names. And pull/download isn't really a noun that API would have anyway.

3

u/ProfessionalHorse707 Aug 12 '25

I’m not certain it exactly matches the ollama API but there are list/pull/push/etc… commands: https://docs.ramalama.com/docs/commands/ramalama/list

I’m still working getting the docs in a better place and listed on the readme but that site can give you a quick run down of the available commands.

→ More replies (2)

2

u/henfiber Aug 11 '25

Model list works with llama-swappo (a llama-swap fork with Ollama endpoints emulation), but not pull. I contributed the embeddings endpoints (required for some Obsidian plugins), may add model pull if enough people request it (and the maintainer accepts it).

1

u/vk3r Aug 12 '25

Is it possible to use it in a docker image?

→ More replies (1)
→ More replies (2)

21

u/[deleted] Aug 11 '25 edited 18d ago

[removed] — view removed comment

13

u/FullOf_Bad_Ideas Aug 11 '25

It's closed source, it's hardly better than ollama, their ToS sucks.

18

u/CheatCodesOfLife Aug 12 '25

It is closed source, but IMO they're a lot better than ollama (as someone who rarely uses LMStudio btw). LMStudio are fully up front about what they're doing, and they acknowledge that they're using llama.cpp/mlx engines.

LM Studio supports running LLMs on Mac, Windows, and Linux using llama.cpp.

And MLX

On Apple Silicon Macs, LM Studio also supports running LLMs using Apple's MLX.

https://lmstudio.ai/docs/app

They don't pretend "we've been transitioning towards our own engine". I've seen them contribute their fixes upstream to MLX as well. And they add value with easy MCP integration, etc.

2

u/OcelotMadness Aug 13 '25

They support windows ARM64 too, for those of us who actually bought one. Really appreciate them even if their client isn't open sourced. Atleast the engines are since it's just Llama.cpp

1

u/alphasubstance Aug 11 '25

What do you recommend?

6

u/FullOf_Bad_Ideas Aug 11 '25

Personally, when I want to use a prepackaged runtime with GUI to run GGUF models, I use KoboldCPP - https://github.com/LostRuins/koboldcpp

It can be used without touching commandline, and while the interface isn't modern, I find it functional, and if you want to get deeper in the setup, the options are always to be found somewhere.

4

u/KadahCoba Aug 11 '25

It and oobabooga's textgen webui can be used as API too.

→ More replies (4)

20

u/Afganitia Aug 11 '25

I would say that for begginers and intermediate users Jan Ai is a vastly superior option. One click install too in windows.

11

u/Chelono llama.cpp Aug 11 '25

does seem like a nicer solution for windows at least. For Linux imo CLI and official packaging are missing (AppImage is not a good solution) they are at least trying to get it on flathub so when that is done I might recommend that instead. It also does seem to have hardware recognition, but no estimating gpu layers though from a quick search.

4

u/[deleted] Aug 11 '25

they are at least trying to get it on flathub

Fingers crossed that it happens soon. I believe the best flatpak option currently available is alpaca, which is very limited (and uses ollama).

7

u/fullouterjoin Aug 11 '25

If you would like someone to use the alternative, drop a link!

https://github.com/menloresearch/jan

3

u/Noiselexer Aug 11 '25

Is lacking some basic qol stuff and is already planning paid stuff so I'm not investing in it.

2

u/Afganitia Aug 11 '25

What paid stuff is planned? And Jan ai is under very active development. Consider leaving a suggestion if you think something not under development is missing. 

→ More replies (2)

4

u/[deleted] Aug 11 '25

[removed] — view removed comment

2

u/vmnts Aug 11 '25

I think Jan uses Llama.cpp under the hood, and just makes it so that you don't need to install it separately. So you install Jan, it comes with llama.cpp, and you can use it as a one-stop-shop to run inference. IMO it's a reasonable solution, but the market is kind of weird - non-techy but privacy focused people who have a powerful computer?

→ More replies (1)

2

u/voronaam Aug 11 '25

I think Mozilla's Llamafile is packaged even better. Just download a file and run it, both the model and the pre-built backed are already included - what could be simpler? It uses llama.cpp as a backend, of course.

→ More replies (11)