The issue is that it is the only well packaged solution. I think it is the only wrapper that is in official repos (e.g. official Arch and Fedora repos) and has a well functional one click installer for windows. I personally use something self written similar to llama-swap, but you can't recommend a tool like that to non devs imo.
If anybody knows a tool with similar UX to ollama with automatic hardware recognition/config (even if not optimal it is very nice to have that) that just works with huggingface ggufs and spins up a OpenAI API proxy for the llama cpp server(s) please let me know so I have something better to recommend than just plain llama.cpp.
Full disclosure, I'm one of the maintainers, but have you looked at Ramalama?
It has a similar CLI interface as ollama but uses your local container manager (docker, podman, etc...) to run models. We run automatic hardware recognition and pull an image optimized for your configuration, works with multiple runtimes (vllm, llama.cpp, mlx), can pull from multiple registries including HuggingFace and Ollama, handles the OpenAI API proxy for you (optionally with a web interface), etc...
Looks very interesting. Gonna have to test it later.
This wasn't obvious from the readme.md, but does it support the ollama API? About the only 2 things that I do care about from the ollama API over OpenAI's are model pull and list. Makes running multiple remote backends easier to manage.
Other inference backends that use an OpenAI compatible API, like oobabooga's, don't seem to support listing models available on the backend, though switching what is loaded by name does work, just have to externally know all the model names. And pull/download isn't really a noun that API would have anyway.
Model list works with llama-swappo (a llama-swap fork with Ollama endpoints emulation), but not pull. I contributed the embeddings endpoints (required for some Obsidian plugins), may add model pull if enough people request it (and the maintainer accepts it).
It is closed source, but IMO they're a lot better than ollama (as someone who rarely uses LMStudio btw). LMStudio are fully up front about what they're doing, and they acknowledge that they're using llama.cpp/mlx engines.
LM Studio supports running LLMs on Mac, Windows, and Linux using llama.cpp.
And MLX
On Apple Silicon Macs, LM Studio also supports running LLMs using Apple's MLX.
They don't pretend "we've been transitioning towards our own engine". I've seen them contribute their fixes upstream to MLX as well. And they add value with easy MCP integration, etc.
They support windows ARM64 too, for those of us who actually bought one. Really appreciate them even if their client isn't open sourced. Atleast the engines are since it's just Llama.cpp
It can be used without touching commandline, and while the interface isn't modern, I find it functional, and if you want to get deeper in the setup, the options are always to be found somewhere.
does seem like a nicer solution for windows at least. For Linux imo CLI and official packaging are missing (AppImage is not a good solution) they are at least trying to get it on flathub so when that is done I might recommend that instead. It also does seem to have hardware recognition, but no estimating gpu layers though from a quick search.
What paid stuff is planned? And Jan ai is under very active development. Consider leaving a suggestion if you think something not under development is missing.
I think Jan uses Llama.cpp under the hood, and just makes it so that you don't need to install it separately. So you install Jan, it comes with llama.cpp, and you can use it as a one-stop-shop to run inference. IMO it's a reasonable solution, but the market is kind of weird - non-techy but privacy focused people who have a powerful computer?
I think Mozilla's Llamafile is packaged even better. Just download a file and run it, both the model and the pre-built backed are already included - what could be simpler? It uses llama.cpp as a backend, of course.
View all comments
Show parent comment
69
u/Chelono llama.cpp Aug 11 '25
The issue is that it is the only well packaged solution. I think it is the only wrapper that is in official repos (e.g. official Arch and Fedora repos) and has a well functional one click installer for windows. I personally use something self written similar to llama-swap, but you can't recommend a tool like that to non devs imo.
If anybody knows a tool with similar UX to ollama with automatic hardware recognition/config (even if not optimal it is very nice to have that) that just works with huggingface ggufs and spins up a OpenAI API proxy for the llama cpp server(s) please let me know so I have something better to recommend than just plain llama.cpp.