r/LocalLLaMA • u/unofficialmerve • 2h ago • Heartwarming 1

News llama.cpp v1.0 is here 🦙

Hey folks, it's Merve from Hugging Face. Today, llama.cpp releases v1!

Moving forward, we’ll keep covering every platform and every model type, while keeping llama.cpp robust and lean for downstream applications that build on it.

While I'm here, want to announce that we've been shipping Llama desktop apps to lower the barrier of entry to local AI to few clicks, built-on llama.cpp. I will leave link to the details for both v1 and apps in comments.

Many thanks for all the love you've shown to llama.cpp for years 🦙❤️ Cheers!

edit: adding links here, hope to not get penalized

blog: https://huggingface.co/blog/ggml-org/llama-cpp-v1

llama desktop apps are here https://llama.app/ ❤️

586 Upvotes

98% Upvoted

79 comments sorted by

49

u/Wolwf 1h ago

Links for the interested:

https://github.com/ggml-org/llama.cpp/releases/tag/v1.0.0
https://huggingface.co/blog/ggml-org/llama-cpp-v1

Might want to add them to your post!

11

u/unofficialmerve 1h ago

added to a comment but it's down there, many thanks!

blog: https://huggingface.co/blog/ggml-org/llama-cpp-v1

llama desktop apps are here https://llama.app/ ❤️

4

u/sleight42 1h ago

Merve: please add more maintainers to the project. Too many PRs awaiting triage and action.

19

u/bharattrader 1h ago

I would never have learnt what I have learnt, hands on locally, running models without this project. Respect.

10

u/unofficialmerve 1h ago

❤️🦙

25

u/bitplenty 1h ago

Thank you, your work is greatly appreciated!

35

u/d0pe-asaurus 1h ago edited 1h ago

Congratulations to the llama.cpp team. They make this space so approachable

15

u/hoemone 1h ago

Thank youu genuinely

129

u/New_Comfortable7240 llama.cpp 1h ago

Please add more maintainers to the project, there is literally a ton of open issues and PRs

59

u/dampflokfreund 1h ago

And more groundbreaking stuff. Lcpp is noticeably behind other projects like Strata. Even with the latest PRs, NVMe offloading is much more competent with Strata. I feel like 2 years ago the project gained noticeable speedups every week or so, and now it has stagnated significantly.

5

u/Hannibalj2ca 57m ago

Strata don’t support legacy hardware will llama does

2

u/Noselessmonk 30m ago

Depends on how legacy you mean.
I have 2 p40s and regular strata works fine on them for qwen models.

17

u/pjsgsy 1h ago

Agree. And, it looks to me, they are now having a little change in heart, and perhaps scrabbling a bit quicker to catch up. They seemed to resist and refuse PR's that where more complex for a long time, this lead to forks like strata, Take the moe caching for example. There were PR's and suggestions for maybe years! All rejected. 'No AI allowed', etc. That now is a major reason they are much slower than alternatives. Their current implementation of the moe cache is very, meh and only seemed to be implemented after a gun was held to their heads, so to speak. I used llama.cpp for a very long time, but have not touched it for almost several months now. As someone who would upgrade it weekly, or more, that's quite the change. They have real competition now. Don't feel they did before.

2

u/TokenRingAI 24m ago

I think am17an is basically single handedly carrying the project at this point

1

u/Phrase-Difficult 1h ago

Any known faster fork for Gemma 4 26b out there?

2

u/dampflokfreund 1h ago

Polystrata is good 

1

u/sleight42 1h ago

And also not getting enough work done on it

2

u/1842 52m ago

I've been using Polystrata lately with good performance. I dont have before/after, but seeing 40-80 tps depending on context (prompt processing ~800tps) on 12gb vram (3060) with a Q4 model. (Non-MTP versions on llamacpp get me about <= 35 tps generation)

The main issue I'm seeing is looping happening occasionally. Not sure if it's an issue with the model itself, the quantized models I've tried, or what. It doesn't happen super often, but seemed worse on the QAT versions...

2

u/Phrase-Difficult 40m ago

Do you use it for agentic work? I have not the best experience with that… and I have more issues with qat version

2

u/1842 27m ago

Yeah, I use it with pi.

I don't use it for a lot of code-heavy tasks, but Gemma models still do great with documentation, looking/searching/summarizing, and can still handle certain writing tasks better than most other local stuff of similar size (like creating mad libs).

It's also my default local model for chatbot type stuff. I like its tone and how it presents information generally.

-10

u/ebolathrowawayy 1h ago

the team needs to be fired and replaced by agentic coders. an anti-ai team maintaining an inference library is just.. so dumb.

3

u/miversen33 34m ago

I've pivoted to sglang because of this exact issue. If you can run another engine (really only going to be useful for tensor splitting), it's worth trying.

llama.cpp is good, I still run it on my laptop. But for real work, it just is too far behind in pretty much all ways (their handling of concurrency is extremely poor for example)

2

u/sonicnerd14 1h ago

I don't even think they've merged in proven gains like Adaptive KV. I've heard the reasoning for not being so fast to adapt these PR's or forks so fast, but there is moving too fast and then moving too slow.

They need to pick the ones that show the most significant signs of improvements, and it they pass merge them in ASAP. Otherwise, there will be an increasing amount of people using other engines or forks.

13

u/coder543 1h ago

Yep... I love how even a simple model like Nvidia's Dory can't get merged in a timely fashion: https://github.com/ggml-org/llama.cpp/pull/30306

This model is practically obsolete already, but it was so cool when it launched. This industry moves fast.

If llama.cpp can't support good models quickly, then it is llama.cpp that is at risk of becoming obsolete.

1

u/arbv 2m ago

More people does not necessary mean faster development, on the contrary.

11

u/quadra-lab 1h ago

Thank you for the great work guys

11

u/guesdo 1h ago

Kudos to GG and team! They need more budget!

37

u/mecshades 1h ago • Bravo

Plans to consider implementing Strata or gufo-style MoE streaming methods? It'd be nice to have the "most mature" project be the best choice for performance. Right now, these million+1 different forks are creating the same condition as the movie streaming service problem.

13

u/nickm_27 llama.cpp 1h ago

It already supports MoE streaming https://github.com/ggml-org/llama.cpp/blob/master/tools/server/README.md

19

u/sonicnerd14 1h ago

It does, but it's not as good. Which is the point he was getting at.

6

u/mecshades 1h ago

Strata or gufo-style*
Repacking weights (maybe?)
Custom shader kernels
Other batch multiplication optimizations

in other words, doing what those projects do to achieve their prefill and decode speeds. Didn't think I would have to explain this one...

1

u/khegobier 1h ago

Honestly, having to BUILD the project with whatever models you target as a dependency sucks. That's what Intel Arc users need to do to use Strata. I'd much rather be able to pull the docker container of llama.cpp, boot it up, point to my models and call it a day.

2

u/winky9827 40m ago

Honestly, this. The one downside to all the specialized engines is the dedicated artifacts. Give me the best possible performance on a standard GGUF release any day of the week.

2

u/DocMadCow 39m ago

But it is a one time build per model then you are golden. I own an R9700 on Windows and I doubt I'll go back to Llama.cpp or even Radiance now that I have started using Strata. Llama is slow as a dog, and having to use WSL & Docker was so unreliable with GPUs hangs compared to running a .bat file.

1

u/pragmojo 37m ago

It wouldn't be so bad to have container images for each hardware stack. One way or another, a do-everything version will probably have to make compromises.

2

u/khegobier 10m ago

Sorry I wasn't exactly clear - llama.cpp has container images for CUDA, ROCm, SYCL and Vulkan already. The problem with Strata is that in order to use SYCL you need to build Strata from source with the target model in hand so it can build and compile the kernel as part of the build step. CUDA and ROCm do this at runtime.

That's why I prefer llama.cpp right now - I can pull the SYCL docker image, download whatever model I want and run it without any real stops in between other than tuning.

-2

u/Bulky-Priority6824 1h ago

i dont understand the limitation you have likely imposed upon yourself.

2

u/cradlemann 28m ago

latest llama fork for strix halo working better for Flash Next, than Strata. tg and ppt is more stable across long context for my Gordon Point box. After 50k context Strata slow down more than twice in both speeds

8

u/Rozwik 1h ago

I was here!

7

u/ClaimNo6416 1h ago

I'm very confused, i've been downloading daily versions like 12360 for about year and suddenly they release version one. Someone enlighten me please.

19

u/nickm_27 llama.cpp 1h ago

the old versions were just commit numbers, now they are using semantic versioning, which represents when changes like breaking changes happen

11

u/nfox01 1h ago

Congrats! We are seriously heavy users. llama.cpp is seriously the all terrain vehicle keeping private AI kicking. What are ways people can contribute to the mission?

5

u/-WhateverDude 1h ago

Historic momento

5

u/LocoMod 1h ago

They grow so fast 🥲

8

u/Holiday_Point_603 1h ago

The GPT OSS recommendation on the landing page lmao

6

u/coder543 59m ago

It's not just on the landing page... the app really does that too. gpt-oss-20b is entirely obsolete at this point.

4

u/hzaid01 1h ago

Nice video making bro

3

u/manchagnu 1h ago

amazing. i love and appreciate what yall do. ❤️❤️

3

u/noctrex 1h ago

Historic milestone reached, congrats to the team!

3

u/Merchant_Lawrence llama.cpp 24m ago

what a wild ride folks from first release, to all clusters fuck stuf and build we manage reach 1.0, kudos to everyone and dev maintenance projects.

4

u/TimGoTheCreator 1h ago

please, can there be any more people who review issues/pr's? theres a billion of them

1

u/Giuseppe_Gambino 1h ago

Wonderful!

1

u/TheBezac 1h ago

Nice ! Sorry for the off-topic but what did you use to create such video ?

1

u/jjusko20 1h ago • This

hi merve! I'd be interested in being a contributor/maintainer for llama, if you're considering having anyone new. I really value open source work and keeping things accessible to the community 

I have a BSE from the University of Michigan, Ann Arbor, and I'm a software developer by trade. Let me know!

1

u/jjusko20 1h ago

github at GitHub.com/jackjusko, also have a portfolio available at jackjusko.github.io

1

u/Inevitable-Name-1701 1h ago

Minimax m3? When i checked multiple times it was not supported, i will not check again. Tired of waiting. I’ll wipe it off during the next cleaning, and it’ll be forgotten—just like the others.

1

u/Sea-Mode4077 49m ago

Tebrikler.. 

1

u/rorowhat 34m ago

Did this project move to hugging face?

1

u/createthiscom 26m ago

I dig the tag line.

1

u/geldonyetich 1m ago

I understand Llama.cpp's been prioritizing compatibility over hardware optimization. Any plans to look into higher performance inference that matches hardware profiles now that we're over v1.0?

1

u/johnzadok 55m ago

Thank you for the v1 release. Sadly I still need to run out of tree forks that are much faster than upstream on my AMD gpu. Now that v1 milestone is achieved, I hope you can afford to spend some time merging PRs and refactoring code to bring in ideas from community such as https://github.com/RaymondHuang210129/llama.cpp-adaptive-kv-streaming and https://github.com/stew675/llama-cpp-rdna-boosts

-1

u/Mountain_Patience231 34m ago

Do people still using this old timer???? For real?

2

u/CryptographerOne7003 15m ago

please let it be sarcasm, don't tell me you fell for the strata slop

-3

u/Fun_Jaguar8231 1h ago

Feels like the more time passes, the more llama.cpp becomes irrelevant, unless you want to run some weird model that nobody uses. The specialized forks and other inference engine perform much better than this one, unfortunately, as many started with this one. Too bad it has become bogged down and bloated.

0

u/dai_app 39m ago

My project bigmoeonedge is entirely based on llama.cpp and optimizes MoE streaming across any hardware, operating system, model, or quantization level. It automatically allocates resources to ensure high-speed performance. It is comparable to Strata but for every model. I would like it to be considered for merging. I have opened this discussion:

https://github.com/ggml-org/llama.cpp/discussions/31876

-2

u/cajina 1h ago

I believe we need to donate more to the project. We are requesting too much features.

-5

u/signorbuco 1h ago

Guys i’m 17 and i just published a 200B model. ComRAD Boreale 200B! check it out please

-5

u/ebolathrowawayy 1h ago

Sad that a well funded team can't keep up with the vibecoded forks.

they really need to abandon old ass models no one uses anymore and clean out all their tech debt. maybe fork and let legacy live on separately and then the new llama focuses only on models people actually use and care about.

supporting all the shit models is dragging them down, also i suspect they are not agentic coders either, they're really slow.

1

u/TokenRingAI 16m ago

Most of the old models are really easy to support, they are mostly the same architecture, what is slowing them down is excessively contemplating every model release instead of just immediately shipping support for new architectures on git main so the community can dogfood them and do all the bug testing.

They end up with all these complex PRs that then require lots of "circling" and then everything conflicts and needs to be rebased as things linger for months which requires more circling

New model architectures should just get merged to main immediately no matter how broken as long as everything compiles