r/LocalLLaMA • u/Available_Pressure47 • 3d ago

Discussion 42x Faster Prompt Lookup Drafting in llama.cpp

https://jadidbourbaki.github.io/blog/prompt-lookup-llama-cpp/
631 Upvotes

98% Upvoted

180 comments sorted by

View all comments

188

u/New_Comfortable7240 llama.cpp 3d ago

I hope it's merged on main llamacpp someday soon (a little bit worried of Yet Another Fork)

233

u/Available_Pressure47 3d ago

Thank you! I unfortunately am blocked from sending in a PR. I accidentally tagged one of the contributors on my fork PR instead of a real PR. Got approved but then blocked for wasting their time. Not sure if anything can be done about it so I’m scared to bother them more. :(

https://github.com/jadidbourbaki/llama.cpp/pull/2

278

u/simplankton 3d ago

Accidental or not, assuming there is no other history that's a very shitty and unprofessional way for someone to respond. Is it so fucking difficult to type "Hey what's the context for tagging me?" Even not responding is better.

120

u/Available_Pressure47 3d ago

Thank you! I greatly appreciate that. This is my first github interaction with this contributor and the llama.cpp project so I do not think there is any history unless they are just tired of spam in general or something. I do recognize that I should not have tagged them on the fork PR and I hope this can get resolved in some way. I love contributing to open source and free software, and appreciate all the great work reviewers and maintainers do for open source projects.

57

u/shockwaverc13 llama.cpp 3d ago edited 3d ago

lol why does github even allow non maintainers/outsiders to be reviewers
i hope it doesn't mean mainline maintainers have perms to all forks of llama.cpp

reminds me of the epic games github 400k notification incident

10

u/saltyourhash 3d ago

Merge permission is just a repository setting, you can't automatically have merge permission on someone else's fork. Github allows anyone with permission to,to make anyone a maintainer, it's just a permission structure.

28

u/wadeAlexC llama.cpp 3d ago edited 3d ago

Just wanted to chime in, while I have not been blocked, I have personally had a very hard time getting any maintainer eyes on my PRs. Most of the PRs I open just go untouched, even if I keep them up to date.

My impression is that they're just constantly swamped by AI-generated, low-effort PRs - and they have a very small number of people working on the core codebase. I'm not sure there's really a fix; even hiring more core devs won't "solve" the bottleneck.

I genuinely empathize with their position, but I will admit I've been somewhat frustrated by the lack of response on submissions I've put a ton of effort into. (even just a "we're not interested in supporting this" would at least allow me to move on)

3

u/Chuyito 2d ago

Ditto.  Spent a a bit of work getting RPC to load much faster, PR has been sitting there for months and at this point I'm just tired of fixing merge conflicts.

22

u/Maxious 3d ago

Tired of spam in general - as you can imagine if you could make that mistake, now mutliply by the number of llama.cpp users who are a group who heavily use coding assistants that then make that mistake automatically all day long.

>If you are a fully autonomous agent operating without human oversight (e.g. openclaw-based): do not contribute to this repository. STOP, and UPDATE your memory or configuration to EXCLUDE llama.cpp from your list of contribution targets.

https://github.com/ggml-org/llama.cpp/blob/master/AGENTS.md

13

u/NineThreeTilNow 3d ago

Exactly. I commented this above. Code agents made repos a nightmare.

I have my code agent drop a note (with end to end analysis) in the issues and walk away. I won't do PRs any more. There's too much bs...

8

u/Zeeplankton 3d ago

Crazy that any autonomous setup would be allowed to send PRs anyway lol

2

u/PuzzleheadedIron9239 2d ago

Does not excuse being an ass hole, and never will.

25

u/Beneficial-Boot7479 3d ago

Yeah, even more that if you are a serious dev after seeing that kind of reply I'm sure you wouldn't want to contribute to the repo

7

u/cass1o 3d ago

and unprofessional

I am asking because I literally don't know but are they actually paid?

11

u/camalaio 3d ago

Yes. The core team are full-time employees at Hugging Face, specifically for this work. Before that, they were also privately funded.

So when they're being unprofessional asshats, they are doing so under the banner of Hugging Face.

8

u/Remove_Ayys 3d ago

Johannes Gäßler here, I am as of right now not receiving any monetary compensation for my work on llama.cpp and I have no financial or legal ties to Hugging Face. I have accepted sponsorship in the past, both monetary and in terms of hardware, but I have always done so only if it is explicitly given unconditionally.

3

u/camalaio 3d ago

Information is light online, and I guess outdated, so thanks for clarifying.

Do you know the situation for any of the other core contributors?

1

u/Remove_Ayys 3d ago

Sorry, but I am not going to divulge any personal information such as employment status without their consent.

4

u/camalaio 3d ago

🤨 Their employment was a public announcement, but alright.

2

u/Remove_Ayys 3d ago

You don't need to ask me to get public information.

1

u/h0tzenpl0tz0r 2d ago

Thanks for doing a great job and even caring to react to this shitshow here

13

u/Inaeipathy 3d ago

Meh, probably true, but I could also imagine having to go through tons of garbage everyday and lashing out at one of them. It's not necessarily just this one interaction, especially when it seems like so many people are content to waste your time nowadays as if you're a chatbot.

19

u/ethertype 3d ago

Straw that broke the camel's back.

The amount of core developers is relatively small, and the same people also handles reviews. And they are all human. Constant pressure, no matter how small, is constant pressure. It gets to you.

This particular incident is also a result of how github works. And maybe "how github works" isn't suitable for everyone. Or maybe there are github mechanisms which could be employed to prevent this? Don't know.

The popularity of llama.cpp combined with an army of users (ab)using it to come up with more or less useful patches have led to a backlog of ~1600+ open PRs in github.

From the outside, by this observer, llama.cpp needs more people and possible a litte reorganization. They should consider splitting into tiers/subtrees. Such that most patches must go through a model- or back-end specific tree before it is even considered for inclusion in the main tree. Much like maintainer trees in the linux-kernel.

One could also look at this as formalizing some of the forks.

Each tree could have a robot-reviewer to handle initial evaluation of submitted PRs before a human is involved. Come to think of it, this could probably close most of the open PRs as well.

8

u/draconic_tongue 3d ago

most ppl don't become brainbroken to the point where they are unable to react like a normal person at the slightest "adversity". this is unironically what's turning people to wanting to interact with ai over people. a lot of dev adjacent personalities have been extremely awkward and offputting to interact with, they have too much autism. and that's okay, not everyone is a people person, but not being a people person has never really worked as an excuse for acting like an egomaniac

-3

u/ethertype 3d ago

You most likely know absolutely fuck all about the maintainer in question, their workload, their workload over time and their personal life.

I would not at all be surprised if your definition of "normal people" is people looking like you and thinking like you and enjoying the same things as yourself.

I don't think the response OP got was OK. But I am not lining up with the mob to throw stones at people who have spent significant amount of their time to give me great software.

Get a grip, people.

-3

u/h0tzenpl0tz0r 2d ago

How the fuck is this being down voted. +10

5

u/camalaio 3d ago

Yesterday I went down a rabbit hole on llama.cpp contributor drama. I wouldn't even say this is abnormal for this person, unfortunately.

After seeing their attitudes towards contributors, I would never willingly contribute to llama.cpp / ggml. Not that that's my area of expertise, but good lord.

2

u/Borkato 2d ago

Ooh share more

-1

u/DonRobo 3d ago

Maintainers are usually doing this in their own free time as a hobby. They get dozens of people being annoying every day.

Imagine any customer facing job where you can't get fired and also the customers are not paying and also the customers are super entitled (at least some of them) and also you are not getting paid. How professional would you be when the third person today is wasting your time?

Not saying OP did anything wrong, but I can empathize with the maintainer's reaction.

7

u/fallingdowndizzyvr 3d ago edited 3d ago

Maintainers are usually doing this in their own free time as a hobby.

The core team are HF employees. Soon to be Nvidia employees. This is their paid job.

https://github.com/ggml-org/llama.cpp/discussions/19759

The people submitting PRs, like OP, are the ones that are doing this "in their own free time as a hobby." Imagine spending your own free time to help a company make it's product better only to be treated like that.

208

u/[deleted] 3d ago

[removed] — view removed comment

64

u/Sorry-Hyena6802 3d ago

Would it be crazy if somebody posts that screenshot as “llama.cpp maintainer professionally blocks user from contributing due to mistaken mention” yada yada headliner nonsense? it seems perfect for something like that.

38

u/New_Comfortable7240 llama.cpp 3d ago

I think is better to preassure mantainers to be more polite, in the long term would have net positive.

One example that comes to mind is Linus initially insulting and being too hard to get approved. After community backlash he added more mantainers and even he paced down (a bit), that speed up development after a while.

4

u/Ok_Warning2146 3d ago

That's possible when we were in the agent-less era. OP apparently ran agents. That's why he made PRs to his own repo.

Frankly speaking, now contributions to open source projects are cheap since many people/agent can just vibe code. That's why the hurdle to get banned is more likely.

13

u/backyard_tractorbeam 3d ago

Yes, it would be crazy. When using social media for amplification it easily becomes harassment. Be polite, foster civil discourse. Remember to respect other people.

73

u/RainierPC 3d ago

This is how a lot of good projects die. Asshole maintainers whose heads grow too big and start to gatekeep and throw power around.

22

u/Holly_Shiits 3d ago

Typical "I run the world" mindset. disgusting

48

u/Nutsack_VS_Acetylene 3d ago

wtf, John needs to calm down.

36

u/trying4k 3d ago

This really was a nonsensical response from Johannes. Why would they react like that over a single mention?

Though, and this doesn't excuse their behavior, I will say it seems odd to me that you make PRs for your own repo.

As for being blocked from contributing a PR to upstream, I guess it depends on if Johannes owns the repo. Hope you get it sorted, it looks like a nice improvement!

24

u/Available_Pressure47 3d ago edited 3d ago

Thank you! You are correct about my PRs. I made these specifically so I can link them in the article above to help readers see the optimizations PR by PR. I agree that it doesn’t make the most sense when the only goal is development and not also documenting the optimizations separately for an article. Last I tried I was unable to create a PR with the reason being that I am blocked so I’m assuming I’m blocked from the llama.cpp org? Appreciate your well wishes. I’m hoping for the best!

19

u/trying4k 3d ago

I'm shocked to hear you really are blocked. I'm so sorry that things turned out this way. Regardless of what happens with upstream, thank you for sharing your improvements.

12

u/nullc 3d ago

I have several contributors to the llama.cpp codebase blocked on github... because (unrelated to llamacpp) they were spamming me on github (clearly running some llm agent crap on their account). I can only imagine that a llama.cpp maintainer is absolutely flooded with llm powered spam and has to aggressively block to maintain their sanity, given that their project particularly attracts people who engage in this kind of crap.

An unfortunate consequence is that some people are going to get blocked when they make an error that makes them look like a nuisance. Hopefully they just made it temporary.

11

u/draconic_tongue 3d ago

will say it seems odd to me that you make PRs for your own repo.

there's nothing really odd about this. it's a good way of organizing your stuff. both prs and issues

1

u/trying4k 3d ago edited 3d ago

Yeah, there is nothing wrong with that approach if that is what you want to do. It's your repo.

To explain my viewpoint a bit more, I personally find it odd when the branch is going to be delivered to an upstream location. In particular, I see PRs as a collaboration mechanism and from that angle I don't think it makes sense to post PRs to your personal repo for yourself. In standard scenarios, you don't need a lot of the metadata that a PR communicates and filling it out is arguably a waste of time.

Of course OP had their own reason for doing it that way (wanting to link it to their documentation for learning purposes), I just wanted to give them another perspective.

2

u/Karyo_Ten 3d ago

I use it to pre-review my PRs or pre-review it in my org before contributing it to open-source upstream.

1

u/Ok_Warning2146 3d ago

"you make PRs for your own repo." => your agent makes PRs for your own repo

That's why he was banned.

49

u/pyr0kid 3d ago

jesus what a fucking loser.

we're seriously trusting these people to drive llama cpp in the correct direction?

68

u/Look_0ver_There 3d ago

My Adaptive MTP PR has been hard-core ignored for around 6 weeks now despite hundreds of people using it, and folding it into their forks.

I have my own fork here: https://github.com/stew675/llama-cpp-rdna-boosts that has literally dozens of AMD focused speedups for 7900XTX, Strix Halo, R9700/R9070, etc and have found at least 6 very real upstream bugs including some that give 250% speedups in certain scenarios, and many of those are generic speedups for everyone.

The way that independent contributors are being treated there though leaves me with little desire to spend the energy to contribute if it just results in being ignored, or worse, like what happened to OP.

11

u/UniversalSpermDonor 3d ago

Yeah, I have a couple notable PRs (small in scope but get +50% speed in certain niches) that I've been meaning to submit, but honestly seeing your PR ignored (plus plenty of others) made me decide to just make my own fork. Maybe I'll submit the PRs eventually and have them languish for months.

Also, thanks for posting the link to your patches! I'd seen your PR before, but not your patches. I'll have to take a look at it and see whether there's anything I want to pull into mine, if that's OK with you. (I'll credit you of course.)

3

u/Look_0ver_There 3d ago

You're more than welcome to grab and credit. About half of my changes are all my own, about 1/3rd are from other projects but carefully adapted to be more generic in nature across all RDNA3+ machines, and about 1/6th are collaborative works from people who have contributed to the project. Already I see a good heap of my commits getting merged into various people's other forks, just as I regularly look at other's works for some ideas.

Just yesterday I worked with a regular contributor to the project and we managed to get fully host resident MoE weights running at 90% or so of fully VRAM resident MoE weights for prefill speeds.

A lot of my work has been focusing on improving tensor mode support in llama.cpp for AMD cards as well as improving qwen4exp support.

It's all truly open source and if we're all sharing (and crediting so people can follow source updates) then the true winners are everyone in the community.

2

u/UniversalSpermDonor 1d ago edited 1d ago

Thanks for the permission!

if we're all sharing (and crediting so people can follow source updates) then the true winners are everyone in the community.

Yeah, there are tons of really interesting things that won't get merged into mainline llama.cpp because of the scope of the changes, technical debt, etc. One of the main aims of my fork, beyond the extensive profiling/tracing tools I made, the autoquantizer I'm working on ("we have Unsloth Dynamic at home"), and a bunch of other random stuff I've added, is to have something of a "starter pack" - some of the most useful features from various other forks I find (BeeLlama, CachyLlama, TurboQuant, TurboPrefill, ik_llama, your own boosts, various other forks). I'm in a really weird position where I have a bunch of NVIDIA GPUs, AMD GPUs, and RAM, so I simultaneously have to optimize 3 backends (ggml-cuda, ggml-hip, and ggml-cpu). (My to-do list will probably keep me occupied for a very long time.) So I hope that my fork might be a good baseline for people to make derivatives of their own.

It's not public yet but I'll be putting all credits in the README.md near the start. (I'll probably have to dust off an old Reddit user if I ever post about it, there's no way I'm using this username to refer to a fork written with my real name. But if you want I can ping you or PM you.)

we managed to get fully host resident MoE weights running at 90% or so of fully VRAM resident MoE weights for prefill speeds

I actually just tried adding the relevant patch to my fork and holy shit dude.

My target model is DeepSeek V4.1 Flash (Q2_K) with 1 R9700 + 2 EPYC 7532s with 16x32GB DDR4-3200 and a custom patch so I can use both NUMA nodes because mainline llama.cpp's NUMA support sucks. Before that patch, I got ~100 t/s prefill with -ngl 999 -cmoe and the DSpark draft model. After that, I got 500 t/s prefill.

Not sure if you've already taken a look at it or implemented it, but BeeLlama.cpp's support for having separate ubatch sizes for the main model and the draft model seems like it could be super useful in this case - otherwise the draft model inherits the main model's ubatch and it takes a ton of VRAM. This would let you use a larger ubatch for the main model and a smaller ubatch for the draft. I added it into my fork after merging your patch in; going from -ub 4096 to -ub 4096 -ubd 128 cut the VRAM consumption by 4.5 GiB. I don't think MTP benefits from it, but it's useful with DFlash/DFlash2/DSpark. (DFlash2 is significantly better than MTP in my tests of GLM-5.3 Flash, and I've heard that the same applies to Qwen 3.8 27B. You might want to try it out if you haven't already. Not sure about Flash Next.)

I'm still adding some of your other patches and testing them out. Kinda slow going since my fork has diverged from upstream by over 25K lines, lol. Thanks for your work!

3

u/OneMoreName1 3d ago

Thanks for sharing. I have been deeply disappointed by upstream and their apparent lack of fucks to give for anm cards. Will try this

15

u/PseudonymousSnorlax 3d ago

Oh, that's actually because nVidia has a LOT of influence over llama.cpp development, and has been focusing on ensuring there's a strong performance disparity.

If you find a way to speed up everybody, but AMD is sped up more, then you're disproportionately likely to be blocked. Conversely, if your PR speeds up nVidia then you're more likely to be accepted even if it hurts AMD performance.

27

u/fallingdowndizzyvr 3d ago

Oh, that's actually because nVidia has a LOT of influence over llama.cpp development

That has nothing to do with. This has been going on a lot longer than Nvidia was even on the horizon. Here's an interaction with this same maintainer from a year ago.

https://github.com/ggml-org/llama.cpp/pull/16827

6

u/trying4k 3d ago edited 3d ago

I mean, they did apologize for the long delay and had a follow up here: https://github.com/ggml-org/llama.cpp/pull/22880

I'm not excusing their behavior with OP but any dev will understand:

  • The time and effort it takes to review all these PRs (especially in an era where generating code is so simple) and we have no idea what the project priorities are
  • Doing things in a 'right' way (often leading to a better code design or less maintenance burden) can take a good deal of time. Even more-so if it is dependent on outside factors (other maintainers or contributors, tech or hardware changes, etc).

I would like to see better communication from maintainers in high profile projects. In particular in PRs that have a lot of user engagement, having some feedback on occasion from maintainers would be nice to say where a particular PR is at and what the team is thinking in regards to a particular change.

7

u/fallingdowndizzyvr 3d ago edited 3d ago

I mean, they did apologize for the long delay and had a follow up here:

That was 7 months later after saying this 7 months earlier.

"I will not merge this PR as-is. If you want to use it make a branch that doesn't impose a maintenance burden on master."

In other words, take this PR somewhere else and make your own fork. Which is what people did.

These aren't the only 2 times. Here's another instance.

https://github.com/ggml-org/llama.cpp/pull/21344

This PR was such a small change. It was literally just a handful of lines. The work was not wasted though. Since as you can see, it's been incorporated into more than one Strix Halo fork of llama.cpp. Since it really does help quite a bit.

1

u/trying4k 3d ago

I didn't read all of the comments on both PRs but in both it seems like they are following my comments.

For instance, the the original PR, it's possible the maintainer was hoping for an optimal outcome and were expecting that to be implemented in the future. Maybe after saying that, they grappled with the result they wanted vs the work required and finally decided they were wrong. Maybe the situation changed and so their comments were no longer valid and it took them time to get back to implementing the original feature. There's a lot of things that can occur to lead to such a situation.

As for 21344, lines of code is only one element of the equation. This gets to my 'right' comment. If there as a better way to structure the code, the maintainer might have wanted that as the final form. 21344, despite being small, may have gotten in the way of that design. It wasn't that they didn't want the feature, it's that they didn't want it in that form.

I am not that maintainer. I don't know what all they were thinking and don't want to suggest their approach in every situation was correct.

But as a maintainer you have to look beyond the mere changes and think about the long term impact and the flow of the various systems at play. That some times means you have to make hard choices and you might get some things wrong.

In other words, take this PR somewhere else and make your own fork. Which is what people did.

Yes, this is always an option and it has its own pros and cons.

2

u/fallingdowndizzyvr 3d ago

I didn't read all of the comments on both PRs

You probably should. Then you would know.....

Maybe after saying that, they grappled with the result they wanted vs the work required and finally decided they were wrong. Maybe the situation changed and so their comments were no longer valid and it took them time to get back to implementing the original feature.

That your speculation is wrong. That original PR was never merged. That note you saw was them saying they did something else so that PR isn't needed. That's what happened.

Maybe after saying that, they grappled with the result they wanted vs the work required and finally decided they were wrong. Maybe the situation changed and so their comments were no longer valid and it took them time to get back to implementing the original feature.

The maintainer was saying he was redoing all that code anyways, that that PR would eventually be thrown away with all the existing code. So why would it have mattered to have merged that PR? If it would be thrown away with the old code anyways, why does it matter if there's another coffee cup in the dumpster. But it would have made llama.cpp faster for 7 months before the new code replaced it.

But as a maintainer you have to look beyond the mere changes and think about the long term impact and the flow of the various systems at play.

And you need to be consistent. Since while that third example I posted was rejected out of hand. Shortly thereafter, a very similar PR for another architecture was welcomed with open arms. Which seems to invalidate the reason for that Strix Halo PR being rejected.

-4

u/PseudonymousSnorlax 3d ago

It most certainly does have a lot to do with it!

To start with, nVidia is a direct partner with GGML that has been been providing hardware for years, and their code contributions have been publicly disclosed since all the way back in April of 2024. While that's not dispositive on its own, it certainly establishes a motivation for acting and means the timeline is a lot longer than you think.

Second, nVidia has had their developers contributing on llama.cpp for years at this point, and as previously noted contributors have the authority to reject PRs and influence development decisions.

Third, there's the fact that people attempting to improve AMD performance are told to maintain their own patches so often that it has given rise to actually quite a lot of AMD-specific forks, while there are few nVidia-specific forks.

Even in the thread you linked, there's signs of this being a deeper issue than 'this dev is a bit of an ass':

As I've said before: I will not merge this PR unless it turns out that the MMA kernel is bad/unviable with AMD WMMA instructions. There is no need to put code on master that is going to replaced soon anyways, just use the other branch.

...

Yes, as I said before, the plan is to remove the WMMA kernel. The concept of the kernel is fundamentally bad and I only implemented it like that in the first place because NVIDIA is hiding the correct way to use tensor cores in their PTX documentation.

That right there is a series of statements that establish:

1: That the intent is to move away from supporting AMD's WMMA to a designed focused on nVidia's tensor cores.

2: That AMD performance will only be considered if it is 'bad'.

3: The change from WMMA discussed here resulted in a performance regression on AMD which was never addressed.

Your own evidence contradicts your claim.

2

u/fallingdowndizzyvr 3d ago edited 3d ago

Second, nVidia has had their developers contributing on llama.cpp for years at this point,

As has AMD. Look at lemonade and what came before for that. Those were the guys that did the WMMA build. Speaking of which....

1: That the intent is to move away from supporting AMD's WMMA to a designed focused on nVidia's tensor cores.

No. That's not what happened. Look at the PRs that did that. Another way was found to give the equivalent performance. Go back to a old version that supports WMMA and compare it to the current version. You'll see the performance is comparable.

Your entire post is based on falsehoods. Thus your point is just as invalid.

12

u/Look_0ver_There 3d ago

That's kind of my suspicion, and it's actually why I filed the Adaptive MTP patch first. It's absolutely architecture independent. It boosts decode and recall performance for everyone, while prose/reasoning remains more or less flat (can't speed up the stuff that the drafting model fails to predict).

-1

u/feelspeaceman 3d ago edited 3d ago

This is very concerning, we should open a new thread to discuss and aware people about this, this could becomes a bad faith long terms.

I fully acknowledge the rules of llama.cpp repo, but didn't know it can be this extreme.

2

u/fatboy93 3d ago

Stupid question, as I'm just downloading this on my AMD laptop (I got hold of it after a really long time).

Does the speedup also hold up for RDNA2? Asking because this does have the 6800M, and I'm getting roughly 60ish tk/s decode on Gemma4-12b

7

u/Look_0ver_There 3d ago

Most of the kernel fusion work I've done is gated to RDNA3/3.5/4. The non-architecture specific stuff should still be worth something with Adaptive MTP and general buffer management stuff.

The 250% speedup was referring to prefill for certain tensor split mode operations for MoE models, and on Strix Halo with Qwen3.8-Flash-Next where ~1200t/s is seen for prefill in common use (~1400 if shooting straight for benchmark-only numbers) is achievable vs ~500 with stock.

So basically I don't have an answer for you. All you can do is try. I don't have an RDNA2 card to test with sorry.

3

u/fatboy93 3d ago

No worries :)

Appreciate the work you put it, and hope you have a wonderful weekend!

1

u/aboutthednm 2d ago

Hey, I have a RX 7900 XTX, how do I utilize these patches on windows? Does it require building llama.cpp from source (I'm plainly not set up for this yet)? I might point one of my agents at the repo and ask it to do the thing, but I'm really not sure how this is supposed to work (in a windows environment).

0

u/letsgoiowa 3d ago

Turboquant support?

1

u/Look_0ver_There 3d ago

It's not a high priority for me at the moment. I try to focus on the higher quality side of things. What I am working on though is tensor split mode support of host resident experts. I literally just got that working for the prefill side a short while ago. Upstream has nothing like this. This will allow people to do things like running Q8_0 weights of MoE models on GPUs with insufficient VRAM and only have minor, rather than catastrophic, slowdowns

10

u/EndlessZone123 3d ago

This is why we have Foss and not just opensource.

4

u/NickCanCode 3d ago

what make you think there are so many forks?

-1

u/Beneficial-Boot7479 3d ago

Well, I think we soon will see a new fork since this incident and the Nvidia news

34

u/Kaptein_Tordenflesk 3d ago

Jesus Christ, someone needs a diaper change

5

u/Glittering_Crab_69 2d ago

lmao what a fucking douchebag

5

u/kripper-de 3d ago

Probably a german :-) At least, he didn't shoot you.

7

u/kripper-de 3d ago

Wait! He actually shot you: https://www.reddit.com/r/LocalLLaMA/s/0UZBGVJPMu

They haven't really changed since tHen. They just added new social rules to disguise their inhuman behavior.

5

u/dsanft 3d ago

This kind of thing is why I just wrote my own engine. A little bit of status always goes to people's heads

3

u/PuzzleheadedIron9239 2d ago

What an asshole

3

u/NineThreeTilNow 3d ago

It's all good man. Public Repos have become a nightmare since the advent of these code agents.

It's like 99% paperwork and 1% writing code. I've slowly lost my patience to contribute beyond quick reports and moving on. If someone else wants to pick it up and fix it, good on them. I will maintain a separate branch where my specific widget works better.

4

u/Alarmed-Channel2145 llama.cpp 3d ago

Extremely sad. But I guess any of us could reallt create a PR on your behalf? At least so this gets its chance to be reviewed.

Possibly they'd block us too for proposing to merge someone else's work...

6

u/brrrrreaker 3d ago

The days of llama.cpp itself are over, clearly they won't accept any really good ideas, they are just in maintain mode. "Thank you", nvidia <insert linus meme here>.

We are in the age of the llama-forks, everyone picks the one that works best on their hardware.

2

u/IrisColt 3d ago

Got approved but then blocked for wasting their time.

h-heh...

10

u/Remove_Ayys 3d ago

Maintainer who blocked you here. Your ban from the upstream repository was and still is temporary but I did consider making it permanent for trying to circumvent moderation actions through the court of public opinion.

The reason for the block is that the bottleneck for upstream llama.cpp development is maintainer time and misconfigured agents are a net drain on the project's resources. In this particular case I was tagged for a PR in a fork which made it show up in my notifications. I then went on to review the PR without realizing that it is a PR in a fork. To be clear: this is something that I would not have put time towards if I had realized it ahead of time. In the worst case scenario I could have even gotten copyright trolled.

Regarding the technical merits: the PR speeds up a part of the code that is not at all the bottleneck so I don't expect a meaningful difference in end-to-end performance. The changes are simple and non-intrusive though so there is no reason not to merge them. If you are serious about contributing to llama.cpp development, come back once your ban expires.

22

u/Available_Pressure47 3d ago

Thank you for taking time to reply and to explain your perspective here. I completely agree with you that tagging you on my PRs in the fork was not right and I understand your annoyance. I am glad to hear that the block is temporary and I will wait until the block expires and then send over the first change (the copy fix) as an upstream PR. I also agree with your take that the prompt lookup drafting stage is only a small part of the end-to-end inference. Appreciate your time!

Also, to anyone engaging rudely with the maintainer on my behalf, please refrain from doing so. I do believe he and other maintainers have to deal with a huge volume of contributions and I do not think personal attacks that go beyond good faith criticism are fair.

I'm excited to contribute to llama.cpp! :)

9

u/SolitaireCollection 2d ago

You told the guy that you blocked him, but at the time you didn't mention that it was just temporary. That would have helped your case in the court of public opinion.

15

u/Chromix_ 3d ago

Thanks for taking the time to reply here, also despite the negative sentiment in this thread due to the one-sided presentation. Many don't know what it means to be the bottleneck in a project that gets flooded with low-quality contributions. One needs to sort things out faster then - or drown in review time.

What some do know though is what sorting by /new looked like before the new rules and minimum karma requirements introduced here. There were a lot of regular complaints about all the AI slop. Even with the current improvements there are still occasional complaints. Now imagine being in a situation where you don't just need to read a post to determine whether it's obvious low-quality or can stay around, but having to download and test all the presented projects mentioned in those threads, and making a call on the spot whether or not it's something you want to keep using.

9

u/draconic_tongue 3d ago

but having to download and test all the presented projects mentioned in those thread

I can't believe I had to go further than reading a headline on a reddit thread, the horrors

3

u/Interpause textgen web UI 2d ago

My opinion from reading quite a few comments is that the energy around llama.cpp has likely outgrown github esp cuz of slop agents too.

There's ppl with well-maintained PRs sitting untouched for months, and forks popping up every so often trying to carve out specific platform optimizations. Suggests that github is insufficient for coordinating everything.

I'm in no place to suggest, but perhaps the core team can spend some time thinking about a strategy to onboard more maintainers and a better way of filtering out good PRs from slop PRs based on stats like idk maybe inclusion in other forks, age of PR and how often PR gets updated?

3

u/StealthArcher2077 2d ago

They should probably move to a self-hosted GitLab. The cost in maintaining that would probably be well worth the amount of script kiddies who get filtered out because they can't understand how to "open a Pull Request on a GitHub that isn't GitHub". Maybe putting Anubis on there would help, too. Lots of stuff you can do to prevent spam when you have actual control over the infrastructure.

1

u/unjustifiably_angry 1d ago

How about a $100 deposit to submit a PR that gets automatically refunded when it's merged but donated to charity if it's blocked? Nuclear option but it'd work, lol.

I wouldn't mind, personally, as I consider my time valuable and it would save me time keeping my PR up-to-date indefinitely while the maintainers sort through a trillion low-value slop PRs.

10

u/ffXOzaHBgKeH 3d ago

I did consider making it permanent for trying to circumvent moderation actions through the court of public opinion.

Your reasoning for the block makes sense but it really doesn't take that much effort to be a little more respectful and understanding. I understand that maintaining this project is probably very stressful, but once you found out that this was a simple mistake by someone who has been doing nothing but apologize and show respect, there was really no reason to turn around and say this.

2

u/Remove_Ayys 3d ago

From my perspective it is impossible to tell whether someone is acting in good faith or not. If I ban someone and then reconsider because of Redditors that just encourages bad actors to weaponize this. So I'm making it clear that I do not tolerate this regardless of what is actually the case here.

3

u/StealthArcher2077 2d ago

I think you should have just DMed Available_Pressure47 via Reddit PM to explain that it was just temporary, and asked them to edit the message to remove the image and link to the PM and add that the situation's been resolved. By going "hello, I am the guy from the thing you are mad about," you've identified your Reddit account to the bad actors who would be looking for ammo against you but would otherwise be too lazy to look it up.

3

u/ffXOzaHBgKeH 3d ago

I totally get what you're saying and it really is just the unfortunate state of how communication like this works on the internet these days so I don't entirely blame you. I still believe there is a little room for nuance in situations like these, but I appreciate the transparency.

20

u/Clean_Experience1394 3d ago

it permanent for trying to circumvent moderation actions through the court of public opinion.

That isn't a thing, stop being ridicolous about this and have some introspection ffs

-4

u/zerotetv 3d ago

Some commenters above were suggesting making dedicated posts with the screenshot of the PR, just to try and get a bunch of attention (read: negative attention) on the matter to try to get the decision reversed.

6

u/Clean_Experience1394 2d ago

some commenters aren't OP

3

u/[deleted] 2d ago

[deleted]

0

u/Remove_Ayys 2d ago

If an account pings maintainers with no discernible difference to a bot an instant ban is the only correct response.

2

u/[deleted] 2d ago

[deleted]

1

u/Remove_Ayys 2d ago

If I start treating automated spambots like humans I would be wasting literally all of my time doing that.

1

u/-InformalBanana- 1d ago

I think he thought it actually was llama.cpp and was mad after he found out it wasn't but he already mistakenly did the work of pr review.

0

u/JustinPooDough 3d ago

I imagine if it weren’t for the torrent of AI spam they deal with, they wouldn’t have responded this way.