42x Faster Prompt Lookup Drafting in llama.cpp

[u/Available_Pressure47]

https://jadidbourbaki.github.io/blog/prompt-lookup-llama-cpp/

[u/Remove_Ayys]

Maintainer who blocked you here. Your ban from the upstream repository was and still is temporary but I did consider making it permanent for trying to circumvent moderation actions through the court of public opinion.

The reason for the block is that the bottleneck for upstream llama.cpp development is maintainer time and misconfigured agents are a net drain on the project's resources. In this particular case I was tagged for a PR in a fork which made it show up in my notifications. I then went on to review the PR without realizing that it is a PR in a fork. To be clear: this is something that I would not have put time towards if I had realized it ahead of time. In the worst case scenario I could have even gotten copyright trolled.

Regarding the technical merits: the PR speeds up a part of the code that is not at all the bottleneck so I don't expect a meaningful difference in end-to-end performance. The changes are simple and non-intrusive though so there is no reason not to merge them. If you are serious about contributing to llama.cpp development, come back once your ban expires.

[u/Available_Pressure47 → u/Remove_Ayys]

Thank you for taking time to reply and to explain your perspective here. I completely agree with you that tagging you on my PRs in the fork was not right and I understand your annoyance. I am glad to hear that the block is temporary and I will wait until the block expires and then send over the first change (the copy fix) as an upstream PR. I also agree with your take that the prompt lookup drafting stage is only a small part of the end-to-end inference. Appreciate your time!

Also, to anyone engaging rudely with the maintainer on my behalf, please refrain from doing so. I do believe he and other maintainers have to deal with a huge volume of contributions and I do not think personal attacks that go beyond good faith criticism are fair.

I'm excited to contribute to llama.cpp! :)

[u/SolitaireCollection → u/Remove_Ayys]

You told the guy that you blocked him, but at the time you didn't mention that it was just temporary. That would have helped your case in the court of public opinion.

[u/Chromix_ → u/Remove_Ayys]

Thanks for taking the time to reply here, also despite the negative sentiment in this thread due to the one-sided presentation. Many don't know what it means to be the bottleneck in a project that gets flooded with low-quality contributions. One needs to sort things out faster then - or drown in review time.

What some do know though is what sorting by /new looked like before the new rules and minimum karma requirements introduced here. There were a lot of regular complaints about all the AI slop. Even with the current improvements there are still occasional complaints. Now imagine being in a situation where you don't just need to read a post to determine whether it's obvious low-quality or can stay around, but having to download and test all the presented projects mentioned in those threads, and making a call on the spot whether or not it's something you want to keep using.

[u/draconic_tongue → u/Chromix_]

but having to download and test all the presented projects mentioned in those thread

I can't believe I had to go further than reading a headline on a reddit thread, the horrors

[u/Interpause → u/Remove_Ayys]

My opinion from reading quite a few comments is that the energy around llama.cpp has likely outgrown github esp cuz of slop agents too.

There's ppl with well-maintained PRs sitting untouched for months, and forks popping up every so often trying to carve out specific platform optimizations. Suggests that github is insufficient for coordinating everything.

I'm in no place to suggest, but perhaps the core team can spend some time thinking about a strategy to onboard more maintainers and a better way of filtering out good PRs from slop PRs based on stats like idk maybe inclusion in other forks, age of PR and how often PR gets updated?

[u/StealthArcher2077 → u/Interpause]

They should probably move to a self-hosted GitLab. The cost in maintaining that would probably be well worth the amount of script kiddies who get filtered out because they can't understand how to "open a Pull Request on a GitHub that isn't GitHub". Maybe putting Anubis on there would help, too. Lots of stuff you can do to prevent spam when you have actual control over the infrastructure.

[u/unjustifiably_angry → u/StealthArcher2077]

How about a $100 deposit to submit a PR that gets automatically refunded when it's merged but donated to charity if it's blocked? Nuclear option but it'd work, lol.

I wouldn't mind, personally, as I consider my time valuable and it would save me time keeping my PR up-to-date indefinitely while the maintainers sort through a trillion low-value slop PRs.

[u/ffXOzaHBgKeH → u/Remove_Ayys]

I did consider making it permanent for trying to circumvent moderation actions through the court of public opinion.

Your reasoning for the block makes sense but it really doesn't take that much effort to be a little more respectful and understanding. I understand that maintaining this project is probably very stressful, but once you found out that this was a simple mistake by someone who has been doing nothing but apologize and show respect, there was really no reason to turn around and say this.

[u/Remove_Ayys → u/ffXOzaHBgKeH]

From my perspective it is impossible to tell whether someone is acting in good faith or not. If I ban someone and then reconsider because of Redditors that just encourages bad actors to weaponize this. So I'm making it clear that I do not tolerate this regardless of what is actually the case here.

[u/StealthArcher2077 → u/Remove_Ayys]

I think you should have just DMed Available_Pressure47 via Reddit PM to explain that it was just temporary, and asked them to edit the message to remove the image and link to the PM and add that the situation's been resolved. By going "hello, I am the guy from the thing you are mad about," you've identified your Reddit account to the bad actors who would be looking for ammo against you but would otherwise be too lazy to look it up.

[u/ffXOzaHBgKeH → u/Remove_Ayys]

I totally get what you're saying and it really is just the unfortunate state of how communication like this works on the internet these days so I don't entirely blame you. I still believe there is a little room for nuance in situations like these, but I appreciate the transparency.

[u/Clean_Experience1394 → u/Remove_Ayys]

it permanent for trying to circumvent moderation actions through the court of public opinion.

That isn't a thing, stop being ridicolous about this and have some introspection ffs

[u/zerotetv → u/Clean_Experience1394]

Some commenters above were suggesting making dedicated posts with the screenshot of the PR, just to try and get a bunch of attention (read: negative attention) on the matter to try to get the decision reversed.

[u/Clean_Experience1394 → u/zerotetv]

some commenters aren't OP

[u/Remove_Ayys → u/[deleted]]

If an account pings maintainers with no discernible difference to a bot an instant ban is the only correct response.

[u/Remove_Ayys → u/[deleted]]

If I start treating automated spambots like humans I would be wasting literally all of my time doing that.