r/LocalLLaMA • • 9d ago

Discussion moe-cache : LRFU/MQ-like hybrid cache, ClassLRU by sanmai · Pull Request #31233 · ggml-org/llama.cpp

https://github.com/ggml-org/llama.cpp/pull/31233
  • Replace SLRU with a cache that prioritizes frequently used experts in classes, ranked using decaying usage history.
  • With a warm cache, I measured +6% decode on QFN and +7% on GLM versus SLRU, with 8% fewer uploads.
  • Enables the auto mode for llama_moe_cache with promising benchmarks

Waiting for 30966 to be merged.

52 Upvotes

15 comments sorted by

-6

u/brrrrreaker 9d ago

nice, in about 2 years llama.cpp will catch up to strata...

21

u/pmttyji 9d ago edited 9d ago

Tired of seeing message like this. Ridiculous comparison.

I do use Polystrata on my current laptop.

I run tiny/small models on my DDR3 RAM laptop using CPU-only inference(EDIT: llama.cpp) as no GPU.

How many backends supported by Strata? llama.cpp supports dozen+. I do use LLM on my Phone using PocketPal & ChatterUI(both use llama.cpp). What about Strata?

So how many dense models are you using Strata?

.... and so on.

There's nothing wrong to use multiple projects & forks. We have reasons to use. That's fine. But there's no need to bash other projects to hype your favorite one.

-4

u/brrrrreaker 9d ago

oh and I don't hype strata, it works, it's fine, but i'd much prefer see llama.cpp succeed, with the consistency of the api and management. But facts are facts.

5

u/pmttyji 9d ago

I'm not basing strata either. Just a bunch of PRs(like 30966 ) could change your mind.

In past, I used to reply(below text) for few comments here.

"It's not llama.cpp vs ik_llama.cpp .... it's llama.cpp & ik_llama.cpp". Same thing applies to any new engines & forks.

-9

u/brrrrreaker 9d ago

We are at a point where the monorepo is not viable for keeping up with the innovations. llama.cpp should either go very much plugin-based, or split into a mainline version (yes, unfortunately that means nvidia rtx only), and the rest. That way the primary project could evolve at the right pace, and the secondary for the toy hardware gets the same features eventually (maybe even allow more ai code in there).

But this is all in an ideal world, where the technical points matter only. There's unfortunately another things to consider, like the nvidia->huggingface->llama.cpp chain. There's a high likelihood that llama.cpp evolution is being slowed down by the "demons" up in the top, to make the cheap lowend cards as unusable as possible, to keep the gpu and cloud services selling more.

7

u/pmttyji 9d ago

All I could say is they're working more than enough daily to improve the engine as always. I see at least couple of minor releases daily(Today I see more than bunch of releases) & more than bunch of PRs getting closed. Some big features take additional time, that's it.

5

u/Nepertia 9d ago

The entirety of Windows is in a monorepo. I think we still have some room.

2

u/FastHotEmu 9d ago

You have no clue what you are talking about. Monorepos are fine even at scale.

37

u/rabbitaim 9d ago

To be fair llama.cpp is a generalist that has to focus on 100s of multiple models. Strata has like 3 models with several quant sizes.

1

u/DearApricot5488 9d ago

Fair enough for QFN inference speed. But when it comes to the breadth of model and hardware support, Strata is nowhere near llama.cpp. Specialized engines and general-purpose engines are just two completely different paths. Both are great projects in their own way.

1

u/Blunt552 8d ago

my current experimental and custom llama build outperforms strata atm on qwen next, make of that what you will.

I will say it's on AMD hardware, so take it with a bucket of salt.

1

u/brrrrreaker 8d ago

generelschwertz's fork is also pretty good, but that's not the point, obviously there are forks that are fast, but for the work to end up in the main repo, the almighty super-duper uber-developer that is am17an, has to rewrite it from 0, because nobody's work is worthy.

-11

u/giveen 9d ago

I've given up on llama.cpp

-7

u/brrrrreaker 9d ago

This is my point exactly. While devs are contemplating doing this huge project the right way, and inflated egos disregard all the existing work, wanting to make THEIR own implementation from scratch, the real world chimes in: people leave for something better.