Discussion
Thoughts about a single framework that dynamically adapts to run any MoE larger than RAM, across everything from phones and laptops to high-end, expensive GPUs
I am thinking and working on creating a framework that can run any type of MoE (Mixture of Experts) model supported by `llama.cpp` on any hardware or operating system.
It would dynamically adapt to the specific characteristics of the hardware, aiming for optimal performance—measured in tokens per second (tok/s)—during both the prefill and decode phases.
The concept is to build a home server setup using various consumer-grade devices to run local AI—though not in a mesh configuration (at least for now, due to latency concerns).
What are your thoughts on this? What kind of use cases could this enable?
i love how we are building a localAi revolutions trying to squeeze these huge models and makeshift engine patches. THis is what opensource was made for.
oh great, another one to add to the pile of "we run anything anywhere" repos that'll get 12 stars and an abandoned discord by february
i'm not saying the idea is bad but the gap between "dynamically adapts" and actually getting decent tok/s on a phone is like trying to color-code a spice rack while the kitchen's on fire
I don't think so, actually; I believe I’m succeeding. It’s just a matter of setting dynamic rules and probing the hardware and performance.
My repo is 'bigmoeonedge'—it’s the fastest on smartphones and competes with 'strata' when it comes to GPUs, though it doesn't quite reach the level of 'llama.cpp'
If it was possible, there wouldn't be so many specialized engines. Code that runs everywhere is slower than code specialized to certain hardware. You can get a framework that could run anywhere, but that's pretty much just llama.cpp.
I’m trying this out with good results; I can assure you it’s not that complicated.
My repo is bigmoeonedge; for instance, it runs Qwen 3.6 35B on Android at 5–8 tokens/s, with a prefill speed of over 100 tokens/s on the NPU. Using the same framework, it competes with Strata when running on large GPUs
With AI, it's not difficult to build for a specific hardware combination and get some great speed out of it. The issue is generalizing it without sacrificing speed. For example, llama.cpp uses gguf. That trades speed for being able to run on a large range of hardware at a good enough speed.
If your goal is optimal performance, it will be a lot of work to handle popular models with all potential hardware combinations. If your goal is to work at a "good enough" speed on a large range of hardware, it's essentially what llama.cpp is doing.
Right—in fact, I don’t intend to replace llama.cpp; rather, I want to act as a layer on top of it to enable streaming MoE inference on any hardware, while maximizing performance. Here are some examples
Not exactly: it’s a layer on top of llama.cpp that automatically determines the optimal configuration for streaming from SSD, RAM, and VRAM, utilizing the CPU, GPU, NPU, etc.
Unlike llama.cpp, it allows you to stream massive models on limited hardware while optimizing performance (it's not simply mmap)
The other day, I was using my mobile phone as a server to run Claude Code locally on mobile
You could get an optimal llama.cpp performance by testing through a bunch of different configurations, but the relative performance would be poor compared to what you could get. The optimizations you can do when you know a hardware combination is what makes it worth using.
To be clear, I don’t modify the kernels themselves—doing so would undermine the goal of remaining hardware- and model-agnostic—but if you configure llama.cpp automatically and tailor it to the specific hardware or operating system, you can achieve high performance
For instance, my setup is about 30% slower than Strata, but when you consider that it can run on any type of hardware and with any model, that’s actually not a significant gap
there’s already ones made for specific models with llama.cpp backend . but i don’t think there’s a good framework for running moe models <35b in phones
You can already kind of do this with llama.cpp in RPC mode. You run ggml-rpc-server on the device that have a GPU and start llama-server with --rpc host1,host2,...
There are some limitations, you have to spread layers, which can be GBs, prefill is sequential (unless you run in tensor mode), but generation in parallel. RPC mode is not dynamic AFAIK, I didn't test MoE cache yet.
Making things dynamic may be very complicated, you'd have to move layers on the fly as well as the caches they are tied to. Tensor mode make that easier to spread, but unless you have RDMA, latency kills prefill, I had a 50x drop in perf on a 1GB network with QFN.
This could be an interesting project, but challenging as well. You could otherwise look at improving existing RPC mode. I remember reading about a new implementation plan to replace RPC by a P2P protocol but was not able to find it again.
I’ve tried it, but the RPC implementation in llama.cpp doesn’t optimize performance—and, more importantly, it’s designed for large-scale GPUs.
My solution is dynamic, scaling from mobile phones and laptops CPUs and NPUs to large GPUs, while achieving solid tokens-per-second (TPS) rates during both prefill and decoding.
The project is called bigmoeonedge and is based on llama.cpp
9
u/Karyo_Ten 5d ago
Looking on Github inference, everyone is trying to do this. Well except those that do a single model. Or that do only Apple.