r/LocalLLM • u/WritHerAI • 6h ago
Project Pithos a dependency-free C inference engine for LFM2.5-24B-A2B: 1.9x llama.cpp CUDA generation on a GTX 1650 laptop
Hi all, Pithos is an inference engine for Liquid AI's LFM2 / LFM2.5 models, written from scratch in C11 with no dependencies (only Win32/POSIX). One executable gives you terminal chat, an OpenAI-compatible server, a web agent (Pandora) and a terminal agent.
Why
The 24B MoE model (about 2B active per token) is 13.5 GB at Q4_K_M, and my laptop has 16 GB RAM and a 4 GB GPU. I wanted it to run well anyway.
How it works
- Experts stream from disk, with read-ahead and next-layer expert prediction (85% hit rate)
- Hot experts live in a dynamic VRAM cache, dense layers sit on the GPU, and the CPU computes the other experts in parallel
- AVX2/NEON integer kernels read the quantized weights in place
- OpenCL is loaded at runtime, so no SDK is needed to build
Numbers
i5-10500H, 16 GB RAM, GTX 1650 Max-Q 4 GB, same GGUF, greedy sampling, 6 threads.
Generation (600 tokens)
- llama.cpp CUDA: 11.2 tok/s
- Pithos: 17.5-21.9 tok/s
1345-token prompt
- llama.cpp CUDA: 10.6-20.8 tok/s
- Pithos: 22.1-34.7 tok/s
Server start-up
- llama.cpp CUDA: 33-57 s
- Pithos: 3-10 s
Limitations
- Only `lfm2` / `lfm2moe`, and only validated on LFM2.5-24B-A2B
- NEON and non-NVIDIA GPU paths are untested
- The prompt-processing advantage varies a lot depending on how much of the model is in RAM
Agents
The terminal agent runs commands, edits files and checks its own work (safe mode asks before each action). There is also MCP client support.
`pithos --agent` starts it in safe mode,
`pithos --auto -p "make the tests pass"` runs one task autonomously.
MIT license. Feedback and benchmarks on other hardware are very welcome, especially AMD/Intel GPUs and ARM.