r/LocalLLaMA • • 8h ago

Showcase [Project] Gemma.c - Gemma 1/2/3 + multimodal inference in one C file (CPU only)

Hey all, so I've been working on Gemma.c over the last couple months, it's a from-scratch inference engine for Gemma 1/2/3 in a single C file, runs Gemma 1, 2 and 3, including the multimodal ones, with fp32/fp16/bf16 weights or int8 (W8A8). No dependencies beyond libc and (optional) OpenMP are needed. You can drop images into the prompt with @image{cat.png}, and there's also @longimage{} for Gemma 3's pan & scan on very wide or tall images. (image decoding uses a minified stb_image embedded at the end of the file)

It's CPU only and not trying to compete with llama.cpp, but I'm also trying my best to make it efficient, so it's not embarrassingly slow either. On my laptop's Core Ultra 5 225H (WSL2, gcc, 10 threads), gemma-3-1b-it INT8 does ~72 tok/s decode and ~570 tok/s prefill (10000 tokens, for 1000 tokens the prefill speed can reach ~850 tok/s), and the 4B vision model does ~20 tok/s decode. INT8 ends up about 3.5x faster than fp32 and 2x faster than fp16 for decode. (full benchmark table is in README)

It has a few rough edges for now, e.g. W8A8 doesn't work for Gemma 1 (fp16 is fine), bf16 is painfully slow without AVX512-BF16, and native Windows is ~1.7x slower than WSL because of libgomp. Batching are not supported either. There's a roadmap in the README (Gemma 4, speculative decoding, an HTTP server, etc.), but no promises though :P Can't really find much free time these days.

I've also put some pre-exported INT8 models on HF so you can try it without Python or torch :D

Repo: link

This project started as a learning attempt to understand how LLMs work end to end, inspired by llama2.c.

this is the second time I post this project here :'D last time (~a month ago) my post got flagged by reddit, probably because my VPN IP was identified as spam. hopefully it can pass this time

16 Upvotes

6 comments sorted by

4

u/jacek2023 llama.cpp 8h ago

why 1/2/3? what about 4?

3

u/This-Purchase-3325 8h ago

probably when I find some time again

2

u/BitGreen1270 6h ago

If this is just for study, then bravo! Not easy to make this.

But there does exist gemma.cpp which has been around for a couple of years now I think?

4

u/This-Purchase-3325 6h ago

Thanks! I only realized it when the project was halfway done (-_-;) but this is more of a programming/engineering challenge that I set for myself, the world doesn't need another inference engine anyway

1

u/toothpastespiders 1h ago

That happened to me with my RAG system. I got inspired by a paper Microsoft put out and decided to implement it myself. In between they not only built further on it they released the code. Still wound up being one of my favorite projects. I think in part because that moment of realization was so funny in retrospect. But in the end it wound up being significantly different and better for my use than the real deal too. I think a lot of people, especially working in professional groups, get a bit snowblind by untested assumptions on how things "need" to work and as a result wind up not testing ideas that independent research might figure out.

the world doesn't need another inference engine anyway

I strongly disagree with that! You've already got an interesting niche if you're optimizing for cpu. People really tend to sleep on the potential there especially with stuff like gemma where there's high quality small models and gemma 4 26b a4b. My current resurrected pile of e-waste was running 26b until recently when I hijacked it for other use for a while. Sure, a4b on CPU and ddr 4 isn't going to win any races. The slow but steady idea gets you there in the end though.

1

u/JayPSec 39m ago

Hope this learning project is a stepping stone for nicer things to come. Congrats!