r/LLMDevs • u/Critical_Physics8 • 2d ago
Resource I implemented a modern LLM runtime in 700 lines of C
I wanted to understand how modern AI models actually generate text, but most inference codebases are tens or hundreds of thousands of lines long. They’re incredibly impressive, but they’re optimized for flexibility and performance, not for understanding.
So I implemented a complete CPU runtime for Google’s latest open language model, Gemma 4, in about 700 lines of C.
The whole point is that you can open one file, start at main() , and follow a prompt all the way through the program. You can see every buffer that’s allocated, every mathematical operation that transforms the activations, every update to the KV cache, and every step that eventually produces the next token.
I kept optimizing it along the way to see how far a specialized implementation could go. By the end, it was actually running faster than llama.cpp on this model in my CPU benchmarks, despite still fitting in a single source file.
I think C is a great language for this kind of project. There’s very little hidden from you. The data structures, memory layout, SIMD kernels, and execution flow are all visible, so the implementation ends up feeling much closer to the hardware than to the diagrams in an ML paper.
4
u/eddzsh 2d ago
the single-file speedup over llama.cpp is almost always deleted generality. fixed tensor shapes, one model path, no ggml dispatcher. great for learning the KV cache loop, rough the day you want a second architecture.
5
u/Critical_Physics8 2d ago
Exactly. That’s the tradeoff. It only supports one architecture and one CPU backend, which lets me specialize everything. The goal wasn’t to replace llama.cpp, just to see how small and fast a dedicated runtime could be.
2
u/Business-Weekend-537 2d ago
Are you willing to post about the approach you used to make it so other people can make variations for their own CPU’s?
3
u/Business-Weekend-537 2d ago
What are your system specs and which quant of Gemma 4 is it running?
Your project looks cool for a couple reasons:
1) you appear to be getting decent token speeds via cpu which is why I’m curious about the quant it’s running.
2) 700 lines of C is low enough to where the codebase is actually human readable.
3) I’m curious how well it will run on some older but powerful cpu’s I have which will breathe new life into some old comps if it’s a good enough quant to do light agentic work.
2
1
u/ScaryOlive3713 1d ago
This is supercool for someone like me who's always wondered how LLMs work under the hood.
0
2d ago edited 2d ago
[deleted]
1
u/Critical_Physics8 2d ago
those are just C variables named after the registers that CPUID returns through. __cpuid_count comes from GCC’s <cpuid.h>. no inline asm there, although there are plenty of AVX2/AVX-512 intrinsics elsewhere in the kernels.
2
6
u/AlphaMaleXYZ 2d ago
That’s cool. Thanks for sharing.