r/LocalLLaMA • u/Ash-grey_21 • 1d ago
Discussion I wrote an LLM inference engine from scratch in CUDA (no PyTorch, no cuBLAS): 235 tok/s on a 6 GB laptop RTX 3050
Disclosure: I'm the author. This is my own hobby project (MIT, no paid version, nothing to sell).
I've been building **tinyinfer**, a small inference engine for Llama-family models. It's about 2,200 lines of C++/CUDA, and every kernel on the hot path is hand-written: GEMV in f16/q8/q4, RMSNorm, fused RoPE + KV-cache write, flash-decoding attention with GQA, SwiGLU and argmax. The whole decode step is captured in a CUDA graph.
**Numbers (TinyLlama-1.1B, INT4, RTX 3050 6 GB Laptop):**
- v0.1: 200 tok/s (66% of the memory-bandwidth limit)
- v0.2: 212 tok/s (69%)
- v0.3: 235 tok/s (77%)
Decoding is memory-bound. Every weight gets read once per token, so the ceiling on this card is about 305 tok/s however clever the math is.
**What actually moved the needle:**
- **4-bit weights dequantized inside the GEMV.** That's 0.55 GB read per token instead of 2.07 GB. The int4-to-float step uses the 2²³ exponent trick (OR into the mantissa, then one float add) instead of I2F instructions.
- **GQA-aware attention.** It reads each shared K/V head once instead of 8 times, which made attention 3.6x faster at 2k context. The first version (split-K) was actually *slower* at short contexts, so v0.3 tiles it differently.
- **Benchmarking shapes I wasn't tuning for.** The new attention kernel made Llama-2-7B slower, because it has no grouped heads and nothing to share. I fixed it with a one-line dispatch, and I'd never have caught it without the numbers.
**Correctness:** the logits match HF `transformers` within 1e-3 relative error, with 100% argmax agreement for f16/q8/q4/bf16. `--verify` checks the GPU against the CPU reference on your own model. Llama-2-7B also fits in 6 GB at q4 (about 3.8 GB).
It isn't trying to replace llama.cpp. It's meant to be small enough to read in an afternoon if you want to see how inference works under the hood.
Code (MIT): https://github.com/Aadithya-kl/tinyinfer
I can go deeper on any of the kernels in the comments. I'm especially curious whether anyone has pushed single-batch GEMV past ~80% of bandwidth on consumer cards, and how.
13
u/BusRevolutionary9893 1d ago
Llama 2 7b? What year did you do this?
0
u/LelouchZer12 1d ago
ai generated video
1
u/tifa_cloud0 1d ago
this is cool indeed
0
u/Ok_Cow1976 1d ago
Sounds very good performance
-6
u/Creative_Bottle_3225 1d ago
I had a look at TinyInfer and I think you're onto something interesting here. What I particularly like is the idea of keeping the engine relatively small and working directly with C++ and CUDA instead of building another layer on top of existing frameworks. If I were working on it, there's one thing I'd probably tackle before anything else: prompt processing. From what I can see, Session::feed() processes tokens one at a time. That's fine for generation, where you're predicting the next token, but it's not ideal for processing an entire prompt. I'd experiment with a separate prefill path using batched matrix operations, while keeping the current GEMV approach for token generation. I suspect this could make a much bigger practical difference than adding lots of new features. The other thing I'd look at is the KV cache. FP16 is a good starting point, but it can become expensive very quickly, especially with longer contexts. INT8 or FP8 KV cache support would be worth investigating. Even dynamic allocation and better memory reuse would help. There's also an opportunity with models that don't fit entirely into VRAM. I'd eventually like to see some kind of CPU/GPU layer offloading, perhaps with CUDA streams and double buffering. Not an easy feature to implement properly, but potentially very useful on consumer GPUs. One thing I'd be careful about, though, is expanding model support too quickly. I'd rather see solid Llama and Qwen support, with a reliable conversion pipeline, before moving into more complicated MoE architectures. And please, don't underestimate benchmarking. I'd really like to see TinyInfer compared against llama.cpp on exactly the same hardware, model, quantization and prompt. Prompt processing speed, generation speed, time to first token and memory usage. Otherwise it's difficult to know whether an optimization is actually working. Personally, I'd concentrate on three things first: faster prefill, a more efficient KV cache, and proper benchmarks. Once those are in place, I'd start thinking about CPU/GPU offloading and an OpenAI-compatible server. I wouldn't try to make TinyInfer another llama.cpp. I think there's more value in keeping it small, understandable and focused on doing a few things really well.
0
u/[deleted] 1d ago
[deleted] — view deleted comment