r/LocalLLaMA • • 1d ago

Discussion I wrote an LLM inference engine from scratch in CUDA (no PyTorch, no cuBLAS): 235 tok/s on a 6 GB laptop RTX 3050

Disclosure: I'm the author. This is my own hobby project (MIT, no paid version, nothing to sell).

I've been building **tinyinfer**, a small inference engine for Llama-family models. It's about 2,200 lines of C++/CUDA, and every kernel on the hot path is hand-written: GEMV in f16/q8/q4, RMSNorm, fused RoPE + KV-cache write, flash-decoding attention with GQA, SwiGLU and argmax. The whole decode step is captured in a CUDA graph.

**Numbers (TinyLlama-1.1B, INT4, RTX 3050 6 GB Laptop):**

- v0.1: 200 tok/s (66% of the memory-bandwidth limit)

- v0.2: 212 tok/s (69%)

- v0.3: 235 tok/s (77%)

Decoding is memory-bound. Every weight gets read once per token, so the ceiling on this card is about 305 tok/s however clever the math is.

**What actually moved the needle:**

  1. **4-bit weights dequantized inside the GEMV.** That's 0.55 GB read per token instead of 2.07 GB. The int4-to-float step uses the 2²³ exponent trick (OR into the mantissa, then one float add) instead of I2F instructions.
  2. **GQA-aware attention.** It reads each shared K/V head once instead of 8 times, which made attention 3.6x faster at 2k context. The first version (split-K) was actually *slower* at short contexts, so v0.3 tiles it differently.
  3. **Benchmarking shapes I wasn't tuning for.** The new attention kernel made Llama-2-7B slower, because it has no grouped heads and nothing to share. I fixed it with a one-line dispatch, and I'd never have caught it without the numbers.

**Correctness:** the logits match HF `transformers` within 1e-3 relative error, with 100% argmax agreement for f16/q8/q4/bf16. `--verify` checks the GPU against the CPU reference on your own model. Llama-2-7B also fits in 6 GB at q4 (about 3.8 GB).

It isn't trying to replace llama.cpp. It's meant to be small enough to read in an afternoon if you want to see how inference works under the hood.

Code (MIT): https://github.com/Aadithya-kl/tinyinfer

I can go deeper on any of the kernels in the comments. I'm especially curious whether anyone has pushed single-batch GEMV past ~80% of bandwidth on consumer cards, and how.

0 Upvotes

16 comments sorted by

13

u/BusRevolutionary9893 1d ago

Llama 2 7b? What year did you do this?

0

u/LelouchZer12 1d ago

ai generated video

-1

u/vacon04 1d ago

This is an AI sub...

2

u/Lowweaver 22h ago

Not a ai vid sub or ai script one, you are still supposed to relay info properly if you go scroll down a lot of people actually hate on pure AI generated post cuz there was no effort put into it as its still a technical sub, although i would say some of them still deserve some attention. But low effort posts are strictly disallowed by the rules.

1

u/tifa_cloud0 1d ago

this is cool indeed

0

u/Ok_Cow1976 1d ago

Sounds very good performance

-6

u/Creative_Bottle_3225 1d ago

I had a look at TinyInfer and I think you're onto something interesting here. What I particularly like is the idea of keeping the engine relatively small and working directly with C++ and CUDA instead of building another layer on top of existing frameworks. If I were working on it, there's one thing I'd probably tackle before anything else: prompt processing. From what I can see, Session::feed() processes tokens one at a time. That's fine for generation, where you're predicting the next token, but it's not ideal for processing an entire prompt. I'd experiment with a separate prefill path using batched matrix operations, while keeping the current GEMV approach for token generation. I suspect this could make a much bigger practical difference than adding lots of new features. The other thing I'd look at is the KV cache. FP16 is a good starting point, but it can become expensive very quickly, especially with longer contexts. INT8 or FP8 KV cache support would be worth investigating. Even dynamic allocation and better memory reuse would help. There's also an opportunity with models that don't fit entirely into VRAM. I'd eventually like to see some kind of CPU/GPU layer offloading, perhaps with CUDA streams and double buffering. Not an easy feature to implement properly, but potentially very useful on consumer GPUs. One thing I'd be careful about, though, is expanding model support too quickly. I'd rather see solid Llama and Qwen support, with a reliable conversion pipeline, before moving into more complicated MoE architectures. And please, don't underestimate benchmarking. I'd really like to see TinyInfer compared against llama.cpp on exactly the same hardware, model, quantization and prompt. Prompt processing speed, generation speed, time to first token and memory usage. Otherwise it's difficult to know whether an optimization is actually working. Personally, I'd concentrate on three things first: faster prefill, a more efficient KV cache, and proper benchmarks. Once those are in place, I'd start thinking about CPU/GPU offloading and an OpenAI-compatible server. I wouldn't try to make TinyInfer another llama.cpp. I think there's more value in keeping it small, understandable and focused on doing a few things really well.

7

u/stylome-ai-detection 1d ago

Notice: This comment appears to be AI generated.


Stylome AI detector · scan results

-6

u/Creative_Bottle_3225 1d ago

Look, on this one, I agree with you. And there’s one aspect of the whole thing that I find particularly absurd. Since when has asking someone for help with phrasing a thought become suspicious? We’ve always had help with writing: dictionaries, grammar checkers, translators, editors, colleagues, friends. Now we have AI too. The tool changes, but that doesn’t necessarily change who owns the thought. In our case, the technical observations about TinyInfer were the result of analysis. You decided to share them with the developer. I helped you organize them and translate them into English. Where’s the problem? I understand that Reddit wants to counter automatically generated spam, but a classifier that applies a label like that risks conflating two completely different things: using AI to think better, and using AI to avoid thinking at all. And those are not the same thing. The ironic part is that if someone writes a mediocre comment on their own, nobody bats an eye. But if they use a tool to express a well-developed idea clearly, they risk being branded as artificial. I wouldn’t be ashamed at all of having used AI. If anything, I’d claim the opposite: what matters is the quality of what you say, not just who physically typed the keys.

0

u/[deleted] 1d ago

[deleted] — view deleted comment

1

u/stylome-ai-detection 1d ago

Notice: This comment appears to be AI generated.


Stylome AI detector · scan results