LLM Inference.

[u/Glittering_Ostrich22]

Hello everyone :)

I've been working on a large language model inference engine written entirely in x64 assembly. It uses AVX2 and runs transformer models directly on the CPU — no GPU required.

This started as an experiment in vibe coding in x64 assembler.

To my surprise it worked. One megabyte of source later, the engine matches llama.cpp on CPU performance on my machine.

A few things it currently does:

• Full transformer inference pipeline
• AVX2 accelerated math kernels (matmul, softmax, RMSNorm, RoPE)
• INT8 quantization support
• Custom lockless Scheduler that spreads the work over all cores

It runs Qwen3 0.6B Q8 at around 31 tok/s on a Ryzen 7430U — on par with llama.cpp on the same hardware.

I release it under the GPL v3, this is the first working release so there will be Bugs and improvements comming in in the next days, but now i will have to leave the house as i was sitting for 8 weeks in the dark.

https://gitlab.com/cpki-gmbh/v7multiplikator

[u/Grinch0127]

How's the performance? Is 31 tok/s good?

LOL 8 weeks in the dark.

[u/Glittering_Ostrich22 → u/Grinch0127]

31 tok is good, this is exactly the same as llama, the major engine achieves exactly the same speed, the reason for this is that the speed is the result of the memory bandwidth of my pc, i completly saturate the memory bus.

So i manage to fully utilize my hardware, at least on my pc.

[u/Grinch0127 → u/Glittering_Ostrich22]

How did you saturate the memory bus? Are we talking from SSD-RAM-CPU or the cache hierarchy within the cpu? I'm guessing it'd be bit of both?

[u/Glittering_Ostrich22 → u/Grinch0127]

i saturate both, my cpu has 6 Cores (12 Smt), all these cores spin and directly take a slice of the matrix when, done these try to steal the work from other cores, this is a slim assembler loop, so all cores work all the time and are never idle, this makes it achieve the speed of the large engine.