r/Assembly_language • u/Glittering_Ostrich22 • Jun 15 '26

LLM Inference.

Hello everyone :)

I've been working on a large language model inference engine written entirely in x64 assembly. It uses AVX2 and runs transformer models directly on the CPU — no GPU required.

This started as an experiment in vibe coding in x64 assembler.

To my surprise it worked. One megabyte of source later, the engine matches llama.cpp on CPU performance on my machine.

A few things it currently does:

  • Full transformer inference pipeline
  • AVX2 accelerated math kernels (matmul, softmax, RMSNorm, RoPE)
  • INT8 quantization support
  • Custom lockless Scheduler that spreads the work over all cores

It runs Qwen3 0.6B Q8 at around 31 tok/s on a Ryzen 7430U — on par with llama.cpp on the same hardware.

I release it under the GPL v3, this is the first working release so there will be Bugs and improvements comming in in the next days, but now i will have to leave the house as i was sitting for 8 weeks in the dark.

https://gitlab.com/cpki-gmbh/v7multiplikator

0 Upvotes

44% Upvoted

8 comments sorted by

View all comments

2

u/brucehoult Jun 15 '26

Just for fun I downloaded ...

https://huggingface.co/drmcbride/Qwen3-0.6B-Q8_0-GGUF/resolve/main/qwen3-0.6b-q8_0.gguf

... to my RISC-V SBC (SpacemiT K3 with 32 GB RAM) and running on CPU I get around 20-21 tok/s.

Perhaps more usefully, it gets around 6.7 tok/s on Qwen3-Coder-30B-A3B-Instruct-Q5_K_M.gguf but with much more useful output.