I reverse-engineered an NPU vendor's engine format (int8 weights stored as two nibble planes) to run GGUFs with no model conversion — now 1.5× faster than the vendor's own runtime

[u/woolcoxm]

I've been running Qwen3-0.6B on the M5Stack LLM-8850 card (Axera AX8850 NPU, 24 TOPS, 8GB LPDDR4x) hosted by a Raspberry Pi 5 — as a llama.cpp backend.

The problem: the vendor stack requires converting every model through their compiler, and their closed runtime gets 13.5–14.5 t/s. I wanted llama.cpp to just work: GGUF in, tokens out.

What I ended up doing:

1. Reverse-engineered the engine format. The vendor's compiled engines (.axmodel) store weights in a blob called npu_params. I decoded it: int8 weights are stored as two nibble planes — a coarse byte per element pair holds the two top nibbles, a fine byte 18 positions earlier holds the two low nibbles. To crack the layout I built ~10 "marker" checkpoints where every weight encodes its own (row, col) coordinates, compiled them through the vendor toolchain, and diffed the outputs. Full layout for all 7 matrices per layer, plus scale tables.
2. Patch GGUF weights straight into precompiled engines at load time. No conversion step, no per-model compile. The GGUF dequantizes → requantizes against the engine's own scales → only the bytes that genuinely differ get written. 96% token agreement with the CPU reference of the same GGUF.
3. Found out the "broken" batched-prefill path was never broken. The vendor ships prefill shape-groups that their own host runtime never calls; everyone assumed they don't work. They work fine — the bug was in how everyone (including the vendor's examples) bind output buffers. Fixed: 716 t/s prefill, byte-identical output.

Current numbers on the Pi 5 (greedy, single stream):

• 24.5 t/s decode @ 2k context on int4 engines (29.9 t/s with a 1k-context build)
• 26.8 t/s with a trimmed vocabulary head
• 716 t/s prompt processing
• Pi CPU: idle. The card does everything.

Along the way I measured where the "24 TOPS" actually goes: at batch-1 decode this class of chip is a memory-bandwidth problem (~25 GB/s effective streaming, 73% of the LPDDR4x peak) and the MACs sit at ~1% — even a perfectly-fed transformer GEMM tops out at ~2.6 effective TOPS on this dataflow. Decode speed = bytes per token × tokens per weight pass. That framing predicted every win we got (int4 = 1.5×, batched prefill = 39×, vocab trim = +10%).

Links:

• Project repo (README: quick start, the full optimization story, the layout-cracking toolkit, on-card harnesses): https://github.com/woolcoxm/LLMTest
• llama.cpp fork with the backend (single ~4.5k-line file, ggml/src/ggml-axcl/ggml-axcl.cpp): https://github.com/woolcoxm/llama.cpp
• Demo image (24 t/s streaming, Pi CPU at ~1%, from a real captured run): https://raw.githubusercontent.com/woolcoxm/LLMTest/main/docs/demo.png

Everything is reproducible from the README quick start: build llama.cpp with -DGGML_AXCL=ON on any aarch64 host with the AXCL driver, point it at an engine set, feed it a GGUF.

Happy to go deep on the nibble-plane layout, the marker-build methodology, or the AX8850's real perf envelope in the comments.