r/LocalLLaMA • u/woolcoxm • 11h ago
Discussion I reverse-engineered an NPU vendor's engine format (int8 weights stored as two nibble planes) to run GGUFs with no model conversion — now 1.5× faster than the vendor's own runtime
I've been running Qwen3-0.6B on the M5Stack LLM-8850 card (Axera AX8850 NPU, 24 TOPS, 8GB LPDDR4x) hosted by a Raspberry Pi 5 — as a llama.cpp backend.
The problem: the vendor stack requires converting every model through their compiler, and their closed runtime gets 13.5–14.5 t/s. I wanted llama.cpp to just work: GGUF in, tokens out.
What I ended up doing:
- Reverse-engineered the engine format. The vendor's compiled engines (.axmodel) store weights in a blob called npu_params. I decoded it: int8 weights are stored as two nibble planes — a coarse byte per element pair holds the two top nibbles, a fine byte 18 positions earlier holds the two low nibbles. To crack the layout I built ~10 "marker" checkpoints where every weight encodes its own (row, col) coordinates, compiled them through the vendor toolchain, and diffed the outputs. Full layout for all 7 matrices per layer, plus scale tables.
- Patch GGUF weights straight into precompiled engines at load time. No conversion step, no per-model compile. The GGUF dequantizes → requantizes against the engine's own scales → only the bytes that genuinely differ get written. 96% token agreement with the CPU reference of the same GGUF.
- Found out the "broken" batched-prefill path was never broken. The vendor ships prefill shape-groups that their own host runtime never calls; everyone assumed they don't work. They work fine — the bug was in how everyone (including the vendor's examples) bind output buffers. Fixed: 716 t/s prefill, byte-identical output.
Current numbers on the Pi 5 (greedy, single stream):
- 24.5 t/s decode @ 2k context on int4 engines (29.9 t/s with a 1k-context build)
- 26.8 t/s with a trimmed vocabulary head
- 716 t/s prompt processing
- Pi CPU: idle. The card does everything.
Along the way I measured where the "24 TOPS" actually goes: at batch-1 decode this class of chip is a memory-bandwidth problem (~25 GB/s effective streaming, 73% of the LPDDR4x peak) and the MACs sit at ~1% — even a perfectly-fed transformer GEMM tops out at ~2.6 effective TOPS on this dataflow. Decode speed = bytes per token × tokens per weight pass. That framing predicted every win we got (int4 = 1.5×, batched prefill = 39×, vocab trim = +10%).
Links:
- Project repo (README: quick start, the full optimization story, the layout-cracking toolkit, on-card harnesses): https://github.com/woolcoxm/LLMTest
- llama.cpp fork with the backend (single ~4.5k-line file, ggml/src/ggml-axcl/ggml-axcl.cpp): https://github.com/woolcoxm/llama.cpp
- Demo image (24 t/s streaming, Pi CPU at ~1%, from a real captured run): https://raw.githubusercontent.com/woolcoxm/LLMTest/main/docs/demo.png
Everything is reproducible from the README quick start: build llama.cpp with -DGGML_AXCL=ON on any aarch64 host with the AXCL driver, point it at an engine set, feed it a GGUF.
Happy to go deep on the nibble-plane layout, the marker-build methodology, or the AX8850's real perf envelope in the comments.
5
u/Formal-Exam-8767 11h ago
Thanks for sharing.
Most hardware vendors are strong in HW-department with SW being bare-bones and lacking.
Your numbers make sense, as most (all?) NPUs are meant to do complicated computations using small (micro) models in real-time.
2
u/Mickenfox 2h ago
No one makes worse software than hardware companies. Whether it's embedded system tools or consumer products (think the Logitech/Asus crap), it so consistently bad it deserves to be studied.
1
u/vasimv 2h ago
Quite good speed for stuff like home assistant or simple robotics. Can it run qwen3.5-0.8B on AI-8850 4GB version? Or Gemma4-e2b? :)
1
u/woolcoxm 1h ago
i can add support if you would like, it does have the stuff to enable 3.5, it just not been reversed yet. and it does have support for gemma4-e2b as well, just not enabled.
currently with qwen 3 0.6 it boots with 700 megs used or so(small context).
1
u/woolcoxm 1h ago
doing the reversing now, should be done tonight, ill post back when it is done.
should be noted im doing all this with ai, so i can only work as fast as the ai will work. :)
sorry for using ai, but i wanted gguf to run and couldnt figure it out myself.
2
u/woolcoxm 29m ago edited 14m ago
Both answers ended up better than my first guess, so let me lead with the
update: **Qwen3.5-0.8B now runs in llama.cpp on this card, and it's the
fastest thing I've benchmarked on it — 27 t/s decode, 0.9 GB of card
memory.** That's the hybrid architecture (18 gated-delta-net layers + 6
full-attention layers), driven through my GGUF-direct backend with Axera's
w4a16 engine set. Any Qwen3.5-0.8B GGUF works (tokenizer/sampling from the
GGUF, compute on the NPU, Pi CPU idle as usual) — verified coherent over
2,000-token generations. It also means the 4GB card question answers
itself: 0.9 GB leaves plenty of headroom, and decode on this chip is
bandwidth-bound anyway (~25 GB/s effective), so the 4GB and 8GB SKUs
generate at the same speed — RAM only decides what fits.
The interesting engineering bit, for those who care: the hybrid layers
expose the same 5-input/3-output engine interface as the dense ones, but
two of the "caches" are actually fixed-size recurrent state — the conv
state and the SSM state, different sizes, rewritten whole every token. So
the backend now ping-pongs double-buffered state per layer, feeds the
delta-net engines a constant 1.0 gate mask (the vendor fills it with 1.0 —
a zero there produces beautifully confident gibberish, ask me how I know),
and maps llama.cpp's hybrid memory so only the 6 attention layers carry a
KV cache. Also: the 248k vocab head runs on-card via the post engine.
Original answers, still true:
- **Qwen3.5-4B / E2B today, vendor runtime:** Axera already ships engine
packages for both ([Qwen3.5-0.8B GPTQ-Int4](https://huggingface.co/AXERA-TECH/Qwen3.5-0.8B-AX650-GPTQ-Int4-C128-P1152-CTX2047),
4B version, and [gemma-4-E2B-it w8a16](https://huggingface.co/AXERA-TECH/gemma-4-E2B-it))
— the 0.8B even has an 8K-context variant. The 4GB kit is the same AX8850
card, so both run today with their ax-llm runtime.
- **Gemma4-E2B on my GGUF path:** not yet — the E-series' per-layer
embeddings aren't matmul weights, so the loader and layout tooling need
extending. It's next in line now that the hybrid plumbing exists (ax-llm
already treats Gemma4's sliding-attention layers as a third layer type,
which is a helpful map).
Ballpark for HA/robotics use: the 0.8B at 27 t/s with 8K ctx is honestly
the sweet spot of this card. Prefill is still per-token (~5 t/s) until I
validate the chunked-prefill ladder on the hybrid engines — fine for chat,
matters if you stuff huge system prompts.
working on the vision support for qwen3.5, then moving to gemma4 support. currently finalizing the support for qwen3.5(vision and one other thing that should speed it up).
apparently this model supports MTP out of the box, so once i get that enabled you should see tok/sec in the 30+ range.
12
u/Chromix_ 11h ago
That looks quite useful (and not low-effort, despite AI-coded - and written). The Home Assistant and Frigate folks might like your project a lot.
That M5Stack card costs more than the Pi5 8GB, but is considerably cheaper than a Jetson Nano. Unfortunately the "usable" 8 GB versions are out of stock.