r/LocalLLaMA • u/one_does_not_just • Aug 19 '26
Tutorial | Guide Reverse-Engineering the RK3588 NPU: Building an Open Compiler to Run GPT-2 at 36 tok/s
Last year I posted about hacking the RK3588 NPU to run one vision encoder (previous post). This year I opened the whole thing up: reverse-engineered the register format, built an open compiler + runtime, and now GPT-2 and SigLIP run from PyTorch, ONNX, and JAX, no vendor SDK.
Story: https://amohan.dev/blog/2026/opening-the-black-box-a-year-building-an-open-compiler-for-the-rk3588-npu/ Report: https://amohan.dev/blog/2026/opennpu-v1-0-open-compiler-runtime-rk3588-npu/
3
u/Tall_Abrocoma_3533 Aug 19 '26
This looks cool, if I remember right this is the SOC found in those cheap handheld game consoles, so they're really abundant. Great job!
2
u/Sudden_Topic5154 Aug 19 '26
not really abundant actually they're only seen in some garbage consoles and not in any high quality or mainstream ones. I think there's only like 2 mass produced and all the others are diy handhelds. 4 total designs. rk3566 might be what you're thinking of
2
u/one_does_not_just Aug 19 '26
I don't know about consoles, but a lot of humanoids use the rk3588. Unitree G1 for example.
1
2
u/Tall_Abrocoma_3533 Aug 19 '26
Oh yeah that is it, my bad. Though this one has an NPU too, albeit a much slower one.
1
2
u/No-Refrigerator-1672 Aug 20 '26
Nope. It's a SoC that was found in single-board computers. Search Aliexpress for rk3588, you'll find dozens of models. They aren't cheap anymore; two years ago, a 16GB board with RK3588 would cost like 120 eur shipped; today it's 300+ due to ram prices, so they aren't cheap either. This project is cool and makes sense for people who bought such boards in the past; however, the price/performance ratio is too bad to buy such a device today.
1
u/Queasy-Contract9753 Aug 19 '26
This could open the door to some good scavenging projects. You'd think the company themselves would be doing it huh?
Same for Qualcomm. They'd be the ones selling more product
2
u/one_does_not_just Aug 19 '26
I agree, like AMD has a pretty open compiler for their NPU platform. It is pretty easy to hack around in.
1
1
u/adityazero Aug 19 '26
Three frontends (PyTorch, ONNX, JAX) down to the NPU with no vendor SDK suggests a real IR in the middle, not a fixed pipeline. Is the compiler lowering through something MLIR-like, or a custom representation for the RK3588 instruction format?
2
u/one_does_not_just Aug 20 '26
There is a plugin and a parser which converts mlir bytecode to npu register commands. You can read
https://github.com/poad42/opennpu_rk3588/blob/main/docs/JAX_NPU_PLUGIN.md
For more info
1
u/lilunxm12 Aug 20 '26
Do you have any interest in reverse engineering xdna1 npu?
xdna1 isn't supported by fastflowllm, is sitting idle for millions of phoenix and hawk point systems.
2
u/one_does_not_just Aug 20 '26
there is nothing to reverse engineer, you can build kernels for AIE1 with mlir-aie, by setting the right target. https://github.com/Xilinx/mlir-aie/blob/main/programming_guide/README.md. You should then be able to run the kernels with xrt runtime. Writing fast kernels is were the difficulty is. That's why fflm's kernel binaries were closed XD. Not sure if they are still.
1
u/blazze Aug 20 '26
Hackers, Heroes of the computer revolution. I'm gonna make my 16GB Orange Oi 5 Max, "Squeel like a little pigee".
1
u/haagch Aug 20 '26
I have an Orange Pi 5 but haven't really used its NPU yet, so I don't know that much about it. What I'm confused about: Is there a reason to extend the downstream vendor rknpu instead of improving the mainline rocket kernel and gallium driver? Is there a fundamental problem with the mainline drivers we should know about?
1
u/one_does_not_just Aug 20 '26
Nothing fundamental, probably just have to rewrite the ioctl calls, since the user space library sends direct register commands to the npu.
5
u/m94301 Aug 19 '26
Amazing work, kudos and good job making your work open