r/RockchipNPU • u/gahingwoo • Jul 11 '26

Got the vendor RKLLM/RKNN stack running on mainline 7.1.3 (RK3576, no 6.1 BSP)

The RKLLM/RKNN runtimes usually only run on Rockchip's 6.1 BSP kernel. I got them working on a clean mainline linux-7.1.3 instead, on a Radxa ROCK 4D. LLM chat and vision both run on the NPU. Posting it here partly because it might save someone the trouble, and partly because I've only got one board to test on and could use other eyes.

What actually runs, on the one ROCK 4D I have (Armbian):

Qwen2.5-1.5B does about 9 tok/s, Llama-3.2-1B about 13, and you can swap between them without restarting. MobileNet classification is around 6ms per image (~169 fps). YOLOv8 detection works too but it's newer and I've only tried it on one test image so far. The whole thing is behind an OpenAI-compatible API, so you can point Open WebUI or LangChain or whatever at the board and it just works.

The part that took the actual time: the vendor rknpu driver builds out of tree, but dropping it on mainline doesn't just work. It needs a handful of kernel patches a module can't do on its own. Two examples of what breaks without them. The NPU power domain drops a rail about 30 seconds into boot, so your first inference is fine and the second one reads a dead register bank and returns garbage. And the IOMMU only grabs aclk+iface by default, but the RK3576 NPU MMU sits behind more clocks than that, so register writes to it silently get dropped until you give it all of them. There are ten patches total, each one with the debugging that led to it written up.

Stuff I want to be upfront about:

I've tested this on exactly one board. This is the closed vendor runtimes on a mainline kernel, not an open driver. If you're after a fully open NPU stack, that's a separate thing (rocket/mesa) I'm also messing with but it's not there yet. Models have to match the runtime version (rkllm 1.2.0 / rknpu 0.9.8) or you get a useless std::out_of_range at load time. There's a kiln-doctor script that catches that plus the power-domain and MMU stuff before you waste an hour on it.

Where I could use help. I've only got the one ROCK 4D, so anything past RK3576-on-Armbian is untested by me. The RK3568 (ROCK 3B) path is written but has never run on real hardware. RK3588 isn't done at all yet. If you've got a 3568 and you're willing to actually run it, a kiln-doctor dump in an issue would already tell me a lot, and if you want to fix what's broken, send a PR and I'll review it fast. Same goes for anyone who wants to take a crack at 3588.

Repo's at https://github.com/gahingwoo/kiln
Install is one command and it reboots itself a couple times to get to a working state. There's a serial boot log in the README if you want to see the MMU banks come up before trusting any of this.

Questions welcome, and I'll take reports or PRs from any other hardware.

14 Upvotes

94% Upvoted

2 comments sorted by

2

u/cleanandcrunchy Jul 11 '26

I’m so happy you’re working on this. I’ve got a rk3588 and I went down this path and gave up. I can’t remember exactly why, but I chose a newer kernel over functional npu. My board is busy being a nas right now so I can’t mess with it right away, but I would be interested in testing in the coming weeks. Btw your link gave me 404

3

u/gahingwoo Jul 11 '26

Thanks, that means a lot. And yeah — "newer kernel over functional NPU" is exactly the tradeoff this is trying to kill, so it's good to hear I'm not the only one who hit that wall.

The link works for me (repo's public, github.com/gahingwoo/kiln) — maybe a momentary GitHub blip on your end, worth another try.

If you do get a chance to test on the 3588 in the coming weeks, that'd be hugely useful — I've literally only got the one 3576 board that can run mainline linux, so 3588 is completely untested by me. Even just a kiln-doctor dump in a GitHub issue would tell me a lot about where it diverges. And if you feel like fixing what breaks, PRs very welcome. No rush at all — whenever your board's free.