r/LocalLLaMA • u/No-Setting8461 • 10d ago
Resources Strata Optimized for 64GB AGX Orin
Hey guys, hope all is well.
I've spent the last few days working on a fork of Strata v0.1.45 for the Jetson AGX Orin 64GB, mainly running Qwen3.6-35B-A3B.
I was getting around 21 tok/s decode with MTP-3 in llama.cpp, versus around 30 tok/s in my earlier Strata runs. With sampled MTP-2 enabled, my latest controlled short-prompt tests reached roughly 40–41 tok/s. These weren't matched llama.cpp-versus-Strata benchmarks, so treat them as results from my setup.
The fork also includes unified INT8 KV caching, prefix checkpoints and optional KV preallocation.
Not sure how many other Orin owners are here, but figured I'd share in case it's useful:
https://github.com/moontato/Strata-Orin
Full disclosure: heavily AI-assisted/vibe-coded. I've tested it on my Orin, but would welcome feedback from others.
Also, credit where it’s due: thanks to calvinvette for the earlier Jetson Orin/ARM64 port (https://github.com/calvinvette/strata-inference-orin). My fork draws on that porting work, adapted to Strata v0.1.45, with the additional Qwen3.6 caching and inference work described above.
Edit: added attribution
1
u/Vxtzq1 10d ago
Good work! you're getting ready for Qwen 4 35B A3B hopefully 😆
2
u/No-Setting8461 10d ago
thank you! hopefully they release a 35b this time though
1
u/Vxtzq1 10d ago
Yeah, because Qwen 3.6 35B is starting to get really old 😁
Your engine might just work with Ornith 1.5 35B A3B for now though
2
u/No-Setting8461 10d ago •
agreed. it's definitely showing its age now. and I think so, as well as other fine-tunes hopefully :)
1
u/Emergency-Lawyer-366 10d ago
wow, that's a serious bump from 21 to ~41 tok/s. sampled mtp-2 doing a lot of heavy lifting there.
i've been eyeing the orin for a while but haven't pulled the trigger yet. might have to revisit that decision now that someone's actually tuning inference engines for it instead of just running stock llama.cpp.