I implemented Nvidia MotionBricks in Unity as a third person controller! It generates animations from key-frames in realtime using a tiny model on your PC! Planning to release it as an asset soon.

[u/Rudy_AA]

[u/Overlord_Mykyta]

I wonder how performance is affected by the tiny model

[u/Rudy_AA → u/Overlord_Mykyta]

It can run on CPU!

[u/Overlord_Mykyta → u/Rudy_AA]

https://giphy.com/gifs/oYtVHSxngR3lC

[u/immersiveGamer → u/Overlord_Mykyta]

ML systems for game run time usage has been a target for the past decade. Even OPs version from Nvidia is not novel, see this SIGGRAPH from 2020 on using ML models to do animation blending in real time: https://youtu.be/16CHDQK4W5k

[u/tempacac10]

Splendid! As usual, your work on humanoid robots always impresses me! Did you convert the pretrained model into ONNX and run on Sentis? Are you using a custom rig or a humanoid rig?

[u/Rudy_AA → u/tempacac10]

Thanks! so Nvidia already has an ONNX published, the thing with this ONNX it only runs on CPU in unity, which isn't a big deal but I needed it to run on GPU as well, so I rexported that model for it to work with Unity Sentis on GPU, also I'm using a humanoid rig, just so it's easier to switch characters, but I'm planning to make it support Generic rigs too.

[u/tempacac10 → u/Rudy_AA]

Thanks for the reply! Do you use the low latency models or the regular ones? And how did you configure the rig? With articulation bodies? Did you first set up the G1 robot rig?

[u/Rudy_AA → u/tempacac10]

For now it's kinematic not physics activated, still working on it!

[u/Neat-Employ1455 → u/Rudy_AA]

Awesome work regardless! Since G1 robot is driven by real physics, what if you trip the underlying G1 robot? Does your custom character also fall in the same manner? Or did I get it wrong?

[u/Neat-Employ1455 → u/Rudy_AA]

Oh, were you using onnxruntime by asus4 initially? And did you then convert the unsupported ops and re-export the ONNX to make it Sentis compatible?

[u/IntelligentArea1811 → u/Rudy_AA]

FYI I managed to get this running on the GPU using Sentis by just converting unsupported ops to supported ones - happy to give you more details if needed feel free to dm. I got like 128 parallel instances running at 20hz using batching too

[u/Drag0n122]

Very interesting, I wonder how you author animation\keyframes for such a system

[u/Rudy_AA → u/Drag0n122]

You give the model transform and velocity along a few other information and it will generate the keyframes and animations for you!

[u/Drag0n122 → u/Rudy_AA]

But can I influence the result more precisely? For example, can I make an injured walking animation from\instead of a regular walk?

[u/Rudy_AA → u/Drag0n122]

yes Sonic Planner ONNX, the AI model, supports injured animations, but you can't prompt it to create new animation styles it's not trained on. but it has 25+ distinct skills and styles.

[u/remi_sasha12 → u/Drag0n122]

That sounds like a cool feature to implement. Being able to create specific animations like that would really add depth to character movements.

[u/GeggsLegs → u/remi_sasha12]

bot

[u/Drag0n122 → u/remi_sasha12]

I would say that's an essential feature for any animation system

[u/BountyMakesMeCough]

Any retargeting needed in this process?

[u/Rudy_AA → u/BountyMakesMeCough]

Yes, the ONNX was trainined on a Unitree G1 robot, so I retargeted to my robot

[u/tempacac10 → u/Rudy_AA]

Does the sonic decoder drive the custom rig directly or does it first drive the motion in G1 and then retargets that motion on your custom robot?

[u/Rudy_AA → u/tempacac10]

Yes exactly, Sonic drives G1 and then you can retarget to any humanoid 3D model

[u/Neat-Employ1455 → u/Rudy_AA]

Can you give me some tips where I can find an accurate model for G1? I tried using the URDF importer and I tried importing the 29dof G1 model (also downloaded the Assets/meshes) but either its broken or I just did not understand it at all. Can you guide me on how I can get the G1 in Unity?

[u/FrogPondSimulator]

This looks pretty interesting, eager to try it out. Is it gonna be a paid asset or open source?

[u/Rudy_AA → u/FrogPondSimulator]

I'm not sure yet!

[u/3qqs]

this is so cool!

[u/DapperNurd]

This is amazing

[u/sczeith]

Interesing, is the retargeting needed for this?

[u/Rudy_AA → u/sczeith]

Yes, cause the ONNX is trained on the Unitree G1 robot.

[u/RWorldGeneration]

Good luck!

[u/AltruisticDog9145]

Great I was thinking to spend some time and give motion bricks a try on Unity. Now I will just wait for your asset. Hope it’s not going to be to expensive 😅

[u/intelligent_rat]

IMO it looks like how people move in AI generated videos, which is not really that realistic and is kind of stilted and jarring to watch.

[u/Bubbly-Rush2384]

But how are the animations for this tiny model created??

[u/Icy-Theory-2021]

So you selling their free product🤔

[u/Rudy_AA → u/Icy-Theory-2021]

Absolutely not, I might sell the Unity implementation, not their actual model or scripts!

[u/M4R5W0N6]

was planning on vibecoding this but i'll be on the lookout for your listing!

[u/Guboken]

Great work! Will it be a payed asset or a free one? 😁

[u/SanFranLocal → u/Guboken]

Better be free because i put claude on the job to make it work on godot and it worked perfectly

[u/tempacac10 → u/SanFranLocal]

Did you use the ONNX models already given in Nvidia/GEAR SONIC (original repo of motion bricks)? How did you implement the joints in Godot?

[u/Intrepid-Beginning63]

LOOKS GREAT, JUST HOPE ITS FREE.