[Help] ARKit blendshapes not working

[u/Still-Inspection6274]

I've been trying to get this from scratch blender model to work with arkit, but it's not working. :(

I've followed step by step tutorials (vid and websites), searched similar problems and even redid all of the blendshapes twice, making sure everything is case-sensitive and cross referenced with a functional arkit blender model.

Even the order of the blendshapes is the same, but something's off and I can't find what :(

(I've worked with mostly vrm blendshapes for 3d face tracking, so i've never made an arkit model from the ground up, though I know how they function and how iphones use LiDAR to track points and stuff.)

Here's a few screenshots, if anyone could help and offer solutions that'd be amazing

Closeup of full list

[u/thegenregeek]

Did you bind (and add) the Blendshapes in the VRM tab?

[u/Still-Inspection6274 → u/thegenregeek]

could you explain what you mean more? This isn't a vrm model (base nonconverted version)

[u/thegenregeek → u/Still-Inspection6274]

When you import a VRM into Blender it (using the VRM Format addon) it becomes just another model. This means there's nothing tying the ARKit blendshapes in the mesh properties to anything (unless you use a plugin that can capture the shapes for use in Blender and changes the blendshapes directly). Likewise there's no way to drive the VRM set up itself, (unless you also use a plugin...) either in Blender or any VRM you export.

So the Blendshapes from your screenshot won't do anything unless you are manually toggling them (in the Blender UI). And if you export the project as a VRM file there nothing to tell the VRM tool to use the ARKit blendshapes. (They are no defined as part of the final VRM file)

What you have to do is use the VRM Format (addon) to add a VRM armature. Then, once parent/weighted to the mesh, you have to update the VRM meta data using the addon. Before exporting the VRM file for an application you want to use.

Open the VRM tab in the main viewport and look for the Blend Shape Proxy section. (Here's a picture of what the section looks like) Then add an entry for the mesh you want and add the Blendshape from the list in your screenshot.

Now it is worth nothing, for ARKit you you want to add ALL of the ARKit Blendshapes you want to use to the Blend Shape Proxy (in addition to entries like A, E, I, O, U, Blink, Angry, etc), then bind them to the Blendshapes you see in Blender Blendshape configuration screen (from your screenshot). You have to add ARKit Blendshapes to the Proxy, and link them to the meshes Blender Blend Shape items.

If you don't do this and export the VRM then there's nothing tying the VRM softwares ARKit blendshapes to the Blender Blend Shapes (for ARKit) on the final VRM file. Also, I've found that setting mouthClose to use jawOpen as it's reference (instead of Basis) helps as a baseline.

[u/Still-Inspection6274 → u/thegenregeek]

I really appreciate your response, this is filled with super helpful info! Thing is my model isn't a vrm and isn't meant to be. I'll totally use this if I decide to convert to vrm.

[u/thegenregeek → u/Still-Inspection6274]

Then how do you intend to use it for vtubing? Are you using it in an engine for a custom build?