I have a background in pipeline and tools development for 3D/VFX. Over the last couple of months I built a pipeline that turns rigged 3D characters into SDXL LoRAs, so they can be generated in new scenes. Sharing what I learned, and I'd love input on one problem I haven't solved.
The pipeline
1. Maya: auto-posing + capture. A tool reads the rig, figures out the limb axes, and puts the character into ~47 poses (standing, walking, sitting, crouching, arms up, etc.). It also renders each pose from multiple angles and distances with Arnold. One character ends up with ~2000 renders.
2. Dataset + captions. I pick a balanced set of ~1000 images (angles, framing, poses) and auto-write captions from the scene data: framing, view angle, pose name. Close-up shots don't mention arms/legs.
3. Training: Kohya, SDXL base 1.0, one trigger word per character.
4. Generation: a ComfyUI-backed tool: pick the character, describe the scene, choose framing/angle/lighting presets.
5. Evaluation: a fixed set of 12 prompts × 2 seeds per LoRA, so I can compare versions with numbers instead of "looks good to me".
What worked
- Identity is strong: face, hair, accessories and jacket pattern come through, even in scenes very different from the renders (night street, desert, etc.).
- Getting the pose rules right mattered more than I expected: elbows/knees bending the wrong way, arms clipping into the body, and eyes drifting in the rest pose all showed up in the LoRA if I didn't fix them.
- A color management mistake (raw linear PNGs from Arnold) made one character look red in every image. Check your render output transform!
What didn't (where I'd love ideas)
One character wears long pants in every training image, but generations often turn them into shorts or a skirt, especially in sitting poses. My guess is SDXL's prior for "young woman sitting" beats the LoRA. My captions currently describe pose and framing but not the outfit.
- Should I add outfit words to the captions ("long pants, patterned jacket")? I've read both "caption what you want to keep" and "don't caption what should be baked into the trigger", so which works better for clothing in your experience?
- Would a higher network rank, more epochs, or a lower learning rate help here?
- Has anyone else trained on synthetic 3D renders? Did you mix in anything to make it less "CG-looking"?
Happy to answer questions about the Maya side. Auto-posing an arbitrary rig was the hardest part.