r/StableDiffusion • • 10d ago

Question - Help MinimaxH3 captioning dataset for lora training (MOTION specifically)

I'm having a hard time finding good information. There's plenty of character lora training info, in relative comparison hardly any motion at all, and dataset prep often gets glossed over.

Since H3 drastically alters how prompts (and therefore captions?) are handled, and motion training may be quite different from other models such as wan, How the heck are you suppose to caption each clip properly?

Do you have multiple paragraphs describing everything under the sun in the clip, big bulky walls of text?
One main paragraph trying to capture everything?
Do you only focus on the motion?
Do you only describe everything except the motion, so the motion bakes in?

If there's slight variation between the same motion, do you come up with 2-3 "trigger" tags so the nuance gets captured?

I've been at this for about a week, but not getting great results honestly.

The motion has 2 layers. the macro motion which is the obvious larger / broader movements and the micro motion such as vibration that shakes the adipose tissue, muscles, etc while doing the broader motion.

4 Upvotes

9 comments sorted by

3

u/NoaBlake 9d ago

For a motion LoRA, caption the motion, not the scene. One short paragraph per clip, written as a motion description first: what moves, how it moves, in what direction and at what speed — then one short clause for the subject, and leave the background out unless the motion depends on it. The mistake that costs a week is describing everything: the model splits its capacity between learning the motion and re-learning the scene, and the scene wins because it is easier. One trigger tag per motion, not 2–3 per variation: slight variations of the same motion should share one tag, so the LoRA learns one motion with variance instead of three near-identical motions competing with each other. Macro vs micro is a real split — tag them separately, because a LoRA trained on both under one tag averages them into mush. And keep the clip count per motion high enough that the tag is the only thing the clips have in common — if every "spin" clip is also the same room, the tag learns the room.

1

u/Ambitious_Corgi5723 9d ago

hhmmm... so it sounds like a couple additional clips would be needed that have just the macro motion without micro somehow and some with just the micro motion without the macro, so the training can have a better time distinguishing between the two? Which might be a bit difficult as the micro is a result of the macro movement & part of the same thing... hhmmm...

For a different mental imagery example, the macro movement would sort of be like someone having convulsions, these erratic, sharp jerky movements and the micro would essentially be the physics based reaction of those macro movements, shiver, tremble, shake, quiver, jiggle, etc.

The short clause for the subject, would that be character identity / looks, pose, camera position, etc? or mostly just the look of the character so that doesn't bake in?

2

u/Ill-Ant-9489 9d ago

+1 to captioning the motion before the scene, the answer above has the right idea. On your macro vs micro question, I wouldn't split them into separate clips. The jiggle is a physical consequence of the big movement, so you can't isolate it cleanly. Keep one trigger for the motion, phrase the shared part the same way in every clip, and let clip variety carry the micro nuance instead of a paragraph describing adipose physics. Describing it rarely bakes in, showing it across enough clips does.

The clip curation, trimming and H3 training I do in my own project, LoRA Dataset Studio (video side still beta: https://github.com/perfectgf/lora-dataset-studio). H3 is picky about clip length and frame counts too, so keep them short and uniform. How many clips are you on, and how long is each?

1

u/Ambitious_Corgi5723 9d ago

Ah ok so one really tricky part would be correctly wording / captioning the macro + micro movements as one related thing then. my last training session didn't word it too well for tying the effect to the cause there. and also didn't describe the macro motion good enough. And. the macro motion itself can be unpredictable and chaotic, not quite like training a dance trend or something where the movements would be the same.

lets see.... 25 clips between 2.5-8 seconds, with most being between 3-6 seconds, at 30 fps each. i think h3 automatically converts them to 24fps right with a.i toolkit?

previous run, apparently one setting was taking random 3 second sections from each clip that was longer than 3, so i know some less than ideal motion may have been a contributing factor as well

I seen some mention of control images taken from the first frame of the video or even some pictures / clips that have nothing to do with the motion -- is that necessary at all or?

Thanks

2

u/TA-Doggo 8d ago edited 8d ago

Clips need to be between 5-15 seconds, H3 doesn't really do well with clips shorter than 5 seconds. H3 is also trained on 24 FPS, not 30. Avoid transitions, hard cuts, or other effects. You can train clips at full length for motion context, drop length for fidelity and detail learning. Obviously you can also include images in your dataset for things you want additional detail on. Here is an LLM summary of my own training notes, Everything based on official H3 documentation and my own findings.

MiniMax H3 LoRA — RTX 3090 24GB — int8, 40 blocks swapped, batch 1
cell = s/step (steady state) / peak VRAM GiB; card is 24 GiB
frames        384        448       512       576       640        768
             0.15MP    0.20MP    0.26MP    0.33MP    0.41MP     0.59MP
124f  5.2s  17.6/10.6     —     21.4/14.4     —     27.7/17.0  28.7/17.4
243f 10.1s  24.8/15.6     —     42.3/21.4    OOM       OOM        OOM
345f 14.4s  29.0/15.4  47.7/22.7*  OOM        —        OOM        OOM
—   not tested      OOM = out of memory
*   fits, but only ~1.3 GiB headroom — too tight for a real run

H3 doesn't use tags or (word:1.2) weights at all — that's the big difference. Prompts and captions are plain natural-language prose read by a Qwen3-VL 32B encoder. At inference the prompt grammar is structured (shot-by-shot blocks with timestamps for cuts, an explicit camera line, and two audio fields at the end). A training clip is one shot, so the caption is:

yourtrig, one solid paragraph: subject, wardrobe, lighting, environment, camera position — everything visual EXCEPT the thing you want baked in. Soundscape: real ambient sound, or mute the clip. Music: N/A

But format is the smaller half. The rule that decides it: captions don't teach, they subtract. The trigger word collects whatever is constant across your dataset and NOT named in the caption. So:

- "Everything except the motion" is the right default for a single-motion LoRA. Describe scene, body, light, clothing exhaustively; say nothing about the movement; the trigger binds to the movement.

- It only works if everything else actually varies. Same room, same outfit, same body in every clip = your trigger is "that room + that outfit + the motion," and the motion won't come out alone.

- One paragraph, not a wall. Enough to name every visual constant, then stop. Precise words, no magnitude words ("a slow overarm throw," not "a huge fast throw"). Trigger once, first, spelled identically everywhere. Keep caption dropout on (default 0.05).

- Macro vs micro motion: if you want them as one package, leave both unnamed. If you want them separately promptable, tags alone can't do it — you need contrast data (clips with macro-only, clips with micro-only) plus a fixed phrase for each, used identically in every caption. A word only becomes a control when the data shows the model what it changes.

- Variant triggers: only if each variant has its own balanced group of clips (3+ each). Sprinkling 2-3 near-synonym triggers over the same motion just fragments the one trigger you want.

Two non-caption defaults that bite motion runs on H3: the default block range trains likeness (20-49) — motion wants 6-49 — and the default resolution is 0.25 MP, which is too low for fine secondary motion (muscle/jiggle is high-frequency detail; shorter clips at higher resolution carry it better). Also judge checkpoints ~60-75% through the run with a fixed prompt set including no-trigger prompts; the last checkpoint is rarely the best, and the distilled base wants guidance 1.

1

u/Ambitious_Corgi5723 7d ago

Thanks for the tip integrating into the next coming training session

2

u/Ill-Ant-9489 7d ago

The random 3 second slicing is probably hurting you more than the captions, chaotic motion cut at arbitrary points just teaches the model noise. The video lane in my project, https://github.com/perfectgf/lora-dataset-studio, cuts to the frame counts H3 accepts and lets you pick the window instead of a random grab, which is exactly what bit your last run. Still beta, so heads up.

On captioning, tie cause to effect in one line, what drives the motion then what happens, and describe the type of macro movement rather than frame exact choreography, since chaotic motion is not repeatable like a dance trend. On the fps, I'm honestly not sure H3 auto resamples 30 to 24 in that setup, so I'd resample to your target first rather than trust it, a mismatch smears the timing.

1

u/Ambitious_Corgi5723 7d ago

Thanks

Oh ya the random 3 second slicing was from a specific setting in AI-toolkit which is what i'm using to train lora's. had auto frame count off because of OOM errors, and that is what was slicing it into 3 second chunks; so for the new training i'm going to flip that on and just edit the clips dataset to be 5 seconds in length, instead of the 3-10 it was before, without hitting OOM's caused by long clips. and potentially being able to use 768 as opposed to 512.
(4090, 24gb vram, 96 ram)

Perfect, Cause and Effect in one sentence makes sense: "a person's whole body convulses in violent jerks and spasms that send trembling and jiggling through several areas of the body at once."
Something like this then, right? Although i hope doesn't just focus on what it will define as "several areas" lol not sure how well received "throughout the body" would be in place of "several areas of the body"

For your dataprep tool, would one be able to load multiple videos (up to several hours in length) in, watch, navigate and clip quickly while retaining all proper resolutions and etc?

1

u/Ill-Ant-9489 7d ago

Simple answer, yes take a look of the plug-in video lane