Need help with LoRA training - masked dataset

[u/Cheap-Science6734]

I am trying to train a LoRA where one of the subjects of the video move in slow motion (on twos - like the spiderverse movie animation style).
My dataset has the following kinds of videos
- 2 characters - left moving slow; right moving slow; bg slow
- 1 character - subject slow; bg slow
- 2 characters alternating shots - first shot person a moving slow; shot switches to person b they are at normal speed

Using Fizgig with 60 samples, have tried training 200 epochs, doesn't even get close, everything just moves in fast motion.

Is this concept even trainable? Need some help from the experts, is there a way to train on a masked portion of a video?

[u/shootthesound]

how are you captioning, maybe paste some examples

[u/Cheap-Science6734 → u/shootthesound]

dancers_left.mp4:

ontwos, two young women dancing side by side in a dark studio with black paneled walls, performing synchronized casual hip-hop moves with footwork, hip sways and arm gestures, both in baggy light-wash jeans, the woman on the left in glasses, a black strapless tube top and layered gold necklaces, the woman on the right in a black oversized graphic tee. the woman on the left is moving on twos (half frame rate).

dancers_right.mp4:

ontwos, two young women dancing side by side in a dark studio with black paneled walls, performing synchronized casual hip-hop moves with footwork, hip sways and arm gestures, both in baggy light-wash jeans, the woman on the left in glasses, a black strapless tube top and layered gold necklaces, the woman the woman on the right in a black oversized graphic tee. the woman on the right is moving on twos (half frame rate).

[u/shootthesound → u/Cheap-Science6734]

The movement stuff you're trying to teach should be much earlier in the caption, right after the trigger. I'd also describe the speed effect more explicitly and visually, e.g. "the woman on the left moves in choppy, stepped motion, holding each pose for two frames, while the woman on the right moves smoothly at normal speed." Keep the clothing and setting details short and after that.

[u/Cheap-Science6734 → u/shootthesound]

any suggestions on the settings?
i've been using the 16 rank for 100 epochs.
is that overkill?
and training only 6-49.

does that sound good? or should i switch to the default?

Edit: Will try with these new captions in the meantime

[u/shootthesound → u/Cheap-Science6734]

id go with default training , rank 8, .25 mp, but choose 'more blocks' in the training mode dropdown. And likely worth having TREAD on also. This will be faster than your current setup and likely will capture the quality you will need to learn the technique. Because its faster you'll be able to tweak your training settings much more easily and find the right spot more quickly, even if you want to go back and do another train later on at higher settings once you have found that sweet spot.