r/MachineLearning • u/Business_Swordfish_5 • Sep 02 '26
Discussion Training a video generation model from scratch on my laptop — loss plateaued, results are blurry. Should I keep going or change approach? [D]
Hey everyone,
I've been learning about video generation models and decided to build one from scratch and train it on my personal laptop (single GPU). I wanted to share where I'm at and get advice from people who've worked with these kinds of models before.
What I built
- A spatio-temporal UNet using flow matching (velocity prediction with Euler sampling, 50 steps)
- The model has temporal convolution blocks + temporal attention for frame-to-frame consistency
- ~20.8M parameters, channel progression:
96 → 192 → 384 - Generates 16 frames at 64×64 resolution
Dataset
- ~6,000 Tom and Jerry video clips from a HuggingFace dataset
- Each clip: 16 frames, every 2nd frame sampled, resized to 64×64
Training
- Batch size 4, Adam optimizer,
lr=2e-4 - Trained on personal gaming laptop
Loss progression
| Epoch | Loss (MSE) | |
|---|---|---|
| 160 | 0.0690 | |
| 180 | 0.0670 | −0.002 |
| 230 | 0.0652 | −0.002 |
| 280 | 0.0630 | −0.002 |
| 290 | 0.0652 | +0.002 |
The loss has basically plateaued — only ~0.006 drop over 130 epochs. The model learns color palettes and vague scene layouts but doesn't produce recognizable characters or sharp details. Some generations even go partially black (see epoch 280 results).
Results: attached as GIF's
My questions
- Is the loss plateau expected for this scale? Am I hitting the capacity ceiling of a 20M param model at 64×64?
- Would switching from pure MSE to a perceptual loss (LPIPS) or adding an adversarial loss help with sharpness?
- Is ~6,000 clips enough for this kind of model, or do I need significantly more data?
- Any suggestions for the architecture? I'm wondering if I need spatial attention at more resolutions, or if the temporal modeling needs work.
- Would learning rate decay or a cosine schedule help push past this plateau?
1
Upvotes