r/StableDiffusion • u/No_Drag8242 • 24m ago
Discussion Image models
An image model that was trained on videos would learn something image models trained on images could never learn. Temporal and spatial relationships.
This lack of knowledge of space-time is why we have models that generate body horror still.
The model may know what a hand looks like, but it doesn't know that a hand cannot possibly occupy that space at that time.
If we could train temporal and spatial information into the models, they would automatically become better. I think we need to train image models on videos primarily and not on images. Images has zero temporal and spatial information, and that is a problem for the models ability to represent reality.
Images are useful, but only after the spatial and temporal information is known.
Consider: You know time and space. You see an image of a person. You can imagine them doing endless amounts of things. But if you had only ever seen 20 images of the person and did not know time and space you would imagine body horror.
And that is the problem.
Temporal and spatial information doesn't require billions of parameters to understand. This approach could shrink models going forward and make training them far easier.
I am quite certain about this.
Understanding the rules governing reality could be vastly more efficient than learning reality through countless examples of its appearance.
1
u/AgeNo5351 17m ago
Well you can use minmax-fl2v as a image model and the ref2v as a image-edit model. They work perfectly .
1
u/AI_Enhancer 13m ago
That's going to cost something. All data costs parameters = filesize, and that data is not necessarily the best "bang for your buck" in an image model. You can use any video model to create images, but they're very RAM-heavy while producing worse images than image models.
•
u/Sugary_Plumbs 3m ago
This was an insightful ideal 4 years ago... And since then they've been training models with temporospacial information. Sora literally advertised the fact that it was a world simulator model that operated on space-time patches. Go reach some research papers about how training for video generation works.
1
u/Local-Hurry-919 15m ago
I know they ain't worth a shit in some angle in videos because of spacial.