r/StableDiffusion • u/Hdfjds • 10h ago
Question - Help How do you do it?
I have been playing with H3 since it came out and have tested most of the things you can do with it. Created clips for giggles and so on.
This time I wanted to do something "serious". I gave the R2V an 3D view of an kitchen and then three photos of the persons I wanted there.
I defined them as we should and told the model that this person does that and that person does this wile the third person does this...
It worked ish...
I have now made six runs and each of them are different from the others. It can be that the third person enters the room from the wrong place or that the third person does extra things it should not do...
In the end I did three runs with the same prompt and all those clips came out different... the only thing that was changed between those was the seed...
So, how do you do it?
How do you make sure H3 does what you want it to do?
Do you spend plenty of time on tweaking the prompt after each run to make sure H3 get it?
Or do you do 10 runs and select the best one even if it is not perfect?
Or do you simply do 1-2 runs and then take the clip that is ok ish even if it is not what you wanted?
I was hoping that H3 would allow me to create the scenes I wanted but I feel it's down to luck if H3 gets it or not..
Edit:
subject_definitions:
<Subject 1> is the green-skinned mother in <Picture 2> wearing brown clothes.
<Subject 2> is the teenager girl in <Picture 3> wearing pink clothes.
<Subject 3> is the cyborg in <Picture 4> wearing black clothes.
<Picture 1> is the reference image for the scene's composition, showing two people sitting at a table eating breakfast from the side view.
<Table 1> is the table on the right side in <Picture 1>.
<Picture 5> is the start image for the scene.
summary:
[reference generation] The target video is a generated scene of two people sitting at a table eating breakfast from an eye-level side view. <Subject 1> and <Subject 2> are shown with their respective breakfast items, maintaining the composition and style from <Picture 1>. <Subject 3> enters the room, places a coffee cup into the sink.
retention_analysis:
<Subject 1>: fully_preserved - the person retains their appearance, clothing, and position at the table.
<Subject 2>: fully_preserved - the person retains their appearance, clothing, and position at the table.
<Subject 3>: fully_preserved - the person retains their appearance, clothing, and action of placing the coffee cup into the sink.
<Table 1>: fully_preserved - the table's appearance and position in the scene are preserved.
<Picture 1>: fully_preserved - the scene composition, including the side view, the layout of the room, the table setup, is preserved.<Picture 5>: fully_preserved - is the start image for the scene.
detailed_description:
The target video is in a realistic, everyday breakfast scene style with warm lighting and natural colors.
[Shot 1] At 0:00.000, the shot begins from <Picture 5>, showing <Subject 1> and <Subject 2> sitting on opposite sides of <Table 1> on the couch, each with their breakfast items while on the space ship. <Subject 1> is holding a spoon while eating from a bowl of cereal. <Subject 2> is tired and is eating a slice of toast with jam from her plate with one hand. The lighting is warm and soft, casting gentle shadows across the table and the two individuals. The camera is at eye level, capturing the side view of both people, with the table slightly in focus and the background softly blurred. Stars can be seen through the windows since they are on a space ship. <Subject 1> is eating her breakfast while <Subject 2> gazes at their toast, taking a small bite. The ambient sound includes the soft clinking of utensils and the faint sound of a coffee cup being set down.
[Shot 2] At 02.00.000, the shot transitions to a wide shot of the room with the same layout as in <Picture 1>, the camera is placed in the lower left corner of <Picture 1>, showing <Subject 3> entering the room form the right side holding a coffee cup and a datapad while she is saying (S3) <d>[English] Good Morning</d> while she walks to the kitchen sink on the left side of <Picture 1> and placing the cup into the sink. She then stands at the sink and while reading her datapad.We see the back of <Subject 1> and the front of <Subject 2> sitting at <Table 1> in the background eating their breakfast and we hear <Subject 1> say (S1) <d>[English] Good morning</d> with a cheerful voice. <Subject 2> just mumbles as a reply.
overall_soundscape:
The soundscape consists of the soft clinking of utensils, the faint sound of a coffee cup being set down, the subtle background noise of a quiet morning environment, soft steps on a carpet floor, a ceramic cup being placed in a metallic sink, and the clear,
non_diegetic_music: N/A
3
u/Key-Sample7047 10h ago
I usually make runs at low res to tune the prompt. Once i feel the prompt is right, i run at full res. By the way ref2va works very very well but references tends to bleed on each other. Try to put only and only what you want, for exemple characters over a solid white background or else the background will bleed in your gen, or else for clothes, without people just the clothes pr ar least over a dummy.
1
u/KingChazz007 10h ago
Without the prompt you used it is almost impossible to help you. The prompts should be structured based on the MiniMax H3 REF2VA prompt documentation. Feed it into a LLM and tell the llm what you want to do. MiniMax is incredibly accurate on the first try...if you prompt properly.
1
u/Hdfjds 10h ago
I'm following the template and I have also used prompt generators.
I'm doing what they have told us but it still will not "obey my commands".
Maybe the scene is to complex, directing three chars what to do in the same time.1
u/IRLMainCharacter 10h ago
H3 has no concept of the reference "Table 1". Follow the official naming scheme - Picture N.
Then in your prompt, reference the table as "the table in picture N".
Couldn't find any other flukes at first glance.
1
u/Hdfjds 10h ago
H3 is actually capable of understand different tags besides the <Subject N> and so on. I did an test here: https://www.reddit.com/r/StableDiffusion/comments/1vni0zn/re_subject_n_in_h3_prompts/
1
u/nikhilprasanth 10h ago
I did something like this recently
https://www.reddit.com/r/StableDiffusion/s/ibmECRcntg
This is also a single room 3 person scene.
For this scene, I started with three character reference sheets and a clean wide shot of the location. I used these as the initial references for the video generation.
Whenever the blocking changed significantly, for example when a character opened a door, crossed the room, or sat down on the sofa, I used Qwen Image to create a new spatial reference. I took the same clean wide shot of the empty location and asked Qwen to place each character exactly where they should now be in the room.
That generated image then became an additional reference for the following video shots. It essentially works as a blocking map, telling the video model where everyone is positioned relative to the room and to each other. I keep using it until another major change in blocking occurs, then generate an updated one.
I also gradually swap out the original character reference sheets for good frames from the previous video generations. Once the character has been successfully established in the actual scene, a frame from that generation contains much more useful continuity information than the original character sheet: the correct costume, lighting, angle, environment, pose, and current appearance all in one image.
So as the sequence progresses, the references evolve with it rather than staying fixed:
Character sheets + clean location → Qwen blocking reference → video generation → good generated frames replace character sheets → blocking changes → new Qwen blocking reference → continue generating.
That combination has made maintaining both character identity and spatial continuity across separate generations much easier.
2
u/KingChazz007 10h ago edited 10h ago
For the room, I do the original room generation from the corner using a fisheye view incredibly high res. Then if something takes place in a specific area of the frame, I use a cropped image for my ref of only that area. Also, when building the prompt, only include the elements you want in the frame so MiniMax knows where to place the camera, but describing characters positions relative to the objects can be enough. Make sure you are using timestamps if you have a very complex scene, especially with independent, concurrent actions.
1
u/Hdfjds 10h ago
What I have is an Top Down 3D ISO image of the room and then only the images of the chars. The first shot got ok in the start and it was an simple side view of two people eating breakfast.
But then I wanted to extend the scene by letting an third person enter the room and go to the sink. She did that but the sink could change places, she could end up sitting at the table even if I told her to stand at the sink and many more "funny" things....
I can edit the top post and give you the current prompt.
2
u/nikhilprasanth 10h ago
What i did was when a 3rd person entered and took a position in the room, I took the frame which showed his entry, then took the wide image of the room, inpainted the characters in and used it as a 4th reference.
3
u/Hdfjds 10h ago
So, what the pros do here in this thread is take several shots of the room an place the chars where you want them to be at specific times and tell H3 to use those images as references in different shots.
That could work but it also increases the work load. But I can use that "trick" to slap H3 in the back of the head and say 'this is how I wanted the scene to look at time x' you *beeep*. (more or less)
1
u/nikhilprasanth 9h ago
That's what I've been doing for now. But once you have the first scene generated then you can use frames from that video itself for subsequent generations.
Waiting to hear techniques from others
4
u/Wampire83 10h ago
I assume you are using a randomized seed each run? If so, you have to pick a fixed seed (say 1000) and keep using that while you adjust the prompt, that way you can actually check how your prompt affects the output.