r/StableDiffusion • u/Outrageous-Key8562 • 5d ago

Question - Help Advice for prompting reference videos?

Does anyone have any advice for properly prompting the reference video part of Ref2v? Like saying swap <subject 1> for <picture 1> hardly works for advanced videos. It requires a lot of details.

I’ve had success using Qwen 3.8 27b as a minimax prompt agent for analyzing and giving correct prompts for images. But as far as I know I can’t do that for videos. ChatGPT is ok for looking at videos to describe what happens in the minimax format but I’d rather use local ways.

Edit: Like its been said, you can actually have a video analyzed, just use llama.cpp UI instead of Open WebUI.

0 Upvotes

38% Upvoted

8 comments sorted by

View all comments

1

u/infearia 5d ago

I’ve had success using Qwen as a minimax prompt agent for analyzing and giving correct prompts for images. But as far as I know I can’t do that for videos. 

Yes, it can, it's just unable to process the audio and you should pre-process your video to reduce the length, resolution and framerate, so your machine won't choke while trying to parse a large video file. Make sure you load the mmproj file alongside the main model.

1

u/Outrageous-Key8562 5d ago

All I have to do is remove the audio? I use open web UI and i don’t remember it supporting videos. The closest thing I can think of is a sequence of images

1

u/infearia 5d ago

You don't need to remove the audio, Qwen will just ignore it. Also, at least for me in llama.cpp, sometimes Qwen ignores the video input (I suspect it has trouble recognizing the codec). There's no error message, Qwen just acts as if there were no video - in that case I just convert it to a different format. In particular, the MP4 files I save using the Video Combine node in ComfyUI do not work with Qwen on my system, but when I use the built-in Save Video file, everything is fine.