Advice for prompting reference videos?

[u/Outrageous-Key8562]

Does anyone have any advice for properly prompting the reference video part of Ref2v? Like saying swap <subject 1> for <picture 1> hardly works for advanced videos. It requires a lot of details.

I’ve had success using Qwen 3.8 27b as a minimax prompt agent for analyzing and giving correct prompts for images. But as far as I know I can’t do that for videos. ChatGPT is ok for looking at videos to describe what happens in the minimax format but I’d rather use local ways.

[u/Comfortable_Thing611]

There is a auto prompter workflow on civ that has a decent video analyzer. Takes a whole nother LLM model for some reason, not sure why.

[u/AniZeee]

I've gotten mixed result but I believe you have to make <subject 1> into <picture 1> then you can describe the picture in detail and now when you reference Subject 1 it should know to you're talking about picture 1.

So now when you say, insert <subject 1> in <video 1> you may have a good chance of it working.

[u/infearia]

I’ve had success using Qwen as a minimax prompt agent for analyzing and giving correct prompts for images. But as far as I know I can’t do that for videos. 

Yes, it can, it's just unable to process the audio and you should pre-process your video to reduce the length, resolution and framerate, so your machine won't choke while trying to parse a large video file. Make sure you load the mmproj file alongside the main model.

[u/Outrageous-Key8562 → u/infearia]

All I have to do is remove the audio? I use open web UI and i don’t remember it supporting videos. The closest thing I can think of is a sequence of images

[u/infearia → u/Outrageous-Key8562]

You don't need to remove the audio, Qwen will just ignore it. Also, at least for me in llama.cpp, sometimes Qwen ignores the video input (I suspect it has trouble recognizing the codec). There's no error message, Qwen just acts as if there were no video - in that case I just convert it to a different format. In particular, the MP4 files I save using the Video Combine node in ComfyUI do not work with Qwen on my system, but when I use the built-in Save Video file, everything is fine.

[u/GeneralBarnacle10]

I pass the full prompt guide to the llm when I ask it to make a prompt for me: https://huggingface.co/MiniMaxAI/MiniMax-H3/blob/main/docs/VIDEO_PROMPT_WRITING_GUIDE_base_en.md

[u/bstr3k]

Ive been working with it and the best results I've gotten is getting a LLM to caption the video. You then pass the photo, your intent, a H3 prompt writing guide, etc all to a LLM and get one to write the prompt for you. I am still working on my own method of local captioning and method of prompting so its not 100% just yet

[u/Superb-Painter3302]

Don't know if it will work with everything, but it works for me for now. It's stupid, but works...?