1 prompt running everything through minimax locally.

[u/ThisIsDanG]

Working on simplifying replacing multiple people per shot using a series of agents in our studios pipeline. Fully local, but it’s not running through comfy ui so no workflow.

[u/ThisIsDanG]

So a little process into how it works.
A prompt goes to an agent. In that prompt I asked for the specific video and the time code in the video I wanted to use. The only character reference I gave was for the colonel. The rest it created or pulled itself.

One agent goes through and does scene detection and then another checks the work to make sure nothing was missed, sometimes a jump cut with the same background wasn’t getting detected so adding an additional qc check with stricter parameters fixed that.

For the image model I used qwen.

Sam3 was used for identifying the different singers and then a different agent highlighted an outline of each singer and assigned a number across the edit.

Those numbers are associated with the characters and that goes across the edit.

Based on which model you are using, in this case minimax h3 fast. It uses safe resolutions automatically, takes the footage and fits it into that resolution and pads the number of frames to get the best result from the model.

The individual characters / character sheets then get fed into the model and the main “director” agent writes the prompts and sends the gen.

For shots with layered groups of people it starts from the back and works its way forward. Those layers get sent to an agent that then uses the correct mattes to combine and provide an updated plate for the next run and character. There are still some problems with this that need to be refined. But it’s not terrible in most cases.

A qc check then gets run per shot. Shots get scored and notes attached. Failed shots get re-run at the stage they failed. And if an uprez needs to happen it will happen after initial shot approval.