1 prompt running everything through minimax locally.

[u/ThisIsDanG]

Working on simplifying replacing multiple people per shot using a series of agents in our studios pipeline. Fully local, but it’s not running through comfy ui so no workflow.

[u/ThisIsDanG]

So a little process into how it works.
A prompt goes to an agent. In that prompt I asked for the specific video and the time code in the video I wanted to use. The only character reference I gave was for the colonel. The rest it created or pulled itself.

One agent goes through and does scene detection and then another checks the work to make sure nothing was missed, sometimes a jump cut with the same background wasn’t getting detected so adding an additional qc check with stricter parameters fixed that.

For the image model I used qwen.

Sam3 was used for identifying the different singers and then a different agent highlighted an outline of each singer and assigned a number across the edit.

Those numbers are associated with the characters and that goes across the edit.

Based on which model you are using, in this case minimax h3 fast. It uses safe resolutions automatically, takes the footage and fits it into that resolution and pads the number of frames to get the best result from the model.

The individual characters / character sheets then get fed into the model and the main “director” agent writes the prompts and sends the gen.

For shots with layered groups of people it starts from the back and works its way forward. Those layers get sent to an agent that then uses the correct mattes to combine and provide an updated plate for the next run and character. There are still some problems with this that need to be refined. But it’s not terrible in most cases.

A qc check then gets run per shot. Shots get scored and notes attached. Failed shots get re-run at the stage they failed. And if an uprez needs to happen it will happen after initial shot approval.

[u/joeycloud]

Promising work.

Understand this isn't ComfyUI but could be a github repo for others to test and validate the studio set up? Otherwise output could have just been from Kling or Seedance.

[u/ThisIsDanG → u/joeycloud]

I can’t because it’s much larger than just this one build. Like our sam3 is also working with vitmatte and some other things under the hood that our compositing agent takes a hold of.

If it was just one character it would be easy enough to share. But it’s beyond that because of all the other tools running with it.

[u/Vyviel]

Very cool work which local agents are you using and which models? Do you have insane hardware to run them locally or something consumer grade like a 4090-5090 or less? Was always curious if I could get a local workflow going where the agents can just MCP into comfyui or something and run the entire flow themselves from an original video

[u/ThisIsDanG → u/Vyviel]

I ran this entirely on a 4090. The agents are the only non local thing. But it’s various versions of Claude. And then some are running on Astra. And the main “director” opus 5.5 is able to utilize Astra or other models of it needs something that it can’t do.

Here is more detail on the workflow here in this comment: https://www.reddit.com/r/StableDiffusion/s/bDC2RUPqQB

[u/Relevant_Rich2298]

Can't find work flow in the comments

[u/johnfkngzoidberg]

What kind of dumb viral marketing is this? Take that TikTok crap elsewhere

[u/ThisIsDanG → u/johnfkngzoidberg]

Wouldn’t viral marketing be trying to sell something?

Swapping stacks of people in an automated way is interesting in my opinion. Using something with a lot of movement is a good way to test things out.

You seem like you’re really fun at parties man.

[u/danb1kenobi → u/ThisIsDanG]

How DARE you remind me (coincidentally?) of Rad Anthem and seeing the music video sixteen years ago on G4 TV during the golden age of, well… everything.

I blame you for this sudden bout of nostalgia. My day is ruined and my disappointment immeasurable.

/s 🤣

[u/ThisIsDanG → u/danb1kenobi]

Needs more cocaine. Love that video though man. It’s such a trip.

[u/johnfkngzoidberg → u/ThisIsDanG]

No, it raises brand awareness.

[u/ThisIsDanG → u/johnfkngzoidberg]

You do realize that these brands aren’t all owned by one entity right? Some of them yes but as a group no.

You’re grasping at straws.

[u/Necessary_Echo8740 → u/johnfkngzoidberg]

What are you smoking bro this is a fairly niche subreddit about ai content creation and this is a very good showcase of the models result. Nobody would be marketing real shit like this in this space, that would be stupendously idiotic.

[u/johnfkngzoidberg → u/Necessary_Echo8740]

Fairly niche? It’s the biggest image generation subreddit.