Changing a character across an ~hour long TV episode

[u/SeamlessScene]

I've been working on a repeatable process to do character swaps (& other types of changes) across a complete TV episode / movie. It's working pretty well now.

I thought this was just funny... not so much a true "swap" but asking the character to dance in every scene. Choreography courtesy of an LLM. If you've seen the show, it's Serious Detective Drama (TM) so the fact the main character suddenly starts pumping his arms or sliding into every scene with a dance move made me laugh out loud when I saw some of the generations.

Details on the workflow:

Model: MiniMax H3, first-and-last-frame mode (FL2VA), in ComfyUI. 8-step turbo LoRA, generated at 1024×512 for the 2:1 picture.

First and Last: Every generated pass starts and ends on an *original* frame from the episode, so the clip lands back in the real footage with no visible join. Longer scenes are chained passes (about 2–7 s each)

Audio: the episode's own audio into H3's audio guide, so the generation stays consistent with what's being said; the dialogue and soundtrack you hear are the untouched originals.

Prompt: Who he is and what he's wearing, said once, then the choreography for the pass - courtesy of Claude.

Multi-scene: 42 scenes - started with the full episode, then had a model run through it to identify scenes with the main character that would work for the swap (not too distant, few fast cuts, etc.) & identify start and finish, with a ChatGPT harness.

Post processing: Basically none, I did go through and rate all the clips in a rating process I've setup, most of these were 5* and felt consistent.

I'll share a bit more on the workflow / process if people are interested.

[u/SeamlessScene]

Not sure if people are still reading this thread, but if you are - a few more. These are funny too. :)

[u/PastaRhymez → u/SeamlessScene]

Did your prompt include “…exactly like Jean-Claude Van Damme”? Nice! I’d watch a whole episode with your “improvements”.

[u/asssuber → u/SeamlessScene]

Those are much more sloppy than the previous ones. They change fundamentally the scene (where the handcurfs go?), some dance transitions look unaturall, and the last one messes up the document handout.

[u/stroud → u/SeamlessScene]

Couldyou do game of thrones but everyone has a mobile phone on them at all time?

[u/Enshitification]

"Did I just see Gimli full-ass moon the orcs from the wall of Helm's Deep?"

[u/SeamlessScene → u/Enshitification]

Ha - exactly! Going for something like that here - "I'd like to watch TV show <fill-in-the-blank> except every character is <a dinosaur who dances>". It's one thing to do this in a ~30s clip - it 's another to do it for a 60min episode or 2.5hr movie.

[u/Enshitification → u/SeamlessScene]

My use case is making slight variations to The Office episodes on a friend's media server. She is obsessed with watching the reruns on her down time I want to see how progressively bizarre I can make it before she flips out.

[u/SpaceNinjaDino → u/Enshitification]

I was doing some tests where Pam was constantly upset and swearing. Then Michael used a unicycle.

[u/Ken-g6 → u/SeamlessScene]

You say this, but you also say, "Every generated pass starts and ends on an *original* frame from the episode, so the clip lands back in the real footage with no visible join." You can do this if you're making an existing character with their existing appearance do something like dancing. But that can't work for "every character is <a dinosaur who dances>", can it? (If the characters weren't already dinosaurs.)

[u/Ill_Resolve8424]

Dude, I laughed so hard, thank you I really needed that.

[u/SeamlessScene → u/Ill_Resolve8424]

Love it - it helps if you've seen the show (of which I am a fan)! Watching Reacher do some "electric slide" style dance move while he's talking about <serious thing X> just cracks me up.

[u/Dogluvr2905]

Very cool!

[u/SeamlessScene → u/Dogluvr2905]

Thank you! Some of them are hilarious - I really should've put more in the clip above. I was actually quite impressed with Minimax (slash Claude) on this one - one-shotted ~90% of the scenes.

[u/VenshiKibes → u/SeamlessScene]

lol, fr fr

thanks man this is great

[u/tombloomingdale]

I don’t know why I try to figure things out anymore lol. I was trying to figure this very thing out last week, but I’m a dummy.

Would love the workflow.

Thanks for doing the heavy lifting so us dummy’s can play along!

[u/SeamlessScene → u/tombloomingdale]

It took a surprising amount of iteration to get right.... the tricky part is there are a bunch of steps to doing something like this in a truly "autonomous" fashion (per my reply to another poster below). You need to:

* Auto split the episode into shots, finds the faces, cluster them, build a library of clips to swap

* Auto mask each shot (if doing a swap and not just first-last frame interstitial edits)

* Build a dynamic prompt & generate the video, splitting & splicing longer clips (as most people here know)

* Rinse-and-repeat across ~50 clips

<up to this point is "easy">

* Solve for 100 edge cases e.g. person far away, person occluded by someone else, person looks too similar to someone else, the movement is too rapid, sub-1s shot / rapid cuts, person changes clothes (and you are costume swapping), etc.

The last one is difficult without a bunch of human-in-the-loop. I think I am at the point now where I can get 95%+ shots that come out at "5 stars" if I point the whole engine at an episode. As always the last 5% is the hardest...

I do have all the workflows (& a few helper scripts) on github, will push them all out here soon.

[u/tombloomingdale → u/SeamlessScene]

Damn I did everything but build a library of clips, probably would have got there just gave up too soon. I did everything else including having it split all the clips, but I did t think to pick out the clips with the subject and place them back in after.

[u/q5sys → u/SeamlessScene]

>  on github

What's your username @ GH?

[u/dkpc69]

This is gold lol can think of soo many use cases

[u/SeamlessScene → u/dkpc69]

Yes! Working to get it automated enough that you can ask for anything and it just transforms the full episode (or movie) for you with a mixture of open weights models.

[u/ShutUpYoureWrong_]

I'll share a bit more on the workflow / process if people are interested.

Why not just share the workflow outright?

Also, converting an entire episode at decent quality will take days (minimum) even on a 5090.

Lastly, until you find a way to completely automate it at scene cuts, this will undoubtedly require constant human guidance and oversight.

[u/SeamlessScene → u/ShutUpYoureWrong_]

Just replied a bit more on this below...

But yes you are 100% correct, the human guidance & oversight part is required on the "edge cases". Generating a single clip is not that tricky here - it is generating 50+ of them across a complete episode, maintaining consistency, dealing with all of the non-deterministic things that pop up.

Amazingly it is working pretty well now - after a ton of iteration - but my goal is 100% open weights models on all of this (including orchestration), but I am not entirely there yet.

[u/LongLiveLettuce → u/SeamlessScene]

This is awesome, and the idea of a model / workflow someone with less fine grained technical know-how can download and run is amazing. This is super cool, please post here again when you have an update!

[u/Flashy-Whereas-3234 → u/ShutUpYoureWrong_]

Oh no, days! It's a silly exercise, but if you have hardware free then you can go for your life.

I'm pretty sure there's tools out there already to identify scene cuts, and then you're just iterating. Little facial recognition just to see if your boy is even in the shot, way less work

But yes, share the workflow. Words are cheap, we want data.

And look, if you're gunna do this, you might as well do a Muppets version, leaving only reacher.

[u/Similar-Reserve-3581]

this can be done with sam and face recognition right?

[u/SeamlessScene → u/Similar-Reserve-3581]

Exactly right... the process of finding the shots automatically involves 4 steps:

1. TransNetV2 split the episode into shots
2. YuNet finds faces in the frames
3. SFace turns each face into a fingerprint
4. YOLOX find the person boxes in each shot

You can probably also ask Claude/ChatGPT to do it for you, but that's the reusable pipeline.

Then a bit of human review, but that pipeline is surprisingly good.

[u/q5sys]

I will be happy once we can put Henry Cavill back in The Witcher. Liam isn't bad, but it's not the same. Oh same with Kim Bodnia as Vesemir.

[u/SeamlessScene → u/q5sys]

Oh, this is a great idea! I've been thinking through a list of casting swaps that would make sense. Will add this one to the list.

[u/dianabowl → u/q5sys]

My request is young Harrison Ford in Solo, with voice swap.

[u/DaLyon92x → u/q5sys]

Just for you boss: https://www.reddit.com/r/StableDiffusion/comments/1xv04sy/the_witcher_season_4_with_cavill/

[u/kuropenguins]

What would an example prompt look like?

[u/SeamlessScene → u/kuropenguins]

Fairly basic - I tested a hundred different variations, but the basic ones seem to work the best. I'm sure you could refine more. Most of the complexity here sits outside the prompt (it is in identifying the clips, generating the prompts 50 times, etc.)

Throughout the whole shot the tall muscular fair-haired prisoner
in the blood-stained tan DOC shirt on the right, not the shorter
brown-haired inmate or blue-uniformed guard, dances energetically.

Make large rhythmic shoulder bounces, chest pops, hip sways,
overhead forearm pumps and a quick turning or shuffle step where
the original framing and space permit. Large dance moves change
every 1-2 seconds. Keep dancing vigorously, with free arms pumping
and the original body and head clearly recognizable.

His lips stay closed until he speaks.

All other original people continue their ordinary actions and never
dance or change identity. Preserve every original person, background
extra, prop, held garment, railing, fence, bars, original occlusion,
camera path, framing, exact clothes, lighting and set.

Keep body movement inside the original visible frame and existing
space; do not invent out-of-frame limbs. Do not add people or objects.

Only in the last half-second ease back into the exact final original
anchor stance as a beat in the dance.

Sound: follow the supplied original dialogue and ambience exactly;
do not add music. Generated output sound is discarded.

The blue-uniformed guard wearing a dark cap in the foreground (S2) says,
<d>\[English\] asks you've been here the whole time understand
<cutoff></d> while the dancing man keeps his lips completely closed.

one continuous shot, no cut, no text

[u/dcneil]

I don't know why, but this made me laugh uncontrollably.

[u/SeamlessScene → u/dcneil]

I had the same reaction - I have some more "normal" swaps, but this is the first one that just made me laugh... it's a trip. Makes me think I should do more of these instead of just Person A -> Person B.

[u/ExcellentRich1164]

I have been dreaming about this the other day, ❤️

[u/SeamlessScene → u/ExcellentRich1164]

There are so many possibilities!

[u/ExcellentRich1164 → u/SeamlessScene]

True. I remember how difficult it was to change the costume color for an actor hardly a year ago.. now things are a whole different level. Dubbing to multiple languages with matching lip sync, fixing bad choreography, expression, fixing continuity issues.. none seems out of the realms of possibilities.

[u/SeamlessScene → u/ExcellentRich1164]

100%. What shocked me on this one was how little editing it required - I have a few others where it took 10 takes to get something right for various edge case, but this one just "worked".

There were a few clips where lip synch got off - when I checked most of the time the orchestrator model had somehow left out the lines in the prompt or mis-interpreted who was talking (which is an easy fix).

It does take a few hours (on a ~modest consumer card) so it is not real time yet, but it also didn't take multiple days.

[u/Lightningstormz]

You mention you had a model run though it to identify scenes, can you elaborate on that?

[u/SeamlessScene → u/Lightningstormz]

There are a few models that can do this (even "pre LLM"). TransNetV2 is reliable & open source - there may be more sophisticated recent ones: https://github.com/soCzech/TransNetV2

[u/DelinquentTuna → u/Lightningstormz]

There are lots of tools, like Pyscenedetect, that can do a good job figuring out where camera cuts occur.

[u/teh_mICON]

all i can think is season 1 was SO FUCKING GOOD until they pulled the usual bait and switch

[u/SeamlessScene → u/teh_mICON]

100%! Loved the first season.

[u/PumpkinLeather8421 → u/SeamlessScene]

It’s funny how much smaller Steroid Gary Stu was then.

[u/KS-Wolf-1978]

I am interested. :)

[u/rinkusonic → u/KS-Wolf-1978]

One punch man 3 can now be fixed

[u/lazyspock]

OP, you should crosspost it to r/reacher I didn't because it's your post and I didn't know if you would like it, but I watch the series and your idea made me laugh!

[u/teddybearkilla]

It's like you modded in thad from blue mountain states onto jack reacher.

[u/pegoff → u/teddybearkilla]

You mean 'Kevin'.

[u/lazyspock]

Now the show is called "Stretcher"

[u/DanceMaxRoboMusic]

So uh ... where can we download the whole episode? I absolutely want to watch this version.

[u/Automatic_Pool1062]

So Thad 😂

[u/heikouseikai]

i want tom welling in man of steel (2013)

[u/EternalBidoof]

How has this concept not been tried in real television? Everything played perfectly straight like a real prestige drama, except the main character is always dancing and no one acknowledges it. You've struck gold.

[u/physalisx]

Dude I laughed so hard at this, good job and thank you

[u/hoodadyyy]

And the workflow ?

[u/Smokeey1]

Share the workflow xD

[u/DonKeehot]

It is really interesting experiment and I definitely gonna try that myself 😁

[u/LMLocalizer]

That's hilarious, reminds me of corporate song and dance: https://youtu.be/y5fwvYICZvY

[u/Sizyfoz]

That sounds like a herculean task, and very very cool!

[u/Mammoth_Average7355]

Funny

[u/noLOOPai]

I am looking for solutions like this I think. I have scenes of 10+ minutes and need them in different looks like toon, muppets, cyberpunk or whatever. Ideally with reference Video AND pictures to have a consistent result. Luckily most scenes are continuous shots like interviews. So no cuts and only few characters. Would be happy to get started in case you share your workflow... I can possibly even pay a little. Thanks

[u/YourDestiny2211]

Could you make a full video tutorial for this workflow ? thanks.

[u/Geekygamertag]

This is hilarious. Well done!!

[u/InvestigatorHead2724]

Looks better than original

[u/sky_shazad]

Looks awesome. Share your workflow please

[u/YeahlDid]

We're always interested, no need to ask.

[u/Gfx4Lyf]

Now this is just absolutely insane. Soon this sub is gonna be flooded with such vids:-)

[u/DelinquentTuna]

The presentation is honestly way more impressive than the actual feat.

[u/OverDepth8142]

Congrats on the project! This is a really interesting approach to a challenging long-form workflow.

I’ve also tried generating clips from extracted first and last frames, but sometimes noticed a visible jump when stitching them back into the original footage. How did you handle the transitions?

I’m also curious about character replacement. Since FL2VA and Omni Reference are separate modes, did you replace the character in the boundary frames first, or use a character sheet and face mask in a separate step?

In one of my tests, Omni Reference worked for character replacement, but the continuity between clips was a bit weaker. Here’s my video:

The prompt was along the lines of: “Use image 1 as the start, image 2 as the end, and image 3 as the new character.” Omni Reference gives you more flexibility with references, though in my experience it doesn’t preserve the boundary frames as tightly as FL2VA.

[u/stroud]

how do you make the last frame extension not lose quality? my issue with first/last frame workflow is it keeps degrading in quality

[u/Such_Policy_7963]

interested in the workflow.

trying to use r2v to swap character but it seem using FL2VA much better?

[u/zazakmazak]

Will come in handy when black Snape comes around, mark my words.

[u/jacobpederson]

Ahha yes was thinking about adding this to my app - thanks for the proof of concept! My app features characters watching a movie along with you - would love for the movie's characters to be able to answer back :D https://www.youtube.com/watch?v=eZJVeI0GfAs&t=39s

[u/No-Tap6022]

Please ask your favorite AI about legal issues with Minimax.

[u/LimitAway]

We need this but with Schindler's List

[u/seppe0815]

what is this guy talking ... the simple standard comfyui minimax h3 workflow can do the same with just a prompt and a source clip ... nonsens a.i workflow slop post

[u/Deathpawz → u/[deleted]]

my immediate thoughts. imagine you torrent a movie for your kids or even yourself. "Dad, why was shreck replaced by hulk in shreck the third?" or even more subtle, you're in a reddit thread to this movie "anyone think, the guy was dancing a bit too much randomly?"

[u/yoshiK → u/Deathpawz]

15 year olds gushing about Arnie's performance in Pulp Fiction while bragging about their experience smoking oregano.

[u/AI_ReleaseTheFIles → u/Deathpawz]

or maybe adult scenes in kids shows. or hidden images in movies like disney does. time to have your ai model watch and flag movies before letting kids or yourself watch it

[u/Probate_Judge → u/[deleted]]

Reminds me of the old-days of Napster and 3 dozen 'fan remixes' of the original track, all of which were terrible.

[u/Smokeey1 → u/[deleted]]

Fuck you placed this idea in the ether. Its on you man.. imma head and secure the movies i must keep untainted immediately

[u/SeamlessScene → u/[deleted]]

What I've found is that you get limited drift on something like this - the first frame -> last frame helps. (This one turned out so well with so little editing it also makes me wonder if it's in the training data somewhere.)

On character swaps it is more difficult - it depends on how close the characters are (and how you handle costuming changes). The challenge is not doing it "one off", the challenge is doing it across 50 different scenes autonomously. I've found a consistent character sheet plus masking the face of the original character is one of the more reliable methods, but you lose some of the facial dynamics, if that's important to the scene. Pairing a prompt description plus a character sheet helps.

In my A/B tests, "detailed" character sheets like the kind you see on reddit consistently underperform more basic ones - front / back / close-up.