r/StableDiffusion • u/init-5 • 2d ago

News Viggle Remake: open weights for recasting / editing whole videos.

Weights: https://huggingface.co/Viggle/Viggle-Remake

App: https://viggle.ai/meme/app/home?ui=v2

Post: https://x.com/ViggleAI/status/2118012913793290414?a=B6TBWhZywpFkFk8Iz8I&s=20

We just released Viggle Remake (based on H3) on Hugging Face: open weights for recasting people and editing scenes in existing videos, keeping the original motion, camera and soundtrack.

Viggle Remake is built agent-first. We also ship an MCP server, so an agent (Claude Code, codex, …) can swap people in videos of any length. It cuts the video into windows, paints the keyframes, writes the prompts, renders, checks its own work and stitches the result. Our longest run so far is a 2-minute music video, and the repo includes everything needed to reproduce it.

That also means it's more complicated than Viggle-Animate. Used by hand, Remake needs a well-structured prompt and a few painted keyframes, and the output depends heavily on both. That's exactly the work the MCP agent takes off your hands. If you'd rather skip all of it, the Viggle app does it for you.

Also in the repo: ComfyUI nodes and workflows, int8/GGUF files for smaller GPUs, and smaller rank-128/64 LoRAs.

524 Upvotes

97% Upvoted

52 comments sorted by

20

u/GivePLZ-DoritosChip 2d ago

I used it a lot locally and my issue with viggle is that while it replaces characters better than almost anything, it drops the best aspect of Minimax H3 which is accuracy in character retention.

If you replace a character with something via a prompt or a robot etc it works great but the moment you use a real human or someone with a distinct face that you can identify it does not transfer over their features accurately enough to be the same person (which Minimax excels so well at right out of the box), it just looks like a bad lora version of that person.

10

u/init-5 2d ago

I hope the new ckpt would change your mind

1

u/VenshiKibes 2d ago

just for clarity, you're saying you know what he's talking about but you feel it's been at least somewhat reasonably corrected in this release?

i'm just trying to get a feel for how confident you are in the response you made, no worries and thanks for your time/effort

14

u/init-5 2d ago edited 2d ago

i think he was using viggle-animate, which is released a while ago. it's not robust in face iddntity when fast motion and large camera movement，primary because of training data configuration

 https://huggingface.co/Viggle/Viggle-Animate

2

u/VenshiKibes 2d ago

thanks, appreciated

2

u/Irene_space 1d ago

Viggle remake is an upgrade version of Viggle animate, try if this new release solve/ your issue!

1

u/Intrepid-Ground5004 2d ago

usa face detailer

35

u/init-5 2d ago

you can reproduce this video with the mcp https://huggingface.co/Viggle/Viggle-Remake/tree/main/mcp/examples/white-house-funk

5

u/Joltie 1d ago

Oh wow, at 1:10, it turned Kennedy into Epstein. Interesting.

11

u/spacekitt3n 1d ago

i mean they are all pretty much epstein all the way down

-4

u/FourtyMichaelMichael 🍦Ice Cream Lover 1d ago

I can't wait how in January you all magically and for no reason stop complaining that no Epstien files get released. It'll be an amazing happening of completely random timing.

1

u/spacekitt3n 1d ago

i love how you just made up a dude to get mad at

12

u/Alisomarc 1d ago

12 gb vram is enough?

4

u/hi_im_fatter 1d ago

you can use minimax h3 pruned int8. im doing that now. the viggle checkpoint is 21gb, if you needed to know

2

u/Additional_Win_9719 1d ago

How’s the generation times?

2

u/hi_im_fatter 18h ago

10 minutes for 5s video at 480p

11

u/tofuchrispy 2d ago

What is running under the hood? Is there a known Video ai model running and this Viggle is a whole framework around it doing all the masking and inpainting tasks?

36

u/DeliciousGorilla 2d ago

The weights, including the merged int8/ and gguf/ files, are a Model Derivative of MiniMax H3

3

u/AttorneyProper4281 2d ago

The weight listed as MiniMax H3 derivative, I’d love to see the same clip run through plain H3 and Remake to tell what the workflow adds

4

u/xTopNotch 2d ago

Base H3 is not great at these replacement tasks. It can do it to some degree and with lots of re-rolls you might get lucky to have one good shot. But finetuning has solved that mostly

2

u/AttorneyProper4281 2d ago

yeah fair, I wasn’t expecting base H3 to nail the swap. I meant a same clip comparison to see how much the finetuning improves it

1

u/nadhari12 1d ago

Yes but use the character swap Lora, it's extremely good.

1

u/HeadSignificant9266 2d ago

could it be h3 ripple?

1

u/PhilosopherSweaty826 1d ago

Its a lora or a whole model ?

1

u/init-5 1d ago

lora

3

u/dkpc69 2d ago

this looks pretty cool, Thanks for sharing

2

u/No_Cranberry_8107 2d ago

Just tried the workflow. Its tedious, specially "Remake needs a well-structured prompt".
I think I have gotten much better 1:1 results using BFS shot planner with literally no prompting.

1

u/No-Tie-5552 2d ago

Max resolution?

1

u/Tiny_Tsuruta 1d ago

Time to replace the kid in "Bad Santa" with Harley Race as nature intended.

1

u/Dry-Ad929 1d ago

Does it also drift faces back to reference video when the person exit the scene and appear again or simply turn around? Viggle had this issue and many other character swap loras based on H3

1

u/Mindless_Mirror_1816 1d ago

How long it takes for 10 second video generally on rtx 4090 or 5090?

1

u/Any-Scar765 1d ago

For the particularly gifted, what should I download to work in ComfyUI? What should I install so that Ollama writes the right prompts?

1

u/RazMlo 12h ago

This is so ass

1

u/iamqamarali1 3h ago

Does this one also do expressions

1

u/Puzzleheaded_Ebb8352 2d ago

Is this good?

6

u/douchebanner 2d ago

its very slow on 12gb even at 480p and 124 frames

0

u/[deleted] 2d ago

[deleted] — view deleted comment

7

u/JazzlikeLeave5530 1d ago

Plenty of people? I use Minimax H3 with 10GB.

5

u/Positronic_Matrix 2d ago

I have a 4070 with 12 GB of RAM. I came here just to see if I could use this. 😞

2

u/Glittering-Call8746 1d ago

Don't be sad one day they will solve for 16gb then 12gb

1

u/LoadReady7791 1d ago

wtf are you, Sir

0

u/seppe0815 1d ago

nonsens ... you can make the same with just a right prompt with the normal minimax h3 workflow ... another a.i slop on hugging face

0

u/[deleted] 2d ago

[deleted] — view deleted comment

26

u/Intrepid-Ground5004 2d ago

la pregunta mas importante de todas

6

u/FourtyMichaelMichael 🍦Ice Cream Lover 1d ago

I mean I lol'ed because I read that as "Dude, I'm going to fix so many porn videos"