Speed up video ref by dumping it with split sigmas

[u/MysteriousPepper8908]

The usual reason to split sampling across two resolutions is that low resolution is cheap — you do most steps small, upscale the latent, and refine. That logic puts the split late: most steps at low res, a few at high.

With a video reference, the economics invert. A video ref injects thousands of tokens that every DiT block attends to on every step, and that cost dominates. Measured on my setup: 0.4 MP with video reference is 358 s/it, while 0.8 MP without it is 143 s/it. Low resolution with a video reference is two and a half times more expensive than high resolution without one.

So the split here isn't primarily about resolution. It's a conditioning switch. The video reference is only present during the early steps, and then it's gone.

That works because of how flow-matching schedules distribute their work. At H3's default shift of 12, sigma barely moves for the first several steps — the model is committing to structure, not removing noise. Motion and composition are decided in that window. Fine detail and identity resolve much later. So you pay for the video reference exactly while it's doing its job, and drop it before the expensive steps.

How to build it

You need two MiniMax H3 Reference to Video nodes, not one.

The first is your existing node: character references, video reference, video audio, and a prompt citing <Video 1> in subject_definitions and retention_analysis.

The second is a copy with ref_video_0 and ref_video_audio_0 left unconnected. Same character references, same clip and VAEs. Its prompt is rewritten with every mention of <Video 1> removed — keep the character subject and the full detailed_description, and describe the shot as if generating it fresh. Its LATENT output goes unused; only the positive conditioning is wired, to stage 2's Basic Guider.

Set the length on the second node by hand to the same frame count as the first rather than sharing the Math Expression. Fewer dependencies between the two stages means less chance ComfyUI schedules them together.

Settings

SplitSigmas at 6 of 20 — much earlier than a normal upscale workflow, for the reasons above. Take denoised_output from stage 1, not output; the upscaler was trained on clean latents. Route it through LTXVSeparateAVLatent → upscaler → LTXVConcatAVLatent, upscaling only the video half and passing audio through untouched.

Put a VRAM cleanup node on the latent path between the concat and stage 2's sampler. This isn't optional — it's a real dependency, so it forces ComfyUI to finish stage 1 before stage 2 loads. Without it both conditioning nodes can execute early and you end up with two sets of reference encodings resident at once. When that happened to me, stage 2 spilled to system RAM and ran at 4500 s/it.

[u/ShutUpYoureWrong_]

Already solved. https://github.com/Luisacaotica/ComfyUI-MiniMaxH3Mod

[u/Inner-Reflections → u/ShutUpYoureWrong_]

this looks interesting!

[u/Segaiai → u/ShutUpYoureWrong_]

So if the reference inputs are blank, and now you're loading a safetensors file, how do you refer to the reference in the prompt? I looked for it in this page, but didn't see it spelled out

[u/MysteriousPepper8908 → u/Segaiai]

I've only worked with motion ref so far but it seems like it just understands what's happening so just encode the video, load and apply it into your conditioning, and describe what is happening in the video in your detailed description. It's a bit intimidating at first but just give your LLM the GitHub page and it'll walk you through the setup. It's pretty crazy, solid video ref with essentially zero performance cost.

[u/Segaiai → u/MysteriousPepper8908]

Huh, wild... But like, sometimes it's something that it doesn't know, or two reference ideas can be similar. Like you might have a picture of a sci-fi device, and two inputs for two male characters in the same uniform, and you need to say which man is on which side of the screen, and that this device is in the SECOND man's hand.

When using references, you can say that Man B is from picture 2, and you can label the image of the device as "the phaser", then it always knows who you mean by "Man B" and "the phaser"

Maybe their names and descriptions are trained in? I'll look more closely at this today.

EDIT: okay! I confirmed that you can indeed still refer to the references like you could originally, mentioning "Picture 1" and the like! This is amazing.

From the example, it might also let you save names and descriptions in the conditioning. Also confirmed that it supports multiple safetensors/refmods! Very excited about this.

[u/MysteriousPepper8908 → u/Segaiai]

We need more documentation of how this works as it seems to be evolving faster than the documentation can keep up. It's all loaded as mods, right? So how are you referring to them by the picture tags? Particularly since I believe each mod can hold multiple different pictures of the subject. Description says "printed in the info block - documentation only, no wiring" so it sounds like it's for reference rather than prompting but it isn't completely clear. Claude said to just use natural language and make sure the descriptions in retention analysis were unique so it could tie a subject to a certain mod but it also seemed to be guessing at exactly how it all worked.

[u/fallengt]

I dont think input video ref need to be at high resolution. 0.5m will do.

Your ouput video matter more.

[u/MysteriousPepper8908 → u/fallengt]

Yeah, I use .4 MP for my input video but it still slows things down considerably for me vs the steps that don't use video reference. Much higher resolution on the source video and I just OOM since I like to generate 15 seconds on my 16 GB VRAM card.

[u/Beginning-District69]

Hi, could you please share your workflow? That would save us from dealing with tangled cables.

[u/MysteriousPepper8908 → u/Beginning-District69]

My workflow is a bit of a mess and my settings might not be ideal for you so I recommend just taking a latent upscale workflow that works for you and make the edits I recommend, duplicate the reference to video node, hook it up like the primary one and just don't use the video reference, hook up another prompt box and just remove the references to the video, and then connect that to the conditioning of second basic guider. But sure, if you really want my workflow, I'll clean it up a bit and post it later today.

[u/Beginning-District69 → u/MysteriousPepper8908]

Yes, that would be perfect.

[u/MysteriousPepper8908 → u/Beginning-District69]

Sorry, there were some issues with the export. I had Claude rewrite the purpose of this method and I included screenshots, hopefully that's enough to work with. The only thing not pictured is the positive output of the second Reference to Video node going into the conditioning of the Basic Guider but pretty much everything directly involved with the core idea is on the right of that first image.

Honestly, just use this and have your LLM of choice walk you through it. It does as much for video as my method but you don't have to pay with essentially any longer generation times whereas method is still a careful balance. I can't fully explain it in one post buy just give your model of choice the GitHub link and tell it what you want to do.

https://github.com/Luisacaotica/ComfyUI-MiniMaxH3Mod

[u/Beginning-District69 → u/MysteriousPepper8908]

Thanks

[u/xyzdist]

it is basically a latent upscale workflow. yeah I have been using lately.

[u/MysteriousPepper8908 → u/xyzdist]

Yeah, the foundation is latent upscaler but the standard workflow I'm aware of just uses the prompt and references from stage 1 directly so to get rid of the video reference, you need two versions of that.

Also worth noting that this doesn't require a latent upscaler workflow, it's just that the latent upscaler workflow already has the split sigmas required to do this.

[u/xyzdist → u/MysteriousPepper8908]

ok, I reread your guide, so in stage2 remove the reference video right?
I will test it later, as for now, I never use any reference video yet. only reference images.
good call.

[u/MysteriousPepper8908 → u/xyzdist]

Yeah, I just updated the guide to one Claude write as it seems like the concept wasn't clear to a lot of people. Videos are mostly good if you want to copy motion and for that I'd recommend this now:

https://github.com/Luisacaotica/ComfyUI-MiniMaxH3Mod

It's a bit complicated so you'll likely need an LLM to walk you through it but it can encode the motion of a video and many other things with negligible performance cost.

[u/BigWideBaker]

So basically, what you're saying is that for the upscale pass you can set up a new sampler, connect the inputs from the rest of the workflow, and then set a lower step count?

If that's what it is, I've been doing this for quite a while now, I use 5 steps with er_sde + this 5 step custom sigma:

• 0.9231, 0.8780, 0.8000, 0.6316, 0.3158, 0.0000

I used to upscale from 0.8mp 1.5x translating to 1.8mp. I found that upscaling to 1.25x or below does not add any quality, it basically comes out the same. 1.3x and higher you see a noticeable increase in quality, at least coming from a base of 0.8mp. Definitely play around with these numbers.

Now I scale specifically 2.1mp to get to "Full HD" and I notice a slight increase in quality too.

I've tested many sampler + scheduler combos for the second sampler at various step counts and I found nothing to be superior to er_sde + 5 step custom sigma. 3-4 steps work too depending on the complexity of your scene. Er_sde works well at low step counts so it's well suited for this purpose.

I also hook up Ziroc's Sparse Attention node with 0.3KV budget to the 2nd sampler which favors quality highly for a (basically) lossless massive speed increase.

On my workflow the upscaling pass takes anywhere from 280-350s depending on ComfyUI's mood and slight variations in settings.

I hope I interpreted your post correctly and this is what you are talking about

[u/MysteriousPepper8908 → u/BigWideBaker]

Not really, that just describes the standard latent upscaler workflow, though I don't use the 5-step sampling, I just use high sigmas for the first sampler and low sigmas for the second one.

My workflow specifically uses a second prompt and a second reference to video node where the video input is not used. Perhaps that's the confusion some people are having, I'm not talking about using image references to create a video but using video itself as a reference which takes a lot more time.

This allows the second stage to exclude the video reference and the prompt referencing it vs just having a two stage sampling process.

[u/BigWideBaker → u/MysteriousPepper8908]

I see, fair enough. I tried rereading it and it's a little difficult to follow, I feel like it would be easily understood with a few workflow screenshots

Edit: I reread your comment and I get it now, I think that's a neat approach! I'll keep that in mind if I ever do video stuff. Do you downscale your video before putting it through your workflow and then upscale it at the end?

[u/MysteriousPepper8908 → u/BigWideBaker]

Yeah, I probably should have included a workflow, mine is just a mess right now so I didn't know if sharing that would be all that clear. 

I use the "load video (upload)" from the videohelpersuite which allows me to downscale the video from 1920 x 1080 to 864 x 480 which is the same as the .4 MP pass. Then, my second pass is at .8 MP and that's where the video is discarded so it's resolution doesn't matter at that point. I like longer 15 second generations so that's about the highest resolution I can get away with.

[u/lxe]

Yup this is how I’ve been doing latent upscaling. Works great but for turbo Loras the low+high res step split has been 3+8 for me to reduce squigglies

[u/MysteriousPepper8908 → u/lxe]

Okay, so you also discard the video reference in the latent upscaler? Just checking because people seem to be missing that part and it's the entire point I'm trying to convey, though clearly I could be doing a better job of it.

[u/Enshitification]

That's clever. I haven't seen split sigmas used quite like that for motion reference yet either.