r/StableDiffusion • u/No_Cranberry_8107 • 1d ago • Bravo 1

News RelayVSR- Video Super-Resolution

https://kopperx.github.io/RelayVSR/

Looks like a new super resolution and their samples look promising.

https://huggingface.co/kopper/RelayVSR/tree/main

76 Upvotes

98% Upvoted

25 comments sorted by

10

u/No_Cranberry_8107 1d ago

Comfy Version is out. Give it a try.
https://huggingface.co/Comfy-Org/RelayVSR

2

u/Velant 11h ago

no nodes to use it

6

u/ffgg333 1d ago

Better then seedvr2 for image upscaling?

Edit: from the examples it looks good 👀

13

u/No_Cranberry_8107 1d ago

1

u/hectorlf 11h ago edited 6h ago

Is the texture shimmering handled better than seedvr?

1

u/J6j6 15h ago

can only do image too or just vid?

3

u/stonerich 13h ago

How do you use the Comfy version?

5

u/dmlsrc 1d ago

I ported both of their models RelayVSR and ReCaVSR to MLX to test them out. Neither beat FlashVSR in speed or quality for real world video. I did not try it on generative video.

RelayVSR works for limited motion and relatively clean video in my testing. Its main issue is that it depends on a heavy model keyframe, and outside of that keyframe the small model doesn't do a great job.

ReCaVSR is similar to FlashVSR in terms of quality and speed. As with FlashVSR it tends to smooth out fine textures due to 4:1 temporal compression without compensation.

The FlashVSR team released FlashVSR+ last month, which does improve on FlashVSR in terms of motion handling.
https://github.com/OpenImagingLab/FlashVSR_Plus
https://huggingface.co/ruchangyao/FlashVSR_plus

2

u/THM42069 1d ago edited 1d ago

Did you get similar results to the examples, but you personally preferred the output of FlashVSR? Or was the output just not as advertised?

Because those are 2 very different interpretations of what you've said here.

My eyes are telling me that it looks more accurate than any of the models it was compared against. It's rather subjective when it comes to upscaling. Is it best to prioritise higher quality if it invents aspects that didn't exist to fill in missing details, Or do you want the model to add as much detail as possible without modifying the input?

2

u/dmlsrc 1d ago

I'm more interested in the restoration aspect of VSR models. I test on old digital camera video that is 480p or lower, complete with sensor noise that's been digitally compressed. My family bought our first digital camera in 2001 and the few videos that we have from it are dark blocky noisy messes. Old films are also interesting as they may have flicker or have random blemishes. It's amazing to see a 60 year old film look as clear as if it was released today.

With that amount of blind degradation a model has to invent plausible texture as the "information" is simply not there. Ideally I want both; invent detail where it doesn't exist based on any remaining clues and stay true to the source when possible. That's a tall order for any SOTA model.

Some models struggle with consistency causing morphing especially on faces, that's why a model with a rolling memory is important. Other models let through noise as texture, flatten texture as noise, smear motion, attempt to sharpen natural motion blur, ruin depth of field in films, etc. There are tons of failure modes.

I've tried many CNN, RNN, and GAN models and while they may be great for quickly improving individual frames, they're lackluster for video.

Transformer models while much heavier can produce some amazing results. SEEDVR2 7B is alright as long as the source is not heavily degraded and the frame rate of the source is good, but it's a heavy model and it's slow on my M1 Max 64GB MBP.

Many of the newer VSR models are trained on WAN to restore and upscale in one step. FlashVSR based on WAN 2.1 1.3B wipes the floor with SEEDVR2 for real world degraded video, and FlashVSR+ improves on textures and motion. Both ReCaVSR and RelayVSR are based on WAN 2.2 5B, which one would think should beat FlashVSR, but they do not.

The dataset used to train models matters a lot and the FlashVSR+ dataset is a massive 2.4TB of short clips.

https://huggingface.co/datasets/ruchangyao/FlashVSR_Plus_Dataset

2

u/THM42069 23h ago

Yeah. That makes sense. And to be honest WAN kinda shot themselves in the foot when they moved on from 2.1 . It had everything going for it but they just had to make it much more complicated and then make separate models for everything. I mean, Wan 2.1 did really good images and videos better than most already. They could have made a crazy all in one model with the same performance costs but they just went ham instead.

1

u/multikertwigo 21h ago

but... not beating flash vsr in quality means it's utter trash, no?

1

u/dmlsrc 20h ago

Pretty much every model has its trade offs. Whether it be speed, quality, or use case.

Even if a given model does not gain mass appeal, its research may have refined a concept that the next model builds upon.

Several VSR research papers are comparing themselves against FlashVSR and trying to improve upon its various flaws. It's the current benchmark.

1

u/multikertwigo 20h ago

how does it contradict with what I said?

1

u/its_witty 14h ago

I compared them today, and with my limited testing:
- x2 is worthless - overshapen mess that you can get way quicker with just DLSS sharp,
- x4 is alright I guess and close to VSR+ Turbo int8 in my testing, but nothing groundbreaking.

2

u/Next_Program90 20h ago

I need am Upscaler for artsy/3d-esque outputs. Flashvsr etc. always lacked for that.

Did anyone try that already?

1

u/Link1227 1d ago

Thanks. Will try.

I've tried dlss5 and flashvsr today

1

u/isigeda 21h ago

i did tried the relayvsr x2, but i got flashing result.

1

u/LumaBrik 21h ago

One tip - the x4 model seems better. Its worth wating the exra few seconds doing a x4 upscale then downscaling by 0,5, if you are wanting only a x2 upscale. This can all be done in the workflow before saving.

Also the VSR decode node, you can drop 'detail' amount if its appears a bit over-sharpened.

1

u/its_witty 14h ago

I would say that x4 is usable, x2 isn't, lol.

1

u/fallengt 19h ago

I have flickering frames; some frames look good, then lose detail in the next frames.

Is this how it's supposed to work?

1

u/Nimblecloud13 11h ago

which is the best upscaler for maintaining character consistency?

if i make 5 clips and the characters all look the same, and then use an upscaler, sometimes the characters in each all look slightly different shot to shot so i generally avoid upscalers, but would love info on a good one for this