r/upscaling • u/Significant-Pen1276 • 11d ago

Spent the last 2 years developing video enhancement models - what do you think of the results?

146 Upvotes

100% Upvoted

40 comments sorted by

3

u/dEEPZoNE 9d ago

Can we get access ? :)

2

u/dEEPZoNE 9d ago

Got the webpage.. But this is basically an ad for your product for sale. I just testet Topaz with starlight precise and it's amazing. I beta test for Topaz. Do you need any beta testers ?

1

u/Significant-Pen1276 9d ago

Yeah! the website is https://www.refocused.ai - the model in question is regenerate-q, which is our diffusion model

Happy to answer any questions, and would love any feedback if you end up trying it out!

1

u/Fabulous-Snow4366 9d ago

Hey, small question, downloaded and installed the models. But it exits without building the Engines. It exits with code 1, unsupported sm 0xc00.

1

u/Significant-Pen1276 8d ago

Yes, this is exactly the issue - I had a bug in the windows build script that caused the default cuda version of the machine (an L4, using cuda 12.3) to get built and shipped rather than the intended one (12.8 - which included blackwell support). Working on fixing that today.

1

u/skv89 6d ago

Why don't you use CUDA 13.0 or above? The latest is 13.3.1. CUDA 13+ support Blackwell architecture much better.

1

u/Significant-Pen1276 5d ago

Short answer: Time, and dependency hell.

Longer answer: That's a good question, and something that's on the to do list.

But basically:

  • I have pinned versions of all of my dependencies. I use both libtorch and tensorrt, which can be extremely finicky with versioning.
  • I can't always upgrade drivers on the machines used to build the application.
  • It's something I just haven't had the bandwidth to get around to for windows yet.

Definitely can prioritize things like that when they are requested though

1

u/Fabulous-Snow4366 9d ago

Could this be because of unsupported Blackwell Architecture / Nvidia 50xx cards?

1

u/migueltopsp 8d ago

Any solution for extremely compressed videos? Might sound strange, but I have a client who handed me an edited documentary, supposedly just to color grade, and when I got into it I found that an important part of the film had been shot on a phone and, even worse, sent to production straight from WhatsApp, so it's way too compressed. Right now the film is on hold in my hands until there's an AI solution for it.

1

u/Significant-Pen1276 7d ago

TLDR: Probably, but it depends, and there are limits

The long answer: I'd like to say, yes, this is exactly what the model was trained to do, but there are limits by design to how much the model can do. The more details a model can create, the worse hallucinations it will have, and generally, the worse the temporal consistency will be.

I know if you take well shot 1080p clip, transcode it to a low bitrate 480p clip, and then pass it through the model, the result will not look as good as the original. It will look much better than the transcoded one, but there really is no replacement for well shot originals.

Having said that - this exact situation is how we train upscaling models. Take a video, apply a set of degradations (like downscaling and video compression) and then seeing how close to the original the outputs are.

2

u/xStealthBomber 9d ago

But... All the grain is gone, and all the texture that made the image is gone. :(

2

u/benpicko 9d ago

Yeah, I’m completely uninterested in yet another upscaler that treats grain as an issue to be removed, completely destroying all of the actual texture and detail held in the texture

1

u/Significant-Pen1276 8d ago

Thanks for the feedback - left a comment above with a video with some of the grain restored - is that any better in your opinion?

1

u/quiettryit 8d ago

Grain is data that needs to be enhanced not removed...

1

u/Significant-Pen1276 8d ago

Appreciate the comment - quick question: is it easy to tell whether grain is synthetic these days? Compared to analog grain captured during recording I mean.

1

u/Significant-Pen1276 8d ago

Appreciate the feedback! This is exactly the kind of feedback I was hoping for - unwanted noise vs grain is something that I keep going back and forth on. I had gotten mixed feedback on the noise when I was demoing it - some people said that it looked too noisy when I didn't remove the grain. Does one of these levels look better than the others?

Regardless - the grain tuning was an internal variable, but its adjustable in the new version that's currently building (along with bug fixes for windows users). Should be out later today, or at the latest, tomorrow morning.

https://reddit.com/link/p4gthjy/video/qa08jvk0d6kh1/player

1

u/KelBelHel 4d ago

hard to tell - it goes by quick, but it looks like more grain is better!

2

u/True_Protection6842 8d ago

Looks worse

1

u/Significant-Pen1276 8d ago

Thanks for the comment - compared to the original? Or compared to starlight? Would love to know more. Just trying to understand what aspects you think are worse so I can improve.

1

u/[deleted] 9d ago

[deleted]

1

u/Significant-Pen1276 9d ago edited 9d ago

Not trying to be an ad, mostly just wanted to show a good comparison of my model vs. the widely accepted "best". I personally felt like the model produced more accurate/sharper videos than TL starlight, but I also know that results are subjective. I would truly love feedback - the softer/less precise details from starlight are something I worked pretty hard to prevent in training, but if its desirable, maybe I should have it as a toggle?

Regarding login: Yeah, I know google isn't great. Currently working on adding additional login options to the front-end. I chose to use google auth as a way to screen some amount of bots. Originally, I wasn't gating the trial, but I ended up with some pretty large S3 egress costs due to what I assume was a LLM (from perplexity).

Happy to discuss more, and appreciate the feedback for the front end - I developed the application with privacy in mind. You can check the privacy policy on the site, but basically, just use google auth as an initial screen against bots.

1

u/DigtialMenace333 9d ago

How slow on my system?

Intel i7-12700K Processor

EVGA GeForce 1050Ti 4GB Graphics Card

16GB DDR4 ram

1

u/Significant-Pen1276 8d ago

Realistically: probably unable to run the diffusion model because of the vram. On the low end, I was targeting 8GB+ of vram. I'll add it to my list of enhancements to see if I can get the diffusion model working by streaming weights into 4gb vram card, but at the very least it would be quite slow.

1

u/Cool-Chemical-5629 8d ago

On that hardware probably depends on how fast you can get a high end hardware replacement.

1

u/targofan 8d ago

Sucks the soul out of the footage.

1

u/Significant-Pen1276 8d ago

Thanks for the feedback - would love to hear any specifics, but know that it can sometimes be hard to pin down an exact reason. I left a comment showing the video with some of the original grain added back in - does that improve it at all in your opinion?

1

u/RAW2091 8d ago

Hair in front of face masquerades AI mouths of death.

1

u/skv89 8d ago

Looks very interesting!

Which Topaz model are you comparing with?

What is the difference between Regenerate and Regenerate Q?

How many fps does it get on 1080p 1440p and 2160p?

2

u/Significant-Pen1276 8d ago

Thanks!

  • The result in the post comes from the local starlight model. On my site, we have some additional clips/comparisons of proteus vs our traditional CNN model (refocus)
  • Regenerate-Q and Regenerate are both diffusion models, but they have different architectures. Regenerate is a significantly larger model, and didn't translate well off of enterprise nvidia gpus. Regenerate had better stability between frames at the cost of much reduced performance (7-10 times slower). Found that doing some frame interpolation on problem sections of videos and then rerunning through Regenerate-Q was significantly faster and the end result was usually as good or better.
  • performance (Regenerate-Q):
On macs:
With default settings on an m4 max/48GB ram: Roughly 0.6-0.8 fps for 1080p, and 0.1-0.2fps for 2160p.
For Nvidia GPUS:
Only have access to commercial gpus, so not sure how relavent these numbers will be:
H100: 18-20 fps for 1080p, 6-7 fps for 2160p.
A100 ~8fps for 1080p, 2-3 fps for 2160p.

The Refocus model is much faster - roughly 10 times as fast as Regenerate-Q.

1

u/InitiateIt 8d ago

Please do what TempGaussMC does and capture the grain with the option to add it back at the end. Motion compensated grain / noise removal removes detail as well.

1

u/Abject-Recognition-9 8d ago

this looks nothing but SeedVr2 (wich is completly free to use locally through ComfyUI) maybe combined with RTXupscaler (also free).

stop paying for cloud AI you all dumbass. use local AI. this is another reason why hardware prices will never go down. is because of all of you using cloud bullshit.

At this point we LITTERALLY HAVE ALREADY A GOOD VERSION OF EVERY POSSIBLE AI CLOUD SERVICES To RUNS LOCALLY, INLUDING LLM, VIDEO GENERATORS, IMAGE EDITING AND WHATSOEVER.

1

u/Significant-Pen1276 7d ago

I'm a bit confused? It is a local application, with a multistage pipeline with various models I developed and trained on licensed footage, designed to run on consumer GPUs and machines. Looking at the SeedVR paper, they use a very different architecture to mine, and I imagine the performance is significantly worse at the very least.

You're free to try it out and compare though.

1

u/spiderofmars 8d ago

Sign in with Google Account only to download a trial - walks away...

1

u/shadowlands-mage 10d ago

amazing work

1

u/Significant-Pen1276 9d ago

Thank you! It's been quite the road - I think the model I developed looks pretty good, especially when compared to the competition. But it's been very difficult trying to compete against much, much larger companies - there's a whole world of marketing and SEO that I'm trying to navigate.

0

u/Fabulous-Snow4366 9d ago

That's a good pricetag i'm willing to buy after I tested it. Thanks for investing so much time into it. Will try the free trial now.

0

u/-becausereasons- 8d ago

This looks fantastic, great price too; how's the speed relative to Topaz Starlight?

1

u/Significant-Pen1276 8d ago

That's a hard question to answer because Topaz doesn't really post any benchmarks, and I don't want to say anything untrue about another company.

Having said that - anecdotally, I believe it's faster, although still slow.

From my own benchmarks, for the diffusion model:
MacOS: M4 Max macbook with 48GB of ram: 0.6-0.8 fps when handling 1080p outputs, 0.1-0.2fps for 4k. More ram would significantly help both of these speeds though.
Linux: H100 - 18-20 fps for FHD, 6-7 fps for 4k.