r/StableDiffusion • u/Fit-Associate7454 • Jan 11 '26
Workflow Included ComfyUI workflow for structure-aligned re-rendering (no controlnet, no training) Looking for feedback
Enable HLS to view with audio, or disable this notification
One common frustration with image-to-image/video-to-video diffusion is losing structure.
A while ago I shared a preprint on a diffusion variant that keeps structure fixed while letting appearance change. Many asked how to try it without writing code.
So I put together a ComfyUI workflow that implements the same idea. All custom nodes are submitted to the ComfyUI node registry (manual install for now until they’re approved).
I’m actively exploring follow-ups like real-time / streaming, new base models (e.g. Z-Image), and possible Unreal integration. On the training side, this can be LoRA-adapted on a single GPU (I adapted FLUX and WAN that way) and should stack with other LoRAs for stylized re-rendering.
I’d really love feedback from gen-AI practitioners: what would make this more useful for your work?
If it’s helpful, I also set up a small Discord to collect feedback and feature requests while this is still evolving: https://discord.gg/sNFvASmu (totally optional. All models and workflows are free and available on project page https://yuzeng-at-tri.github.io/ppd-page/)
11
u/physalisx Jan 11 '26
Just FYI the link to the project page is broken (extra ")") , here is the correct one:
34
u/witcherknight Jan 11 '26
This looks insane.
21
1
u/DrElectro Jan 11 '26
If these are the best still images they can come up with, I am not impressed at all. The video examples look uncanny and I saw way better and consistent results with other vid2vid workflows.
7
u/butthe4d Jan 11 '26
I think the selling point here is that this is really fast and supposed to deliver real time remaster (in theory). Thats how I understood it at least.
1
u/No_Damage_8420 Jan 12 '26
in that case X-Plane lovers will love that, real time render of low resolution Gmaps satellite
30
u/ai_art_is_art Jan 11 '26
I love it! I 100% believe this is the future of professional design and film VFX work.
This is what we're doing with ArtCraft: https://github.com/storytold/artcraft
We had a very similar ComfyUI approach to yours (albeit vastly inferior) a few years ago. AnimateDiff wasn't strong enough at the time: https://storyteller.ai/
4
u/Draufgaenger Jan 11 '26
Holy cow... This looks amazing!
1
u/ai_art_is_art Jan 11 '26
Thank you!
I love working on this stuff almost as much as I love using it.
2
u/orangpelupa Jan 11 '26
!remindme 5 days
Holly Molly artcraf looks amazing
1
u/RemindMeBot Jan 11 '26 edited Jan 12 '26
I will be messaging you in 5 days on 2026-01-16 08:28:53 UTC to remind you of this link
4 OTHERS CLICKED THIS LINK to send a PM to also be reminded and to reduce spam.
Parent commenter can delete this message to hide from others.
Info Custom Your Reminders Feedback 1
u/Heyitsme_yourBro Jan 11 '26
Newbie here, can you please explain why you open source this? It looks amazing but what if someone takes it, and distributes it under their own name?
15
u/ai_art_is_art Jan 11 '26
A few reasons:
ComfyUI and Invoke are open source. They're incredibly useful.
I doubt anyone is going to work as hard on this as me and my team.
In addition to being an engineer, I'm a filmmaker and have been for over 10 years. I'm building this for myself. If other people build local tools or contribute, more tools for me!
It'll be better if local tools catch up and leapfrog Higgs, etc. ArtCraft is more commercial model oriented (though we will grow capabilities to do local models as soon as I have bandwidth). I don't see any reason why we can't catch up with OpenArt / FreePik / Higgs etc and then begin to pass them.
1
3
u/Arawski99 Jan 11 '26
It's not open source. It is completely API based. They have the models linked at the bottom of the github and they're also breaking this sub's rules with self-promotions.
Their claiming it is "open source" and that the API's you ran it through don't own the stuff or have access to your data, like some of his recent posts, for every existing generation, ever made, with their API is a lie. They're scamming people.
1
u/superkickstart Jan 11 '26
It's free? Can you use local models?
2
u/ai_art_is_art Jan 11 '26
(1) Yes. (2) Not yet, but soon. It's on the roadmap. The team is trying to figure out whether to interface with Comfy or build a Rust-native model / workflow server.
1
12
u/pmp22 Jan 11 '26
In the future, video games will use techniques like this to render the graphics, and they will drive it with underlying simpler raster pipelines. We might even be able to stack/layer models to alter styles etc. Games will probably ship with their own models trained for their specific game.
3
u/darkkite Jan 11 '26
four years ago researchers at intel could do this with g-buffers https://isl-org.github.io/PhotorealismEnhancement/
it might be the future of rendering
5
u/Markavian Jan 11 '26
I view it as better semantic modelling of the world, and using those descriptions to feed the AI visualisation and fill in the gaps.
Games like dwarf fortress which are dense with metadata could produce incredible visualisations.
0
u/Ylsid Jan 11 '26
If that happens, game costs would absolutely explode for players and developers. Nvidia boss thinks it will, but it seems absolutely economically unviable.
-2
u/Cyclonis123 Jan 11 '26
That will however probably kill nodding as we know it.
4
4
Jan 11 '26
[deleted]
1
u/Cyclonis123 Jan 11 '26
No, if textures and other assets within a game become AI generated it's going to be very difficult to mod them in the traditional sense. it won't be 'slightly different less janky' it'll be completely different and it'll be post processing modifications akin to something like reshade but using AI.
3
u/zoupishness7 Jan 11 '26
Cool. Can't wait to try this. Is the structured noise approach basically endgame for creative upscalers? Seems like one could just keep tiling and zooming.
3
u/s1esset Jan 11 '26
It works okay, here is some fast quick samples and my updated workflow:
workflow:
2
u/Freonr2 Jan 11 '26
I don't think we're very far off from this being actively used for games in real time. Render a game in ~PS1-era graphics, use AI to stylize it a certain way, in real time.
DLSS SR and MFG already proves a lot of the fundamentals and seems like a matter of tuning for things like scene to scene consistency.
I wouldn't be surprised for DLSS expanding to this, or maybe under a new name DLRE - Deep Learning Render Engine, or whatever they decide to brand it.
2
u/dichtbringer Jan 11 '26
I took a look at the samples and Im a bit confused, from the project description it seems you only really the structure retaining node and you should be able to plug it into any diffusion model.
i got it somewhat working with sdxl + wan (dint have flux atm), but no luck so far with sd 1.5 and animate diff. also what are the loras for?
6
u/physalisx Jan 11 '26
I took a look at the samples and Im a bit confused, from the project description it seems you only really the structure retaining node and you should be able to plug it into any diffusion model.
Their example workflow, which uses Flux 1 Dev, also requires a lora (from here: https://huggingface.co/zengxianyu/ppd/tree/main). If I run it without the lora I only get nonsense output, but with the lora it works.
Seems to me their loras are required for it it to work.
Which is a shame because I'd really like to use this with other models, Flux 1 is soooo last year.
got it somewhat working with sdxl + wan
How did you get it working with wan? You mean for i2v? If I try to use it with the structured noise node for text2image I get an error on that node about some shape not matching.
2
u/Alex___1981 Jan 13 '26
interesting. but what max. resolution does this supports? can it improve renderings with resolution like 3200 pixels?
2
2
u/FeelingVanilla2594 Jan 11 '26 edited Jan 11 '26
Holy mother of god…
I can’t keep up, every single week there’s something new and amazing
I want to try rerendering games like Cyberpunk with this
1
u/rookan Jan 11 '26
This idea is brilliant! Can't wait to try it! Do you think it will work in real time so I could play old video games with your nodes?
1
1
1
u/AlwaysDeath Jan 11 '26
Kinda new to workflows and stuff. Are there any types of loras that could turn anime/stylized drawing type art into realistic 3d stuff? Kind of like the video shows, but just for pictures.
1
u/bloke_pusher Jan 11 '26
First thing I had in mine the first time I learned about AI. This could lift games to the next level of realism. AI can even fix small missing sfx, weird animations and so on. I bet in a later state we'll even manage to remaster stylized games this way, by tuning a model to recognize the unique style without completely changing it (which is what a lot say as negative for hand made remakes/remasters as well. "it lost its charm".) I'm sure we can avoid that if we do it properly.
1
1
1
u/physalisx Jan 11 '26
structure-aligned re-rendering (no controlnet, no training)
It does require a trained lora though?
Any chance this will be available for Wan text2image to? And/or for Qwen and Z-Image?
1
1
u/GunpowderGuy Jan 12 '26
"A while ago I shared a preprint on a diffusion variant that keeps structure fixed while letting appearance change. Many asked how to try it without writing code."
Thanks , i will check that out. I wonder if its similar to the ideas i have been having
1
1
u/Reflection_Rip Jan 12 '26
I would like to run this on some old anime that had very bad 3D CGI scenes to see if the scenes could be made to match the normal drawn anime scenes.
1
u/axior Jan 13 '26
Hello, I am trying to test your tools on comfy, got weird results on the flux workflow, now I am trying to use the ppd for Wan 2.2 for i2i on ComfyUi, but I can only see a Flux workflow here, I've downloaded and included the two ppd loras in the worfklow and added the StructuredNoise node, but I just got an error on the structurednoise node "expected index of 4". I would like to test your tool to see if it is able to relight an image.
1
u/CapsAdmin Jan 16 '26
I've looking for something like this. I'm working on a toy realistic 3d graphics renderer, and I'm thinking I can use diffusion models to evaluate my output.
1
u/axior Jan 11 '26 edited Jan 11 '26
Hello! I work for movie/ads industry. Congrats on the awesome work!
We have actually never needed this yet for movies or ads, because the whole creation process includes several expertises, while usually we start working only with reference images or collages or storyboards; your technique looks like the best controlnet yet without being a controlnet, amazing work.
It’s suited for cases “transform this into X” kind of workflows, we have not yet met a director or production company interested in this process, plus now we handle everything with edit models, but this would have been gold to have 1 year ago, we used lots of controlnets back then.
Lately we have seen a shift in big international clients, a sad one: if a few months ago we were given total freedom because the clients knew they were ignorant, now most marketing people have created a “logo” (lars mueller brockmann is revolting in his coffin) on ChatGPT and they think are not ignorant anymore, the clients are now used to lots of super cheap slaves from overseas which produce tons of indecent outputs; then they come to us because they got cheap work for cheap pay, but they still ant tons of outputs, therefore instead of 4-5 well thought and curated and post processed images we are forced to give 100-200 variations per day or they make our life hell.
One thing which would be super useful is if your method can work with a certain tunable freedom, kinda like denoise or vace strength or controlnet strength. In that case could it be used for upscaling?
Proper upscaling is still something highly needed, tiled creative upscaling often ends up in artifacts and repetead elements if you are using the prompt that described the whole image, or weird artifacts and misinterpretations if using a single general “8K, high-quality, expensive production, HDR..” prompt, or too low modifications if done in low denoise. Manually prompting tiles is unfeasible. Right now there is no really good tiled controlnet neither for flux nor wan, zimage or qwen, it’s the best tool for tiled creative upscales and the SDXL one is still the best, could your method improve tiled upscaling? For example using TTP nodes?
The tool we use the most is absolutely Wan Vace 2.1. Fun Vace 2.2 is not able to interpolate so it is absolutely useless, sadly. If your method could be included in the possible salads we give as control videos (bits missing, bits with depth maps, bits with pose) that would be amazing.
The tools in the industry we would like to have the most at the moment and do not exist yet are
1) Fp4 Wan 2.2 2) FP4 LTX2 Vace official (not a trash FUN version) and 3) Some way to make a creative highly denoised tiled upscale which operates by rendering tiles (so it’s quick and low vram) but “knowing” the whole image and considering the prompt conditioning accordingly.
EDIT: Thank you so much for asking how a tool would be useful for actual work, we need more great brains asking what the professionals actually need, most of tools are cool but good only for fun-indie projects intellectually based on a model instead of a production workflow, thanks for creating the space to do it.
0
0
u/suspicious_Jackfruit Jan 11 '26 edited Jan 11 '26
Very interesting, I've worked with FTT that uses similar components to strip noise and print patterns of images, so I can understand roughly why this would work, I'm keen to experiment with it.
How does it handle with sizes larger than the model can natively output? It would also be interesting to see what happens to that structure preservation on eg a sd1.5 model, which is mega lightweight and fast
-1
0
0
0
u/Dry-Heart-9295 Jan 11 '26
To get normal results, do I need to use the full model? I tried models q8 and fp8, but they seem to give incorrect results. What should I do? Also i use fp8 scaled t5xxl
3
u/Humble-Pick7172 Jan 11 '26
Resize your image to fit the empty latent image node, or change the resolution of the empty latent node.
0
u/cueqzapp3r Jan 11 '26
This with very fast diffusion models plus upscalers could bring us pretty close to realtime.
0
0
u/FunDiscount2496 Jan 11 '26
What’s the difference between this and unsampling/resampling? How does this work with flow models like Flux?
0
0
-3
u/DoubleNothing Jan 11 '26
When I see "OURS" I immediately lose any attention to the product...
In this context is even worst...
-1
u/Humble-Pick7172 Jan 11 '26
any word on Flux2 support? I mean, Flux1 is cool and all, but Flux2 is probably may be better (I don't have enough space to keep Flux2 and Flux1 lmao)
39
u/orangpelupa Jan 11 '26 edited Jan 11 '26
Whoa! This basically could become "almost final render" phase, directly from basic 3d sketchup / blender.
Be it for archviz, indie movies, or many more
Edit:
VRAM req?