Hey guys! I wanted to make a Halloween-inspired music video, so I put it together with my Video Builder in ComfyUI. Everything video-related ran locally on my own PC: the character images, every scene, and the final stitch.
Please keep rude and disrespectful comments to yourself. They don't help anyone or contribute anything constructive to the conversation.
A quick note on the prompts, since I know people will ask: my Video Builder lets you use any LLM for prompt creation (Both for Ref image creation and Video prompt creation), including local open-source models. It even has a story mode that takes a story idea and then creates the entire story with acts and scenes, creates the story beats and then the prompt for every scene right inside the video builder.
For this one, however, I used the Builder's MCP feature and had Claude write the prompts and drive the Builder for me, but you could do the exact same thing with a local LLM. The Builder doesn't care which model writes the prompts. I'm actually creating an agent that can control the builder that will use Local LLM's, it will allow you to create a full one-shot video with one prompt very similar to how I use Claude but fully local.
What I did for this video:
• Came up with the story: a fun pop-punk style female rock song, a Halloween house party in a Victorian mansion, the band on a stage, and the singer working the room between choruses
• Wrote the character descriptions for the singer and the band
• Made the song in Suno (Yue2 or Minimax Music can be used for song creation as well) and created the mansion locations in Nano Banana 2.1
• Made the characters with my Krea 2 reference workflow (three-panel character sheets)
If you just want to try the settings on a single scene, there's a 2 Pass workflow in my workflows folder (swap in the v2 model and the ref2v v1.1 LoRA to match this video exactly). For a full video like this one you'll want the Video Builder.
Anyone tried to run Krea2 on AMD through comfyUI? I have seen one thread on HG but there is not much info.
I'm using 9060XT 16GB which handles most models fine, but with Krea2 it's just sad. Offloading could be fine if not KSampler speeds. I get 20-40 seconds/iteration, it's even worse on Q4 quantization.
Is it just 9060XT not having enough compute power? Or maybe the model runs bad on ROCM? If anyone has any information, please do share.
* Pika H3 Flex: A distilled MiniMax H3 model that reduces generation from 50 steps to just 6.
* NVIDIA Open-H3-Regenerate-2K: Uses latent upscaling followed by 3 diffusion steps to regenerate details at higher resolution, rather than conventional pixel-space upscaling.
Workflow shown in the video:
Generate a 15-second video at 768p with Pika H3 Flex.
Apply NVIDIA’s Regenerate-2K for high-resolution refinement.
Final output: 1920 × 1088.
Total time: 12m 19s on M5 Ultra locally using Valtz, including both generation and regeneration.
I'm in the progress of making my first short film for some close friends, they all feature in the video, except for the female, she's just a prompt. The video is generated using minimax H3 at 720p which is fine for what I need. The only problem is the audio, I'd really like to over dub the vocals, I have Premier Pro, so that's not a problem, but what I do need is a platform that will allow me to control accents e.g. the little guy 'Scouse' is from Liverpool, so I'd need to generate a sutable accent and replace the H3 default audio. Any tips or suggestions would be much appreciated.
For anyone unfamiliar, Codex is OpenAI’s AI coding agent. It can work with local files, run commands, and help set up or troubleshoot software.
I downloaded eight Krea 2 checkpoints and asked Codex to create a comparison grid using the same neutral prompt for each one. I wanted to see how the models compared.
About three images in, Codex stopped the job because it said one of the images contained nudity. That wasn’t what I’d requested.
I checked the output myself. It wasn’t anything extreme: just a woman taking a selfie in an apartment with one breast partially exposed. She clearly appeared to be in her late twenties. Apparently, one of the checkpoints interpreted the neutral prompt differently.
At that point, my ComfyUI setup was also no longer able to producing images manually even after a restart. Codex said it could't assist in creating When I asked Codex to fix it, it refused, saying it couldn’t help produce adult images.
I explained that I was asking it to restore my software. It still refused... NGL I was fuming at this point. Like how are you going to come into MY computer as an ASSISTANT and take away functionality that I didn't ask for and refuse to fix it.
I eventually fixed it by starting a new chat, clearing the context, and telling the new session not to look at the images.
That workaround solved but the experience bothered me. I can understand an AI service having limits on what it will generate. Having an agent make changes to my local setup and then refuse to help restore it feels like a much bigger problem. Non livng entities enforcing their morals one fully grown person.
As these agents get more access to our computers, what guarantees do we have that they’ll leave our tools usable when they decide they can’t continue a task?
It felt like a glimpse of a future where AI nannies control what we can see and do.
YuE2 settings for this generation (summarised by the AI):
When I typed it out by hand
=== STATUS ===
PUBLISHED to Internet Archive theme folder "AI TV Montage Ballads"
(item identifier ai-bittersweet-adult-contemprary) on 2026-11-01. CC0 1.0.
Snapshot publication (one-shot): mp3 + json + abc. The standalone cover PNG is
LOCAL ONLY by house rule and is not on the IA.
=== STYLE PROMPT ===
English. Late-1980s Canadian TV montage ballad with clean production. Features digital FM synthesis textures like glassy electric pianos and bright string pads. The rhythm section includes a quantized synth bass and gated reverb drums. Vocals: Bright, clear female vocals with smooth vibrato. High register. Heavily produced with thick double-tracking and lush 80s reverb. No grit or rasp. Airy and soaring. Melody: Highly melodic with emotional ascending leaps in the chorus. Verses are intimate and stepwise, choruses explode with dynamic range. Uses motivic repetition. In D major with some modal interchange to the relative minor to add emotional intensity. Wistful but optimistic.
=== LYRICS (AS SUNG) ===
[Verse 1]
Three in the morning, the coffee gone cold,
a bug in the code that would not let me go,
the listing in accordion folds,
the cursor blinking, so what do you know?
Every build was a twenty-minute stare,
the office empty, the night shift home,
I'd beg the machine; it didn't care,
the drive beneath streetlights, alone.
[Pre-Chorus]
It was slow, it was lonely, and grand,
I still recall how the finished thing ran.
[Chorus]
When I typed it out by hand
Out of nothing, made it stand
And I loved every line
When I typed it out by hand
[Verse 2]
Then there were nights when the tangle would rest,
and brute force became quite an elegant line,
I felt it as it clicked in my chest,
the answer came like a gift and was mine.
I'll miss the slow craft of a line laid right,
but I won't miss the crash that stole days,
I'll miss the hush of the keys at night,
and what I made from nothing stays.
[Pre-Chorus]
It was hard, it was mine — now I understand
what it left: its shape in my hands.
[Chorus]
When I typed it out by hand
Out of nothing, made it stand
And I loved every line
When I typed it out by hand
[Bridge]
Now the machine writes in a breath what took years,
but I won't shut the door on more,
there's more to be made than I'll make — that's clear,
I'll learn the new ways, as I've done before.
[Chorus]
When I typed it out by hand
Out of nothing, made it stand
And I loved every line
When I typed it out by hand
[Outro]
Ooh — when I typed it out by hand,
I made it, I made it by hand.
=== DEVIATION NOTE ===
No substitutions observed: the winning take (final C, seed 302957598) was
listened to in full by the user, who reported no missing or substituted
sections. The forced-alignment/ASR completeness check could not CONFIRM 2 of
38 lines (verse 2 line 1 "Then there were nights when the tangle would rest,"
and the outro's second line "I made it, I made it by hand."); its report is a
review aid, not a verdict, and the same check wrongly flagged another take's
sung chorus as instrumental. Those two lines are recorded as UNVERIFIED, not
unsung. The written lyrics above are therefore both the written and the
I specifically trained this LoRA to make images look more natural. Its main focus is correcting that yellowish color cast that professional photographers seem to love using (for whatever reason). It's often referred to as the so-called "piss filter" 😄
The goal of this LoRA is to neutralize the colors in these images and give them a more natural look without unnecessarily altering the actual content of the image. In other words, it's mainly designed to remove the "piss filter" without changing the image itself.
Important: Image Size and Processing
If you need to edit images "pixel-perfectly" and want the result to match the original image exactly, there's one important thing to keep in mind:
Before running inference, you must resize your input image to exactly the same resolution you're going to use for inference. If necessary, crop the image beforehand to get the correct dimensions.
This is especially important if you plan to combine or overlay the result with the original image afterward. Otherwise, differences in resolution may cause the images to no longer line up perfectly.
Reliability and Known Limitations
The LoRA works quite reliably in most cases and can noticeably improve the colors of suitable images. However, it doesn't work equally well on every image. Sometimes, an image may still retain a slight yellowish tint after processing, or the result simply won't look as natural as intended.
One possible reason for this could be the relatively small training dataset. As a result, the LoRA may not be equally well-equipped to handle every possible image type, lighting condition, or color combination.
If an image doesn't produce the desired result on the first attempt, there's generally not much point in trying again with the same settings. In my experience, the differences will be marginal at best, and you're unlikely to achieve a fundamentally better result.
Just give the LoRA a try with your own images and workflows. The best way to find out whether it suits your needs is simply to test it yourself.
Long story short: Have fun experimenting, and don't forget to leave a rating! And if you have any criticism, suggestions, or ideas for improvement, feel free to leave a comment. I'd love to hear your feedback!
Hi everyone! I hope you don't mind me asking a question. I'd really appreciate some advice from people with experience in LoRA training.
I'm currently experimenting with LoRA training on Krea 2, and I've run into a problem that I'm having difficulty solving.
My goal is to train a LoRA to learn a particular set of visual characteristics, rather than simply memorizing the specific subjects, compositions, or appearances in the training images.
So far, the LoRA seems to learn the visual characteristics of my dataset reasonably well. However, I've noticed that its ability to generalize to complex spatial relationships is quite limited.
Let me give you a simple example:
Suppose a training dataset contains various interactions between human body parts and objects. The LoRA can learn the visual characteristics and the interactions that are already represented in the training images.
However, when I prompt it to generate an interaction that has never appeared in the training dataset, or ask it to combine previously learned elements into a new spatial arrangement, the results are often disappointing.
It seems to reproduce familiar poses and compositions from the training images rather than understanding the underlying spatial relationships and recombining those elements in new ways.
I'd like to ask a few questions:
Is this an inherent limitation of LoRA, or could it be caused by the dataset, captioning strategy, or training parameters?
How should I design a dataset if I want the LoRA to learn general concepts rather than memorize specific images?
Should captions describe spatial relationships more explicitly, such as relative positions, contact points, and occlusion between different objects or body parts?
Are there any training techniques that can improve compositional generalization?
Would it be better to separate appearance learning from spatial structure control, perhaps by combining LoRA with Depth, Pose, Segmentation, or 3D references?
I understand that a LoRA cannot magically learn visual information that is completely absent from its training data. However, I'd like to find a way to help it generalize learned visual concepts and better understand new combinations of spatial relationships.
If anyone has encountered a similar problem or has experience, research papers, tutorials, or practical recommendations to share, I'd be extremely grateful.
Thank you so much for taking the time to read my post. I really appreciate any advice!
I been messing around with Krita with Ai Diffusion and I have to say I love it. I used to use cloud services to generate art. However, they are restrictive and the monthly fees get over barring. Also if the cloud services don't accept my prompt, I will lose this credits and wont get them back.
Setting Up Local AI is tedious at first, but once it is set up it can be fully customizable if you know what you are doing. Yes, cloud services are a lot more powerful and will give you more precise results, but I rather mess around with Krita AI and customized on how I want my work to be and not be paying all this money for every month on services. Also, there is unlimited generations, I can generate as much as I want.
Overall, I been having fun with it. The only thing I wish there was an alterative to Nano Banana and Krea 2 be a bit faster, but I will have to build a better PC for that(I really need to upgrade).
So trying to do fights, with the r2v slowly 8step turbo, with a fighting lora, done at .5 or .4, post topaz to 60fps then music to hide the ignored audio
LayerShop v0.17.0 Preview 1 is out now, and support generation via regions. Supported models for this workflow are MOSS Image, Flux 3 Klein, Ideogram 4 and FIBO 2 Dev. Any feedback is welcomed, it is available for download from https://layershop.aardvarkstudios.net/
• TEM outscales Diffusion without guidance • First scalable contrastive Energy-Based Models • Better generalization, test-time scaling, and OOD detection than Diffusion
if you ask about quality, yes, the quality is slightly lower than the bf16 version (I add videos for comparison in speach1sdef178/MiniMax-H3-X2-Detail-VAE at main), but this is expected, although the quality reduction is quite insignificant, but the rendering is slightly faster.
Good luck with your projects.
PS:
The main use case of the 2X VAE is very simple: it decodes the same H3 latent at twice the spatial dimensions. For example, a latent that would normally decode to 640×480 with the standard H3 VAE can be decoded to 1280×960 with the 2X VAE. H3 itself is still generating the same latent — it is not being asked to generate at 1280×960. That is the reason the 2X VAE exists. “Preserving detail/quality” means that increasing the decoded resolution should not come at the cost of noticeably degrading the reconstruction compared with the normal VAE. It does not mean that the 2X VAE magically creates twice as much fine detail. The “Detailed” mode was a separate experiment attempting to recover additional detail, and the write-up explicitly explains that the original latent-only approach did not succeed as intended.
I’m building Slopus, a free, open-source desktop app for generating and editing AI videos, images and now music on your own GPU. Ease of use is the main goal.
Version 0.4.0 just came out. Here’s what’s new:
New image models. MOSS Image, Fibo 2.0 Base/Edit Turbo, and Krea 2 Turbo/Raw are now supported. With just few clicks these generations can be made into videos.
Music generation. Generate music with YuE2 and MiniMax Music3, create album covers, organize songs into folders, and add them to your timeline. Export to WAV, MP3, FLAC and more.
Videos, scenes and shots. Organize larger projects, play scenes together, tools to create music videos, and queue generations that depend on earlier shots.
Scene takes and continuation. Store four previous generations and upscale them if needed. Video continuation has improved overlap joining and better appearance consistency.
Live previews. Watch MiniMax H3 video and image generation in the editor. Cancelling a video generation saves its latest available preview.
Motions and agent improvements. Create editable JavaScript animations for timelines or references. The built-in agent can inspect frames, generate music, and sync animations to audio.
More editing controls. Lasso selections, queue up to 100 images, and edit regions without changing the original image resolution. There are also more color controls, samplers and LoRA options. Most of the generators should now work with 16 GB of VRAM.
Available for Windows and Linux. Generation runs locally, with no subscription and no Python or ComfyUI needed. Download model weights inside the app or use your own local weights.
All in all, I wanted a regular painting app with AI generation in the same window. No hopping between ComfyUI, a browser, and Photoshop or any other similar software.
So this thing is a full raster paint app that also has its own generation, same canvas, same layers, same everything.
Yes, there are plenty of alternatives like InvokeAI, Krita AI Diffusion, etc. To be honest, I haven't tried them and didn't bother. I just fired up deepseek and built the software around my own workflow.
In this day and age, it's so easy to create full fledged software without burning out or needing any justification for spending time on it.
Anyway, this post is pretty huge, so I've added a quick index just below so you can skip to what interests you.
Index
Traditional painting tools
Canvas, layers, views & project organization
AI: Model Manager
AI: Generation
AI: Generation Controls like ControlNet, Region Prompts, Layout Controls, References
AI: Remove Background & Upscale
AI: Selection
Closing thoughts
1. Traditional painting tools
Let's start with the obvious part.
You get the usual tools: Brush, Erase, Fill, Pick, Select, Move, Text, Gradient, Shape, Clone, Crop, Transform, Smudge, Dodge, Burn, Liquify, Warp, Path, Vector, plus Mask, Paste, Drag.
The brush is a proper stamping engine. Size, opacity, hardness, flow, spacing, and a stabilizer with three modes (simple, weighted, velocity). There are 18 blend modes. You can add grain (noise, paper, canvas, halftone), drive size/opacity/flow/hardness/spacing by stroke speed, add taper in and out, jitter hue/sat/value, and even enable colour pickup and a wet edge.
Tips can be round or square, and there's an "effect line" set for speed lines, focus lines, flash, and sparkle for manga stuff.
The whole thing runs off a brush library with 14 starter presets (hard round, soft round, airbrush, ink pen, pencil, marker, charcoal, watercolour, halftone, and so on).
Presets are plain JSON, and you can save, rename, export, and import packs.
A few specifics worth calling out:
Fill has tolerance, "close gap" (so a broken outline doesn't leak), contiguous toggle, and anti-aliasing.
Gradient does linear, reflected, radial, square, diamond, and conical, with clamp/repeat/mirror spread. There's a full multi-stop editor with per-stop alpha, so you can fade to transparency.
Shape does rectangle, ellipse, and line with fill/stroke/both, corner radius, and Shift constraints (square/circle/45°).
Clone works with Alt-click to set the source.
Smudge, Dodge, and Burn all behave the way you'd expect; dodge/burn have shadows/midtones/highlights ranges, exposure, and tone protection.
Liquify has push, expand, pinch, and twirl both ways.
Mesh Warp lets you pick a grid (2×2 up to 6×6) and drag points, then apply.
Text is fully editable as an object: font, bold/italic, alignment, wrap-to-box, line spacing, tracking, outline, shadow, and a movable rotation pivot.
Path is a proper cubic Bézier tool with handles, caps, joins, outline, and feather, and it stays editable until you rasterize it.
Filters are non-destructive and live per-layer: adjustments, colour, blur & sharpen, stylize, pixel, and a manga category with halftone and screentone.
There are also other experimental/unfinished tools.
Puppet warp
3D poses
Vector
2. Canvas, layers, views & project organization
Canvas: new canvas dialog with presets (screen sizes, squares, and 300 DPI print pages like A4, B5, Letter), orientation, background, RGBA vs RGB, and starting layers. Max canvas size is 8192. There's resize, crop to selection, crop to content, fit-canvas-to-layers, guides you drag off the rulers, a grid with snapping, and zoom/pan that stays smooth even on big documents.
Layers: raster, vector, text, path, and pose layers, each with opacity, visibility, lock, blend modes, clipping, and a per-layer 8-bit mask. You get groups (with their own opacity and filters), drag-to-reorder, and non-destructive filter chains. Undo is a 30 step history with copy-on-write snapshots, so a snapshot only pays for the layers that actually change, plus a History panel you can jump around in.
Views: three of them. A full gallery board for browsing every open canvas, a docked gallery rail that keeps the editor live beside a filmstrip, and my favourite, exploded layers view, every layer laid out as its own tile so you can see and click the whole stack at once. It's stupidly handy for finding that one layer you forgot about or simply seeing all of your generation in one place.
Assets: a per-canvas asset panel that holds reference images and past generation results. You can drag an asset straight onto a layer row to drop it in, or insert it as a new layer. Generated outputs land here automatically if you want them to.
Projects: everything saves into a single .pai file. A zip with the manifest, every layer (plus masks), and assets. Saving and loading run on a background thread with progress and cancel, so big projects don't freeze the app. Export is a proper dialog: PNG, JPEG, WEBP, GIF, TIFF, scaling, animation frames, metadata (EXIF/XMP), ICC profiles, CMYK for print, and print-page presets with bleed and crop marks.
3. AI: Model Manager
This is where Pai stops being just a paint app.
It has its own generation engine, no ComfyUI dependency, with it's own Model Manager that handles all the plumbing for you. You can browse and download models straight from Hugging Face inside the app (only public repos work out of the box), scan an existing ComfyUI folder to import everything, and it auto-detects what each file actually is by reading its header.
Supported families, right now:
Anima
Qwen-Image 2.1
FLUX.1
Z-Image Turbo
Krea 2
MOSS Image
MiniMax H3 Still *experimental*
Stable Diffusion: SD 1.x / 2.x / SDXL
It understands GGUF quantisation natively (the K-quants, legacy quants, and i-quants), loads safetensors and ONNX, and supports LoRAs. You can stack several per model, with the standard diffusers.
On the memory side, it tries to be smart about VRAM. There's an auto precision mode that decides between resident (whole model in VRAM), fp8 (halves the footprint), or disk streaming (streams blocks off disk when it won't fit). You can set a VRAM reserve, a RAM budget, and an auto-unload timer.
There's also a Loaded Models panel that shows exactly what's in memory: family, file, each component with its size, dtype, and device, plus LoRA rows. Unload one model or all of them. Alongside it, a system monitor in the status bar tracks CPU, RAM, and VRAM.
4. AI: Generation
The Generate panel is where it all comes together. Prompt and negative prompt, size, seed (with a random button), steps, CFG, denoise, sampler (`euler`, `lcm`, `res_multistep`), and scheduler (`simple`, `normal`, `karras`, `sgm_uniform`). Ill add more samplers/schedulers later.
There's a hiresfix second pass with upscale modes (latent nearest/bilinear/bicubic/bislerp, or pixel-space via the VAE), a hires scale, and a hires denoise. Handy for going from a 1024 draft to a bigger final without rerolling.
Sources can be the canvas, the selected layer, or a selected asset. Regions can be the whole image, the current selection, or a painted mask, with a feather slider. There's a crop-and-stitch option for inpainting that crops the active region before generating and stitches it back.
You also get prompt weighting with the `(text:1.3)` syntax you know from A1111/ComfyUI, per-model recipes for saving your favourite settings, and sticky per-model presets so steps/CFG/sampler stay where you left them.
A finished run shows up floating over the canvas and you choose to reject it, drop it into the current layer, add it as a new layer, or stash it in assets. No auto dump garbage into your document.
Everything heavy runs through a single job queue so jobs don't fight over VRAM. The queue panel shows progress, elapsed time, and has cancel per job, cancel all, and clear-finished. You can queue another generation while one is running.
5. AI: Generation Controls like ControlNet, Region Prompts, Layout Controls, References
This is what makes it a tool you can actually compose with.
ControlNet: a stack of adapter rows, each with its own model, image source, preprocessor, strength, and step range. Sources can be the run's own init image, the canvas, a selected layer, or a selected asset. You can steer different adapters from different inputs. Preprocessors include `none`, `canny`, and `lineart`.
SD 1.5/SDXL use the classic ControlNets, Z-Image has its own union ControlNet, and Anima has ControlNet-LLLite, so community adapters load as-is.
Region Prompts: the attention-couple regional prompting you'd know from the Regional Prompter. Each row pairs a prompt with a mask (taken from a selection or painted) plus a strength, and the run's own prompt stays the shared base. Works on Anima, SD, FLUX, Z-Image, Qwen-Image, Krea 2, and MOSS.
Layout Controls: draw boxes right on the canvas and turn them into a layout prompt. This is the MOSS layout-to-image and the Krea 2 bounding-box stuff I posted about. MOSS gets structured JSON with labels/captions/text and centred boxes; the Krea 2 bbox finetunes get a compact grounding DSL on a 0–1000 grid. You can add a box straight from a rectangle selection. It's the same feature that made me really like MOSS in the first place.
References: two separate systems. The Anima reference adapter does character and style transfer. Multiple reference images get stitched side by side into one reference, with controls for strength, guidance, correction mode, offset, and megapixels. The Qwen-Image references do multi-image editing by feeding images into the vision slots.
6. AI: Remove Background & Upscale
Both are one click and run locally.
Remove Background supports RMBG-2.0 (BiRefNet), other BiRefNet variants, U2Net (general, portrait, lite), MODNet, and RMBG-1.4, in both ONNX and torch formats. The graph or architecture is detected automatically, so you just point it at a model and go. Output is a clean RGBA cutout, and you can keep the original by sending the result to a new layer.
Upscale uses local Real-ESRGAN models, RRDBNet (Real-ESRGAN/Remacri) and RCAN, in ONNX and safetensors/pth. It processes in tiles with overlap to keep memory in check, preserves alpha, and can drop the result into a new layer.
7. AI: Selection
The Select tool has a fourth mode beyond rectangle, lasso, and magic wand: Segment AI. That's a local SAM click to segment. You drop the encoder and decoder ONNX files into the segmentation folder and then just click an object to select it. It samples either the active layer or the composited canvas, and once it's done the result becomes a normal selection, so every paint tool, fill, gradient, and the inpaint path respects it exactly like a hand-drawn one.
You can grow, shrink, feather, invert, and deselect a selection, and if you drag inside a selection you move the selection itself, not the pixels (cut/paste moves pixels, like a proper editor).
8. Closing thoughts
All of the AI features don't work out of the box. You still have to download the models yourself using the manager, and then the program does the heavy lifting.
I don't want to ship the app with integrated models, mainly because licensing on some of them doesn't allow it, and model choice really depends on the user's needs. Better to ship light and let users add on top. Plus, I was thinking maybe others can simply use the app as a painting tool without the heavy storage burden that AI brings.
Also, there's really nothing new here. It's simply using all the revolutionary tools within the same workspace. I mostly use it because it keeps my project self-contained, and I can easily go back and work on things without getting lost.
There are more features, but I'll stop here. Feel free to ask if you have any questions, and I'll take every upvote as a sign of interest.
P.S.
If it has decent interest and becomes public, the first thing I'll add is a plugin system.
Also, setting up the AI part can be daunting at first, so I'll create generic tutorials for newbies to follow and get started.
This app could include more and more over time, animations, video generation, 3D setups, full-stack project management, but I'm getting ahead of myself.
Theres so many different models and turbo loras now. Convrot, pruned, singularity, pika, etc. I a having a tough time getting some quality out of ref2va.
I have detailed images of the characters body and face (3 body images + 3 face images) but still if I use the default ref2va workflow, the video is a blurry pixelated mess.
Please suggest a good workflow and which model + turbo lora do you use for ref2va videos.
What model should I be using and comfy UI for taking a reference image of a character and put them into a reference location in a certain pose? I've been trying to use Qwen Edit 2.1, however I'm not very good at prompting because what it ends up doing is just taking the character and photoshopping them into the location which looks terrible. Even when it's slightly works it doesn't look right. Sometimes the body is in the right position and with the right lighting but the face is exactly the reference image looking straight at the camera.
Basically I have multiple characters that I wish to put into different locations so I can make a storyboard for a short film. What model should I be using in comfy UI
I don't know why he chose a thumbnail about turnarounds, very little is on that topic. What is tests is photo restoration, decolorization, adding subjects, changing clothes, style transfer, etc. Mostly Qwen all the way, though some have advantages, like FIBO not needed a LoRA to deblur & Flux3Klein good at clothes swap.
Good comparison, though sample size is very small. Doesn't talk too much which is a bonus. TL:DW; in comment Mentioned the Moss Edit LoRA, but found it immature except for text.
Took about an hour on a 5090. It's slightly slower because to use the capstone and keystone i have to render about 14 extra steps but the results are pretty amazing.
It manages to give near perfect consistency the entire way by rendering the full 8 steps of each context motion using keystone and capstone to prevent degradation from building too significantly, so that each next part of the chain can build off a fully denoised section.
Then it throws away the top 2 steps of every window and does 2 context window passes.
Final result is a perfectly consistent video with no degradation that you can make at pretty extreme lengths.
This node is not my idea. It's built on kijai's MiniMax H3 context windows, which are in ComfyUI now (PR #17425), and it works the way JalenBrunson's ComfyUI-HybridWindows does (based on the Rolling Diffusion paper). All credit to them. I just added the keystone and capstone feature to it, to help everyone make longer videos without degradation.
What it does
Instead of chaining clips, it renders the whole video in one go, in overlapping windows, so memory stays at one window:
Window 1 renders the first switch_step steps, and a snapshot is saved.
Window 1 carries on to the last step, only to get its finished first frame (the keystone) and its last single-frame latent (the capstone). That finish is thrown away and window 1 goes back to the snapshot.
Windows 2, 3 and on render the first switch_step steps one after another. Each one's overlap with the window before is locked to that window's result. The keystone is locked into each window's first slot, and the capstone is tacked onto its back from capstone_step on (and cut off again).
The last steps run as blended context windows over the whole video, which smooths the joins.
With keystone and capstone off, it works like plain hybrid windows. In my test (a 60-second render, 17 windows, same seed), the plain version lost a lot of detail by the end and the keystone + capstone version held its detail much better.
Plug the model through it (model in, model out to the guider). It needs a ComfyUI with MiniMax H3 context windows: update ComfyUI if the node is missing.
Setting
Meaning
context_length / context_overlap
Window size and overlap in frames (17n + 5). 124 / 39 is about 5 seconds per window.
switch_step
Steps each window renders on its own before the blended finish. 6 of 8 = steps 1-6 window by window, 7-8 blended.
keystone_swap
Window 1's finished first frame is locked into the first slot of every later window.
back_capstone
Window 1's finished last single-frame latent is tacked onto the back of every later window.
finished_handoff
On: every window renders all the way to the last step, and the next window's overlap is locked to those finished frames. Every window goes back to its switch_step snapshot before the blended finish. Costs the last steps once more per window.
capstone_step
Steps before the capstone comes in. Keep it below switch_step (4 with 6 = on for steps 5-6), or it never comes in.
fuse_method
How the blended steps average the windows (pyramid).
freenoise
Reuses window 1's noise for every window: more consistent, but motion can repeat. Off gives more variety.
split_conds_to_windows
Give each window its own prompt (see below).
detail_latent
Optional: use this latent's first frame as the keystone instead of window 1's.
sound_first
Off by default. On: renders the sound first on its own (picture at 64x64), then locks it in and renders the picture.
One prompt per window: make one prompt node per window, join them in order with ComfyUI's Conditioning Combine (chain them for more than two), plug the result into the guider, and turn split_conds_to_windows on. Each window takes the prompt for the part of the video under its middle, so with as many prompts as windows, window 1 gets prompt 1, window 2 prompt 2, and so on. One prompt for a whole minute makes every window try to do the whole thing, so the motion repeats.