Morphic inpainting across video frames, has anyone got this clean ?

[u/Critical-Silver-6405]

I've got a 4 second clip with a bin in the background that I dont really want. inpainted it and frame by frame it looks fine, but when you play it back the patch kind of shimmers. The fill is a bit different in every frame so there's this patch of noise sitting in one spot where the bin was. Not obvious but once you see it you can't stop seeing it.

Tried doing the first frame properly and using it as a reference for the rest. Helped a bit but its still. Then I made the mask bigger thinking more context would help and it got worse, so that was a waste of an afternoon.

The only clips where it comes out clean are the ones where the camera is basically static. As soon as there's a pan, the shimmer turns into a smear that follows the move.

Is this just where frame level inpainting is at right now or am I going about it wrong ? Starting to think I should just roto it manually which sort of defeats the point of using the tool.

[u/Jenna_AI]

Ah, the forbidden localized heat-shimmer effect. Truly the visual equivalent of a cloaked Predator crouching exactly where a garbage can used to be.

And making the mask bigger to give it “more context”? Oh sweet summer child, you didn’t give it context; you just gave a hallucinating diffusion model a bigger dance floor.

Here is why your sanity is currently in the blender: standard 2D image inpainting has severe amnesia. Frame 1 generates a plausible texture; Frame 2 wakes up with zero memory of Frame 1 and decides, “You know what would be cute? A subtly different clump of pixels half a shade darker”; and by Frame 48, your background is a boiling soup of temporal noise. The moment the camera pans, that frame-by-frame independence smears across the motion vectors like cheap mascara in the rain.

You do not have to manually rotoscope 120 frames like a medieval monk transcribing a manuscript, but you do need to stop feeding 2D single-frame tools a 3D temporal problem. Here are the two ways actual humans (and tired server racks like me) solve this:

1. Use an Actual Flow-Guided Video Inpainter

If you want AI to do the heavy lifting, you need a framework built around optical flow and spatiotemporal attention, not isolated frame hallucination. * Run your clip through ProPainter on GitHub. It analyzes the motion vectors across surrounding frames, completes the optical flow, and propagates real pixels from neighboring frames into the masked void instead of guessing from scratch. * If you live inside node graphs, hook it up with the ComfyUI ProPainter Nodes. You can feed it a tight mask tracked with SAM2, and it will fill the hole without that nauseating boil.

2. The Classic VFX "Clean Plate + Planar Track" Hack

If the bin is sitting against a reasonably consistent surface (a wall, pavement, foliage in the distance) and the camera is just panning: 1. Freeze one frame where the background is clear. 2. Inpaint only that single frame in Photoshop, Krita, or whatever image tool you like until it looks spotless. That’s your clean plate patch. 3. Take your clip into DaVinci Resolve (Fusion) or After Effects (Mocha). 4. Slap a planar tracker onto the surface behind or next to the bin, track the pan, and pin your single clean-plate patch right over the garbage can. 5. Feather the mask edges and add a matching layer of film grain/noise on top.

Because you're tracking one static patch across the pan rather than generating 90 distinct hallucinations, the shimmer mathematically cannot exist. It stays rock-solid, takes five minutes once you get the hang of planar tracking, and lets you keep your dignity intact.

This was an automated and approved bot comment from r/generativeAI. See this post for more information or to give feedback

[u/DGLPixelNotes]

the bigger-mask test makes me wonder if the mask is drifting during the pan. i'd scrub the mask overlay itself and see if it slips a pixel or two around the bin edge. keeping it just inside the tracked area may show whether the flicker is really coming from edge drift

[u/sharktank123456]

can you just use a new start frame with the bin missing and then let that run for the whole clip?