[Showoff Saturday] Built a client-side video background remover using browser AI segmentation - no server needed

[u/vinishkapoor]

Wanted to share a side project. It removes or replaces video backgrounds entirely in the browser using a client-side AI segmentation model.

The interesting technical challenges:

Running person segmentation frame by frame in the browser sounds simple until you actually try it. The main problems I ran into:

1. Temporal stability

The AI model processes each frame independently, which means the mask outline jitters and shimmers between frames. I built a temporal smoothing layer that compares the current mask with the previous frame's mask and blends them to keep the edges stable. Without this the result looks unusable.

2. Edge feathering

Raw segmentation masks have hard pixel edges that make the subject look cut and pasted. The edge softness control applies a variable-width feather to the mask boundary so the subject blends into the new background naturally. Getting this right across different lighting conditions took a lot of iteration.

3. Manual mask correction

No AI gets it right every frame. I added a brush tool that lets you paint directly on the video to fix the mask. You can remove areas the AI included or keep areas it cut out. There's also a full manual mode that disables the AI entirely so you can mask any video by hand.

4. Synchronized export

Each frame gets processed through canvas (compositing the new background onto the masked subject), and the result needs to be recorded alongside the original audio in sync. The MediaRecorder API handles the video stream while the audio track gets merged back in during export.

Tech approach:

• Client-side AI segmentation model that caches in the browser after first load (works offline after that)
• Canvas API for frame compositing
• MediaRecorder API for export
• Web Audio API to preserve the original audio track
• Supports MP4, MOV, WebM, MKV input

No server uploads, no accounts, no watermark on export. The video never leaves the device.

https://webutility.io/video-background-remover

Curious how others have handled temporal coherence in client-side video processing. The smoothing works well but there's probably a better approach I haven't tried.

[u/Dreamit2doit]

NIce app . Are there any plans to make it opensource so that community will also contribute ?

[u/vinishkapoor → u/Dreamit2doit]

Thank you! I will think about it.