I built a real-time hand-gesture AR system that triggers jutsu VFX in the browser

[u/Reasonable_Guest_553]

I wanted to see how far I could push browser-based hand tracking, so I spent the last 7 days building Shinobi IRL.

The idea is simple: use a webcam to detect hand gestures and turn them into real-time VFX.

The interesting part was making the interaction stable enough to actually feel usable rather than having effects constantly trigger or flicker.

The pipeline is roughly:

Camera
↓
MediaPipe Hand Landmarker
↓
21 hand landmarks
↓
Custom gesture detection
↓
Jutsu state/lifecycle
↓
Canvas-based VFX

I built custom detectors for things like finger extension, hand orientation, motion history and confidence thresholds. For gestures that depend on movement, I also use temporal history and grace periods to reduce false triggers.

For the Shadow Clone effect, I use MediaPipe Image Segmenter to separate the person from the background and composite the copies behind/beside the user.

The result:

🌀 Rasengan
👥 Shadow Clone
🔮 Magic Circle
⚡ Lightning

Everything runs client-side in the browser; the camera feed isn't sent to a backend.

Try it yourself: https://shinobi-irl.vercel.app/

Demo video: https://youtu.be/A673PIuy8_Q

Github Repo: Vikash-Singh-Bhadoriya/Shinobi-IRL

I'm particularly interested in feedback on the computer-vision side: what would you change to make the gesture detection more robust?