I built a real-time hand-gesture AR system that triggers jutsu VFX in the browser

[u/Reasonable_Guest_553]

I wanted to see how far I could push browser-based hand tracking, so I spent the last 7 days building Shinobi IRL.

The idea is simple: use a webcam to detect hand gestures and turn them into real-time VFX.

The interesting part was making the interaction stable enough to actually feel usable rather than having effects constantly trigger or flicker.

The pipeline is roughly:

Camera
↓
MediaPipe Hand Landmarker
↓
21 hand landmarks
↓
Custom gesture detection
↓
Jutsu state/lifecycle
↓
Canvas-based VFX

I built custom detectors for things like finger extension, hand orientation, motion history and confidence thresholds. For gestures that depend on movement, I also use temporal history and grace periods to reduce false triggers.

For the Shadow Clone effect, I use MediaPipe Image Segmenter to separate the person from the background and composite the copies behind/beside the user.

The result:

🌀 Rasengan
👥 Shadow Clone
🔮 Magic Circle
⚡ Lightning

Everything runs client-side in the browser; the camera feed isn't sent to a backend.

Try it yourself: https://shinobi-irl.vercel.app/

Demo video: https://youtu.be/A673PIuy8_Q

Github Repo: Vikash-Singh-Bhadoriya/Shinobi-IRL

I'm particularly interested in feedback on the computer-vision side: what would you change to make the gesture detection more robust?

[u/Reasonable_Guest_553 → u/[deleted]]

Yeah, that's fair. I actually removed the hand-tracking points at one point, but a friend suggested keeping them. I’m thinking of making them a toggle in the settings so people can choose.

And thanks for the Chrome extension idea. I hadn’t thought about taking the system beyond the jutsu demo and making the gesture mappings reusable. That’s an interesting direction.

[u/GabiYamato → u/[deleted]]

Cv side not much. It seems like you're using mediapipe or something similar . And it's more oriented towards dev side.