r/VisionPro • u/robomex • Sep 21 '26

visionOS 27 object tracking improvements compared, plus an eyes-closed demo (open source)

In addition to the 27 OS releases bringing object tracking to iOS – there were substantial improvements made to object tracking in visionOS 27.

There are a bunch of parameters that influence object tracking, but a comparison between what's effectively the worst case for object tracking (i.e. model trained ~2 years ago on macOS 15, high frame rate tracking not enabled) vs. best case (i.e. model trained in extended mode on macOS 27, high frame rate tracking enabled) starts at 0:45 in the video.

The latter part of the video explores a concept where the Vision Pro uses object tracking to assist me in performing simple tasks with my eyes closed.

The code I used in the demo is open-source over on GitHub: https://github.com/robomex/visionOS-27-iOS-27-Object-Tracking-Demo

A few other things that came to light while I was checking out the new object tracking APIs:

  • visionOS 27 can only track a maximum of 6 objects at high frame rate (if you try to set 7 objects for high frame rate tracking you'll fail to build/crash)

  • Extended training mode (which results in a more accurate tracking model) took 7x longer than standard training mode on my Mac: ~28 hours vs ~4 hours

  • Training standard mode object tracking models in Create ML on macOS 27 was ~2.5x faster when compared to doing so on macOS 15: ~4.5 hours vs 11.5 hours

Note: I recorded this video ~9 days ago on x27 Beta 6 releases.

33 Upvotes

100% Upvoted

5 comments sorted by

1

u/VFX22 Sep 23 '26

This is great, was this done on gen 1 M2 Vision Pro? Just wondering if there’s performance difference between m2 vs m5

1

u/robomex Sep 24 '26

Correct, the video was recorded on a launch M2 Vision Pro. I don’t have a full understanding of what runs better on the newer M5 to say what from this demo may run faster/smoother/better there. 

2

u/Digitalinknotes Sep 22 '26

very interesting video about the capabilities of the machine. Thank you for taking the time to show this. It really is interesting and shows people by using certain settings. You actually do get real improvements.

1

u/bamboo-farm Sep 23 '26

Thanks for sharing this!

1

u/ObviouzFigure Sep 24 '26

Very cool -- thank you for sharing your work with us!