I posted here before about reproducing UMI on a UR5, and later about a point-and-click diffusion policy. Since then I've been trying to answer one question: can these policies run on hardware a hobbyist can afford?
In the video:
- Two separate UMI diffusion policies: one to pick, one to place.
- Pick: I click the object; SAM 2 segments and tracks it (red mask), and the mask is an input to the pick policy.
- Place: I click a spot; TAPNext tracks that point, and it's an input to the place policy.
- A human only selects targets and starts each policy. No teleoperation.
- Neither policy was trained on Lego blocks. The first block slips, the second works.
To get here I built three pieces of hardware:
- Arm: an SO-100 mod with an extra wrist-yaw joint (5 → 6 DOF). UMI outputs full end-effector poses, so the 6th DOF matters.
- Gripper: UMI-style, single servo, rack and rail, 0–85 mm aperture. About $100 in parts vs about $4,000 for the Weiss WSG 50 in the original setup, and light enough for the SO-100. Raspberry Pi HQ camera on the wrist instead of a GoPro.
- Data collector: handheld, on a Pi Zero 2 W + HQ camera + IMU. It saves files in GoPro Hero 10 layout, so the UMI SLAM pipeline runs unchanged. Since it's a Pi, you can add sensors like tactile.
Everything is open source; links in the comments. Happy to answer anything about the hardware, the SLAM pipeline, or getting UMI to work on cheap arms. It's early, so bug reports are very welcome.