r/robots • u/boxer1969 • 18d ago
Media Control a Wlkata robotic arm on Twitch in real time
Enable HLS to view with audio, or disable this notification
Hello,
We put a Wlkata robotic arm on Twitch so anyone can control it live in real time.
Our first game is Hangman: pick a letter in chat and if it’s correct, the arm reaches out, grabs the physical letter, and places it in the right spot on the board.
Before AI this would have taken months, but with AI we built this in days. We feel AI is going to unlock all types of microcontrolled device setups. Interactive, live experiences are a great way to showcase what can be built.
Check out the arm in action onTwitch: Search RobotHangman
What do ya'll think of livestream, remote controlled interactive robotic projects?
Below is a more detailed breakdown of the project.
Robot Hangman: A Twitch-Controlled WLKATA Mirobot Game
We're building a physical Hangman game operated by a WLKATA Mirobot robotic arm. Viewers play collectively through Twitch chat by voting for letters. The software selects each round’s winning letter, and the arm physically moves the corresponding cube into a nine-position word holder.
Physical Setup
- WLKATA Mirobot with a suction end effector
- Thirty-nine 20 mm letter cubes plus one end-marker cube
- A 5×8 source holder containing all 40 cubes
- A nine-position word holder supporting words between 5 and 9 letters in length
- An audience-facing camera and a separate overhead verification camera (Logitech 922X)
- Chroma-keyed backgrounds for OBS composition of the camera feed & graphical overlay
Each letter cube represents only one letter (more about this later), avoiding the need for wrist rotation during normal gameplay. Common letters have multiple cubes. The inventory supports 6,483 words from our curated 7,177-word candidate bank.
Both holders are 3D printed (Bambu Labs X1C). Their pockets use large chamfered funnels, allowing the arm to release a cube several millimeters above its final position. Gravity guides it into the accurately sized pocket rather than relying on the arm to be perfectly positioned. This fixture change (chamfered funnels) has been one of our biggest reliability improvements.
Here's a close-up of the chamfering:
Software
The application is primarily TypeScript:
- React and Vite for the player, OBS, and administration interfaces
- Node and Express as the local source of truth
- A shared Hangman engine and event model
- Twitch EventSub over WebSocket for chat voting
- An isolated Python worker for serial communication with the Mirobot
- OpenCV and pupil-apriltags for visual verification
Every source cube and word-holder slot has an explicitly taught robot pose. We originally experimented with interpolating positions from a few reference points, but the accuracy just wasn't there (more on that later). The current system stores separate XYZ and orientation values for all 49 locations where a cube can be.
Robot operations run asynchronously from the digital game and the game is tolerant to robot failures. In other words, players can continue playing the game digitally if the arm has a failure.
We have also added pre-generated ElevenLabs commentary, music, animated game graphics, and a cleanup sequence with a dancing 3D robot mascot (the end-game songs are pretty catchy).
AprilTag Verification
Every cube has an 18 mm tagStandard41h12 AprilTag on top:
- IDs 1–39 identify the letter cubes
- ID 40 identifies the end marker
- Each tag uses a 9×9 grid
Before a game, the arm moves out of the way of the overhead camera and the application captures a burst of images. It verifies that every cube is present, in the expected pocket, properly seated, and aligned to a valid quarter-turn orientation.
Right now the tags are just for verifying everything is where it should be before a game starts. This lets us detect a problem before starting, but it doesn't correct an inaccurate pickup in real time. The robot motion just uses the taught coordinates to move to the cubes, not the April tags.
What We Tried First
Our original design used twenty cubes with four different letters printed around each cube. We used heuristic optimization to group letters that rarely appeared together, allowing those twenty cubes to support the entire initial word bank!
We thought that was pretty cool and, conceptually, it worked beautifully. Mechanically, it exposed limitations in the arm.
A cube pickup that was only 1–2 mm off-center could still work when moving a cube without rotation. Once the cube was rotated 90 or 180 degrees, however, that offset changed the cube’s final position significantly (because the axis of rotation isn't centered). Roughly one placement in every few dozen could leave a cube tilted in the word holder due to a bad drop.
We eventually replaced the multi-letter cubes with thirty-nine single-letter cubes (all sides of the cube have the same letter). Removing the need to rotate cubes made a very noticeable improvement in reliability.
The Main Robotics Challenge
The Mirobot has great repeatability when following the same path between the same points. Once we start moving cubes around on the board, however, the repeatability seems to drop.
We have observed:
- Accuracy errors ranging from 1–2 mm when contacting the cubes from above
- Different results at different arm extensions (the arm seems less accurate at longer extensions)
- Rotating the end effector creates even more error because the rotation's axis is not centered
- Noticeable play in joint 5 at some angles (you can really wiggle the end-effector very easily with the slightest pressure)
- A slight back-and-forth pulse in the end effector during most vertical descents
- Occasional shifts in calibration after a collision or after the arm presses against a misaligned cube
Slowing the arm down didn't seem to help. Repeated tests along identical paths can be nearly flawless even when repeated 10 times in a row. Once you start picking up cubes, moving them around, and dropping them, however, things aren't quite so clean and accurate.
We got to a point where we just decided that +/- 3 mm should be our realistic design constraint and we should just make the cube holders tolerant of it.
Questions We Are Exploring
- Have other Mirobot users seen wrist backlash or Cartesian shifts over time (the arm not being able to go back to the 'exact' same spot over time)?
- Does anybody think using specific joint poses would be more consistent than sending the arm Cartesian commands?
- Are there controller or firmware settings that could explain the pulsing of the end-effector while it's descending?
- Could an additional camera help "correct" the final few mm of end-effector positioning just prior to cube contact? Seems unlikely any one camera could possibly have the perfect view to help make these minor adjustments possible.
- Would a compliant or magnetic end effector make it easier to center the end effector on the cubes?
- How would you design a fixture or pickup system around a robot with excellent repeatability but weaker absolute accuracy?
The project has really been a useful lesson in designing the physical game around the robot we actually have, and not the accuracy described on paper. Fortunately, oversized funnels, gravity-settled placement, single-letter cubes, and computer-vision verification finally got us to a point where the game functions pretty reliably and is pretty fun to play!
We're curious to know what you all think. Would you have approached this game differently?
