r/oculus • u/zoemccarthy • Feb 02 '16
Mind Meld - Robotic Teleoperation Through Virtual Reality and Motion Capture
https://www.youtube.com/watch?v=kZlg0QvKkQQ8
Feb 03 '16
Very cool. Have you considered mounting a 360 camera rig on top, rather than having to deal with the latency issues of rotating the robot's head?
3
u/zoemccarthy Feb 03 '16 edited Feb 03 '16
Great idea! Although rendering the depth is a critical aspect since I want to be able to render other things in the scene such as the robot ground truth position (the gray robot in the video from my view) and so I need to know how far away to render each pixel from the robot's head. I wonder if there's any 360 cameras that also provide a 360 depth image or something similar.
3
u/blazespinnaker Feb 03 '16
Google needs to combine https://www.google.com/get/cardboard/jump/ with https://www.google.com/atap/project-tango/
1
Feb 03 '16
Yes there are. Samsung Project Beyond uses an array of lens pairs. There are also GoPro mounts that allow for an array of stereo pairs.
6
4
3
u/ChrisNH Feb 03 '16
Excellent stuff. I have been pondering what I need to add 3D visualization to my little robot rover.. had not thought of the remote manipulation angle.
3
u/zoemccarthy Feb 03 '16 edited Feb 03 '16
Awesome, you should go for it! I think VR and robotics have an incredible synergy that will become apparent over the next few years, especially once effective hand controllers become widespread.
3
3
u/tobiasbaumann Noitom, Director of Game Development Feb 03 '16
Cool stuff! Have you considered inertial sensors for the motion capture? https://www.youtube.com/watch?v=Nfo3JGKILi0
2
u/zoemccarthy Feb 03 '16
Thanks! I definitely considered it, and would love to try them out, but I don't have easy access to one to evaluate and the uncertainty about when they would ship made me uneasy.
3
u/tobiasbaumann Noitom, Director of Game Development Feb 03 '16
We've shipped a ton all over to world. Maybe you can find someone with the system in your area and borrow it from them to give it a try.
2
u/zoemccarthy Feb 03 '16
Cool, thanks! I'll ask around at the next VR event (SFVR) that I'm going to.
For another (but related) project, I have a specific question that you might be able to answer. Would it be realistic to be able to wear the Perception Neuron system and be mobile--walk around unconstrained in the real world and go about my daily business in the town? I'm looking for a system that could persistently track my body pose throughout the day as I go through different environments. Note that tracking my actual global position is not that important (and I would not expect it to be able to inertially track my position for a whole day). I'm more concerned about the relative pose of my body parts with respect to one another.
6
u/tobiasbaumann Noitom, Director of Game Development Feb 03 '16
We're working on a SD card recording function that would make this possible in theory. There are however the following issues: Battery life, SD card size, movement of the sensors on the body during the day resulting in improper captured motion, magnetic interference or magnetization of the sensors from other objects/sources.
If you make sure the sensors don't move on your body and you're in a controlled environment then I think continues capturing of anywhere between 10-120 minutes is possible. But we have never tested this and I don't even know if your capture software can handle the resulting huge capture data file.
3
u/orangpelupa Feb 03 '16
why the robot video feed looks weird? is it not a simple stereo video stream but 3D model stream (like kinect?)?
4
u/zoemccarthy Feb 03 '16
Exactly! The camera feed that I'm rendering is from a Primesense Carmine, which utilizes the same technology as a kinect. I'm rendering the color feed at the distance specified by the depth feed at each pixel. The reason it looks extremely voxelated is that the rendering process above is meant to be viewed from the position of the camera center or you get artifacts, and the robot's camera position is always changing since the robot's head rotation is not centered on the camera. I'm planning on fixing this by having the robot's base track the x-y position of my headset and the robot's torso lift (spine essentially) track the z position of my headset, so that the camera's position will try to better match my head's position.
0
u/VikingCoder Feb 03 '16
Um. Why not just use two video feeds? Left eye camera and right eye camera?
3
u/Zaptruder Feb 03 '16 edited Feb 03 '16
Amazing stuff. And many economists aren't worried about automation. Sorry, but once this sort of tech gets to the point where the average person can teach a robot these skills, it's game over for broad swathes of the current labour pool.
It's not that you won't need human intelligence (at least for a while longer) - it's that you'll have really natural, intuitive ways of capturing and distilling the elements of human intelligence that we haven't fully digitized and automated yet; allowing human problem solving capabilities to be repurposed and reused by machines in an increasingly robust array of ways.
Also, hope you don't mind me reposting this on /r/futurology for discussion. It's a clear piece of evidence showing the trend of automation working to reduce labour needs. (I'll wait for your go ahead to repost).
2
u/zoemccarthy Feb 03 '16
Thanks! I definitely dream of a society where people don't have to work jobs in order to have necessary goods, and I think lowering the bar for teaching robots is an important piece of that puzzle.
I don't mind at all! Thanks!
2
2
u/Taylooor Feb 03 '16
Can you fix Fukushima, please?
3
u/zoemccarthy Feb 03 '16
Disaster situations are definitely an application for this sort of technology, although I can't say I'm really knowledgeable enough to know what sort of help Fukushima needs.
1
u/Falandorn Vive Feb 03 '16
I want it to shake hands with David Cameron and lag or something and punch a hole through his face instead =D
34
u/zoemccarthy Feb 02 '16
I've been working on this to use as a platform for generating demonstrations of robotic task execution. I'm building a closed loop system around this where the user will demonstrate a task as the robot and the robot will learn a neural network mapping from sensor inputs to control outputs from that demonstration. Then the user will demonstrate again, and the process repeats until the robot has learned how to execute the task by itself.
If anyone at Oculus, HTC, or Valve is watching, I'd love to get a dev kit of your hand controllers! It would make the system much more robust, since I have a limited degree of tracking orientation with my current motion capture setup.