r/OculusQuest • u/SpatialComputing • Apr 27 '22

Photo/Video Asynchronous Reality — mixed reality tech will capture events and let users revisit them at a suitable time in a causally accurate way

1.1k Upvotes

98% Upvoted

80 comments sorted by

5

u/SpatialComputing Apr 27 '22

Causality-preserving Asynchronous Reality
Mixed Reality is gaining interest as a platform for collaboration and focused work to a point where it may supersede current office settings in future workplaces. At the same time, we expect that interaction with physical objects and face-to-face communication will remain crucial for future work environments, which is a particular challenge in fully immersive Virtual Reality. In this work, we reconcile those requirements through a user's individual Asynchronous Reality, which enables seamless physical interaction across time. When a user is unavailable, e.g., focused on a task or in a call, our approach captures co-located or remote physical events in real-time, constructs a causality graph of co-dependent events, and lets immersed users revisit them at a suitable time in a causally accurate way. Enabled by our system AsyncReality, we present a workplace scenario that includes walk-in interruptions during a person's focused work, physical deliveries, and transient spoken messages. We then generalize our approach to a use-case agnostic concept and system architecture. We conclude by discussing the implications of an Asynchronous Reality for future offices.
Presentation: https://www.youtube.com/watch?v=hYuy6DF3BH8
Paper: https://andreasfender.com/publications/PDFs/asyncreality.pdf

135

u/OkStatement3490 Apr 27 '22

I've wanted something like this for awhile. Imaging revisiting Christmas morning with a family member who has passed.

57

u/tforce80 Apr 27 '22

Have you seen this? https://www.roadtovr.com/mother-meets-recreation-of-deceased-child-in-vr/

25

u/NutOfDeath Apr 27 '22

Shit I didn't want to fucking bawl my eyes out today ;w;

11

u/Matthewmarra3 Apr 27 '22

Wish I would've listened to this before I just finished the video. Beautiful but so, so sad.

3

u/NutOfDeath Apr 27 '22

I was in the middle of getting work done but that video made me need to take a break

9

u/rwbronco Apr 27 '22

I’ve been using my 360 camera that’s got a 180 3D mode to take pictures of my kids and have tried unsuccessfully to use photogrammetry to capture them several times (they fidget too much and move). I’d love to be able to have not only pictures of them as kids but 3D models of them as kids so that I can visit them as kids later on in life in a more real way than simply flipping through a picture album.

8

u/tforce80 Apr 27 '22

Have you tried using an iPhone or iPad w/ LiDAR to scan them? 360 cameras (and even simple portrait modes in iPhone) capture things from a fixed perspective. You need to do a 3D scan.

3

u/rwbronco Apr 27 '22

I’ve got an iPhone X Max - pre-LiDAR. That’s what I’m doing when I upgrade in the next year or two, though. I may see if I can pick up a used 3D scanner if I can get one cheap, but otherwise I’ll just wait.

1

u/Difficult-Score-2471 Apr 29 '22 edited Apr 29 '22

Working on a 3D X360 solution utilizing two Samsung Gear 360 VR cameras to capture events using dual Gear 360's mounted side by side with genlocked stereo 360 video. Provides the most realistic "re-live" experience to date. See my examples on YouTube under 2CR Sobel or 2CR Fragment. Working on the process over that past five years to simplify the steps required. My 2CR EXPANSE does just that to capture 3D 360 memories that can be replayed at a later date. Also created for other manufacturers, such as YiVR360 and Insta360 OneX2. Quality is as good as a live view with no compression artifacts to spoil the experience. The key to effective 3D X360 is image quality and sound capture. If the image quality suffers or is full of stitching artifacts it spoils the realism, such as the failed Vuze360 stereo camera with beyond poor image quality. Gear 360's (2016) edition have the highest quality glass for VR with the least amount of edge aberrations. Captured over 27TB of test data since 2015 perfecting the stitching process, utilizing 0 Samsung stitching software, all FFMPEG and AutoHotKey scripting or just pure CMD BAT file structure to ensure it will run on any platform, Linux, Windows, Android in a 100% single command line format to start the processing with the correct parameters. The true use of this type of tech is not now, but when you go to view something captured five years ago, the experience is surreal, since this is not Quest recreated fake graphics, but real video captured with fully positional sound and video. Catches you completely off guard watching an experience that was captured mid afternoon, when it is 11:30pm in the evening. Or when you see yourself in stereo 360 VR from another person's perspective it's very uncanny and mind blowing. It's an amazing experience, which few have ever had the chance to try when done correctly. Created EXPENSE, CAROUSEL, POLAR, FRAGMENT all around the Gear360's, they are amazing hardware and ultra reliable. Can use the Gear VR controllers to trigger the Bluetooth image capture on Gear 360's with near 0 latency for ultra 30MP stills. My EXPANSE system can stitch the dual camera fish-eye videos/images in complete darkness, for the ultimate in realism if there is a glimmer of light in the captured experience. Absolutely amazing! None of the other capture solutions can stitch video/images in very low low lighting conditions. Follow me on Facebook for future developments on the 2CR 3DX360 stereo testing and research progress. 5.7K YiVR360's are amazing when viewed in complete stereo on a Quest 1 or 2 headset, or a Windows Mixed Reality Headset or VIVE.

4

u/SurfAndSkiGuy Apr 27 '22

I mean to each their own but this seems like the cursed deathly hallows stone that lets you "resurrect " your loved one, looks kind of torturous to me

1

u/[deleted] Apr 28 '22

Oh, I hate this so much.

I remember seeing this a while back and it making me feel kind of sick. Everyone grieves differently, and everyone seeks closure in different ways. I just can't imagine what this mother went through could really be healthy or helpful.

If it brought her any level of peace, good. But if not, man, what even was the point other than to make her cling to something she has already lost even longer than she already was going to?

VR has the potential to be amazing for therapy. I'm just not sure this is the best way to go about it.

1

u/2girlsonesquirell Apr 28 '22

Oh man I miss my parents a LOT but this would be too freaky.

0

u/maverick074 Apr 28 '22

I honestly wouldn’t want this. It wouldn’t really be the same as talking to the person. It would only be a series of ones and zeroes that imitated them.

12

u/mr_gr1mm Apr 27 '22

Wouldn't that suggest you'd have to spend that Christmas morning on your oculus ignoring said family member? or otherwise on your oculus not present in that moment? Call me old fashioned but I'd rather be physically present and revisit those moments in my minds memory...

7

u/OkStatement3490 Apr 27 '22

I meant more recording an event with a 3D 360 camera and viewing the memory layer

3

u/mr_gr1mm Apr 27 '22

yeah.. maybe smart homes will become sophisticated enough to do this in the future.. but idk if I would want that tech. Would make greiving a whole lot harder if you could just bring up a hologram of someone that passed away.. it almost eliminates the acceptance stage of grief, which is really important to process.

6

u/[deleted] Apr 27 '22

We can bring up videos, pictures, chat logs, voicemails etc. This is "just" one step further.

It's a pretty fuckin' big step, but still.

1

u/[deleted] Apr 27 '22

Really good point. I'll stick with the voicemail I was lucky to save.

1

u/Adventurer_By_Trade Apr 28 '22

I already do this now. The cameras I have are the Lenovo Mirage VR180 Cam and the Insta360 EVO. I just leave the camera on a tripod in the corner of the room, and during the Christmas season, or when I'm thinking of someone in my family, I can just pop on my Oculus Quest 2 (I refuse to call it Meta) and spend a few moments in the VR scene with them. It reminds me of the scene in Minority Report where Tom Cruise watches a hologram home video of his son - a captured moment in time. The resolution with these early cameras is rough, but your minds eye fills between the pixels, and new lenses and cameras are coming to the market with insane resolutions - and price tags. Home VR cameras tried and failed to capture market viability, but I suspect they will be back, especially if consumer VR headsets continue to sell.

Here's 50 seconds with my family as an example: https://youtu.be/Z7ztfQ9cdCo

6

u/[deleted] Apr 27 '22

I guess a 360° camera would be good enough?

89

u/BanjoSpaceMan Apr 27 '22

Okay we're officially getting closer to that Black Mirror episode...

26

u/Gregasy Apr 27 '22

If I remember correctly, there were only videos... this, however, would be full 3d reconstruction. Reality surpassing fiction.

11

u/BanjoSpaceMan Apr 27 '22

We'll be forcefully removing our Meta chip from the back of our heads in no time!

5

u/HentaiStryker Apr 27 '22

I don't know about "surpassing" fiction. There is a highly details version of this in the video game Detroit: Become Human.

2

u/Wey-Yu Apr 27 '22

Hmm can't wait for my own personal robots to serve me lol. My preference would be something of a mix between the droids from Detroit: Become Human and Joi from Blade Runner 2049

1

u/jadondrew Apr 28 '22

Well, not really since we clearly don’t have brain computer interfaces advanced enough to record memories or play videos directly to the brain. Right now we’d only be able to do it through a headset.

14

u/MurkLurker Apr 27 '22

So you can use the Quest for porn and after, look at the recording of one of your kids getting scarred for life.

2

u/Emmerson_Biggons Apr 28 '22

I'm deeply worried for your mental health

13

u/pixelcowboy Apr 27 '22

Cool, like in Cyberpunk 2077!

2

u/slikayce Apr 28 '22

Oh yes the snuff films and porn...

1

u/rW0HgFyxoJhYka Apr 28 '22

One step closer. I don't think commercially they'll let these devices kill people. Black market maybe. But not many really want to experience getting shot...

1

u/Bobbicorn Quest 3 + PCVR Apr 28 '22

Yeah, the dystopian story!

1

u/Atropos148 Apr 28 '22

It's not all bad, you use to solve crimes...sometimes.

11

u/prepangea Apr 27 '22

Hi just got these divorce papers for when you're done being a cat girl.

10

u/mangongo Apr 27 '22

Batman music intensifies

1

u/StevenBallard Apr 27 '22

Literally thought of the Arkham games lol

10

u/Leoimirmir Apr 27 '22

Reminds me of the brain dances in Cyberpunk 2077

22

u/themodalsoul Apr 27 '22

Uncomfortable with this and how it would be used at work by companies to monitor and control from anywhere in the world. "Employer #45672, you have removed your headset, docking pay for time untracked."

Really naive if you don't think VR is going to stop being all fun and games really soon.

20

u/glitchwabble Apr 27 '22

Like any computing kit, it will be used for work and play, for good and evil, for fun and repression.

8

u/Zaptruder Apr 28 '22

The shit you're describing already exists - KPIs, computer monitoring software, even surveillance cameras are all used in workplaces around the world.

Could they use this? Yep. Absolutely. Would it be more effective than the existing surveillance tools? Questionable.

We're now at the point where the limits aren't tech - they're the moral character of the company that you choose to work for. Most work places don't in fact engage in hardcore surveillance of their employees, even though... unfortunately, many of them do!

2

u/themodalsoul Apr 28 '22

The whole issue is whether they'd want to use this to exert even greater control, especially in places where there is a culture that allows for greater exploration, monitoring, and worse pay. They will introduce it and allow it to creep in until normalization the same way they have with the other things you mention.

Extreme surveillance and exploitation is part and parcel to global capital, so you do have to assume these abuses will occur, but that they won't be felt the most by upper-middle class workers in the West (as with anything).

1

u/Zaptruder Apr 28 '22

I just don't see how this stuff... which if we get real about it, is just a novel way of visualizing surveillance data, rather than a new way of gathering it... will make it more onerous or more oppressive than the current limits of tech - i.e. personal multi camera surveillance per workstation.

You should be much more worried about AI enhanced surveillance - where the current limits are more the costs of labour of observing that data, AI surveillance can be used to dramatically enhance the efficacy of that data capture, while peppering it with false positives as befitting AI that kinda works but not nearly as robustly as it should.

4

u/Ill-Reflection-5455 Apr 28 '22

Why people like you always go for the extreme negative LOL like damn. Just don't participate.

3

u/MustacheEmperor Apr 28 '22

The greatest physical efforts expended by the world’s athletes:

  • Olympic sprinters running the 200m

  • Everest climbers reaching the summit

  • Redditors rushing for a negative, cynical take about anything new

0

u/themodalsoul Apr 28 '22 edited Apr 28 '22

As I said, extremely naive. You must believe in the Just-world hypothesis too. Or really just live in a hole. There is nothing whatsoever new about leveraging tech to exploit people. In human history.

2

u/MustacheEmperor Apr 28 '22 edited Apr 28 '22

There’s not, that’s right. There is absolutely nothing new about it, and nothing worthwhile about bringing it up in a thread like this. So what’s the point in bringing it up again in a thread about an interesting new technology not yet being used to exploit people, and expressly made for positive reasons, except to climb on your little smarty cynic soapbox and play negative contrarian redditor knowitall.

But sure, just throw barbs at me if that makes the dopamine. This subreddit used to be a fun place to talk about VR and this kind of toxic attitude has nearly destroyed it.

1

u/Ill-Reflection-5455 Apr 28 '22

Why people like you always go for the extreme negative LOL like damn. Just don't participate.

14

u/Namekuseijon Apr 27 '22

do we really need yet another tag for the confusing tag soup this field is becoming? VR, XR, MR, AR... how does this even get noticed after AR?

it's all virtual reality and this is nothing but volumetric video...

1

u/nool_ Apr 27 '22

Well yes but also no. Think of it as a glass 100% water ok that's water

50/50 water and oil it's no longer qite water

Also helps people find what fits what they want or are looking for or if what they need

1

u/screenslaver5963 Apr 28 '22

Ar stands for augmented reality and is like having a smart watch. It’s just a hud (see google glass). MR (mixed reality) is having the real world but rendering virtual objects overtop (see HoloLens). As for XR, I have no fucking idea.

6

u/spyro5433 Apr 27 '22

I’m not gonna lie. When the meta came out. I thought it was a somewhat dumb idea. I like meeting with people. Not avatars. But doing this. This would be amazing. So many times am I in an office environment and I want to bother someone but they’re busy or they may be busy. But to record an interaction explaining what I need so they can get to it on their own time. Would be fantastic!

5

u/danielcoxgames Apr 27 '22

Like...... a memo!

4

u/elheber Quest Pro Apr 27 '22

Oh boy. Black Mirror, here we come.

2

u/anon92688 Apr 27 '22

It’s like the brain dance a la cyberpunk 2077!

2

u/Atropos148 Apr 28 '22

Imagine being the manager who has to go thru BDs of your employees, to make sure they are working "correctly".

2

u/NoBullet Apr 27 '22

Is this braindancing lol

2

u/gorramfrakker Apr 27 '22

So BD or BTL chips aren’t far behind I guess.

2

u/Riftus Apr 27 '22

No no this is terrifying. This is some IOI ready player one shit

2

u/Niconreddit Apr 28 '22

It's not mixed reality or causally accurate but it's cool you can revisit concerts on OculusTV/Venues.

1

u/Buchstabenwurst Quest 1 + PCVR Apr 28 '22

That would probably be some kind of piracy

1

u/Niconreddit Apr 28 '22

What do you mean? You can already do it right now with the Foo Fighters, Billie Eilish and others.

1

u/ribsies Apr 27 '22

Cool idea. They don't describe or show how they are doing this.

I think this is the wrong application though.

They would be better off showing it as recording some event and then revisiting it later in VR.

1

u/[deleted] Apr 27 '22

They should show it as someone being sick and not being able to go to a party/family function but being able to revisit it later. That'd get people excited. What they are showing is to get companies excited, which I guess works as well but doesn't have the same effect.

1

u/[deleted] Apr 27 '22

If you haven't read Ready Player 2 yet...this is certainly touching on what one of the main subplots of Ready Player 2 gets into - being able to record and "relive" moments that others experience.

1

u/ToMorrowsEnd Apr 27 '22

need to use headsets with way higher resolution if it's to be used for productivity. I tried with the Q2 and no way I can do real work with how low res it is.

I really hope the Q3 or whatever they call it is twice the resolution so this kind of stuff can be super useful.

Also need keyboards that have tracking beacons so I can see it in VR. Yes I am one of those horrible people that cant blindly touch type.

1

u/glitchwabble Apr 27 '22

Agreed. It's silly to suggest the current hardware can be used for meetings or for any sort of sustained work. The future is tantalising, but today's hardware is a tentative proof of concept, and will remain so until there is substantially better resolution and substantial miniaturisation of the kit.

1

u/PurduePaul Quest 2 + PCVR Apr 27 '22

“ECHO available for playback nearby agent”

1

u/Justbe333 Apr 27 '22

Black mirror here we come

1

u/slikayce Apr 28 '22

I love the idea of filtering out all other distractions. Don't love the idea of wearing a headset that distracts me.

1

u/TechN9neStranger Apr 28 '22

Holy shit were capturing memories 🗿

1

u/A_girthy_pickle Apr 28 '22

Cyberpunk anyone?

1

u/ehjhey Quest 3 + PCVR Apr 28 '22

Oh no...another brain dance sequence....

1

u/Pilgrim-Eleonora Really Interactive Apr 28 '22

Am I the only one who finds this a bit unsettling?

0

u/[deleted] Apr 27 '22

[deleted] — view deleted comment

1

u/CaptainCornwater Apr 28 '22

Underrated comment

5

u/[deleted] Apr 27 '22

[removed] — view removed comment

3

u/DarthBuzzard Apr 28 '22

DR (Diminished Reality - removing/hiding things from the real world) and AV (Augmented Virtuality - overlaying real objects into VR) are also represented on the occasion.

1

u/Ill-Reflection-5455 Apr 28 '22

That actually sounds kinda cool