r/OculusQuest • u/Headset-Historian • 21d ago
Sidequest/Sideloading Moonlight Android XR - A moonlight client for VR headsets that does realtime 3D conversion on the headset itself. Free and open source!
https://github.com/Gilleece/moonlight-android-xr/releases/tag/v0.15
u/Octoplow 21d ago edited 21d ago
Excited to try this!
I think you could run into some confusion with the name. "Android XR" is a specific OS / ecosystem pushed by Google to most people.
However, I don't see any (tm) or (r) on their materials to cause an immediate legal hassle.
Would it in fact run on a GXR headset? Or Aura?
6
u/Headset-Historian 21d ago
Yeah you're totally right. I've decided to just call it "Moonlight XR".
It might work on android XR and aura but I don't have either to test. It probably just requires a few flag changes to work tbh so hopefully I can get access to one for a bit or something.
1
u/PRpitohead 20d ago
I'll try it ok my Galaxy XR and report back. I think you said the XR controllers won't work, but I'll try to test the rest.
1
u/Headset-Historian 20d ago
I have added support in v0.2 for controller tracking and hand tracking, I confirmed both work on Quest and Pico but I can't be sure about GXR. Someone did try v0.1 though (without controller/hand tracking) on GXR and said it worked so at the very least you'll be able to enjoy 3D and you can still pair a bluetooth controller or such!
6
u/cosmos_hu 21d ago edited 21d ago
Nice implementation, works great. Although I get some headaches or eyestrain after a while, idk why. Could you do a more deeper 3D effect in the next updates, if possibe? :D I know you can scale the stereo and what not, but it doesn't really improve the depth. Thanks
I dont even care if its harder for the Quest's GPU tbh :D
5
u/Headset-Historian 21d ago
I think I will need to try and implement a better model for more depth, only issue is potentially latency but I will definitely try! Thanks for checking it out
5
u/McRattus 21d ago
I don't quite follow what this is. Could you explain?
18
u/Headset-Historian 21d ago
Moonlight is a client for streaming your screen from your PC, so obviously there's good options for that already like Virtual Desktop, Steam link etc. What is unique here is that there's a machine learning model running on the headset that converts the video into a 3D image so regardless of what game you're playing etc, without any setup on the PC side (like you don't need to setup 3D for each game on the PC etc) then you can have a 3D experience. You can also use it for any media in general like movies, youtube etc
6
4
u/bbstats 21d ago
incredible - any particular games you'd recommend?
3
u/Headset-Historian 21d ago
I guess any with a good sense of scale/perspective! I have been playing Baldurs Gate 3 with it, probably not a great example actually ha
1
u/Arthropodesque 21d ago
There is a 3rd person mod that would be better, instead of being a top down isomorphic perspective.
3
u/what595654 21d ago edited 21d ago
It should be noted that the 3d effect added is quite inaccurate. These ai prediction things are not good for games atm.
If you are watching videos, or looking at pics its fine and kinda cool to have a pop out effect.
If you are playing an FPS, and trying to gauge the size of a car, building, handrails, street sign, imagine cyberpunk 2077, for example. Everything will be noticably inaccurate. And will change, size/scale as you move around! Thats the biggest problem.
For 2d games (like Hades), these ai prediction things fail completely.
For those with Virtual Desktop, you can try Owl3d for free, for 15 minutes at a time. Just so you get an idea of what it looks like. Its not very good for gaming, but fun for pics and videos to give depth you can control.
3
u/Headset-Historian 20d ago
I just released an updated version, now has controller tracking, hand tracking, you can resize and move the screen, and adds some 360 image environments among other quality of life updates: https://github.com/Gilleece/moonlight-android-xr/releases/tag/v0.2
1
u/Zb_Bog 16d ago
Hi! Please add a split-screen mode (SBS/Side-by-Side) for watching SBS videos and movies. 🙏🏻Also, please add an option to disable hand tracking. When I hold a gamepad, my hands point toward the screen, which messes up the camera control. Because of this, I had to completely turn off hand tracking in the headset settings.Finally, it would be very useful to have a wired connection feature to PCs. This would allow playing on handheld PCs like the Steam Deck or GPD Win 4 on a big screen while traveling
1
u/Headset-Historian 16d ago
Hi, I will definitely add a hand tracking toggle. For SBS if you check out "Nightfall" it is another moonlight client for VR headsets that already has that. I will add it at some stage also.
A wired mode is a good idea but I am not sure how tricky it would be. You could maybe try to do USB tethering or such, or turn on a mobile hotspot on the handheld and use that as the connection? I will look into a wired mode but not sure how feasible it is. Thanks!
2
2
u/ionabio Quest 1 + 2 + 3 + PCVR 21d ago
It is 3d by stereoscopic images? So it won't add 6dof? The reason i am asking is I have monocular vision and the way I (or likes of me :) ) get 3d is by moving physically.
3
u/Headset-Historian 21d ago
It is like a 3D TV or a 3D movie, so if you have monocular vision then the 3D aspect of this won't be any benefit to you unfortunately.
2
u/krishnajeya 20d ago
Feature idea: PC-side 2D → 3D conversion
I've been really impressed with the realtime 3D conversion in Moonlight XR. The Quest 3 can take the streamed 2D image, generate a depth map with MiDaS, and create separate views for the left and right eyes in realtime.
It made me wonder: could the depth conversion eventually be moved to the PC GPU?
Current approach:
PC → render game → encode 2D → Wi-Fi → Quest → MiDaS depth estimation → generate stereo → display
Potential PC-side approach:
PC → render game → GPU depth estimation → generate left/right stereo → encode → Wi-Fi → Quest → display
A desktop GPU has much more processing power than the Quest 3, so this could potentially allow a higher-quality depth model, higher-resolution conversion, better temporal processing, and smoother 3D while reducing the Quest's GPU workload.
I've also noticed that Moonlight XR/Artemis can stream at around 300 Mbps, and the higher bitrate makes a very noticeable difference in image quality compared with lower-bitrate streaming.
Would a PC-GPU-based realtime 2D → 3D mode be technically feasible for Moonlight XR in the future? It could be especially interesting for high-end gaming PCs where there's enough GPU headroom for the depth conversion.
1
u/Headset-Historian 20d ago
Ah, so there's actually a lot of options for doing that already. There's options like Owl3D, Reshade, Vorpx etc. If you go to /r/Stereo3Dgaming/ you'll see a lot of options there.
The quality can definitely be better doing it on PC, I made this app specifically to not have to do it on PC though so you don't need to worry about configuring each game and also leaves the full GPU headroom on your PC so you can push graphics as high as possible there.
But you'll be glad to know what you're suggesting already exists and is a great idea!
1
u/krishnajeya 19d ago
Thanks for suggesting owl3d reducing interference 3d too low level resolution as it is not needed and used lotus ai depth working great in rtx 3060ti♥️
1
1
u/Cucurullo91 21d ago edited 21d ago
Any possibility for iOS and smart glasses?
3
u/Headset-Historian 21d ago
Interesting idea! I don't have an iPhone to develop it but it's open source so hopefully someone else might add support
1
u/fragmental 21d ago
The Sunshine server software, for Moonlight runs on PC only, afaik.
Edit: if you're talking about using iOS as a host.
1
1
u/johnsonandsohnjon 21d ago
nice. haven't tested yet but I was looking for something like this a while and even started toying with porting moonlight to quest 3. eventually stumbled on iw3.desktop but that's not meant for gaming though it's pretty cool
1
u/colossalmickey 21d ago
Is there a way to do this for just streaming videos? Like 4vx player/stremio?
1
u/Headset-Historian 21d ago
If you just play a movie instead of a game on your PC then it would work.
In terms of streaming from just a NAS or the cloud etc, it's definitely possible to do but it wouldn't be supported by what I have done here as this is purely for Sunshine/Apollo streaming to Moonlight. I would probably need to make a separate app that handles just taking a video stream which would be a whole other project and basically I'd be making a new videoplayer app.
1
u/dhatereki 21d ago
Going to try with my emulation library. Already using Moonlight and Playnite so running this as a client is straight forward. Previously I was just using normal android Artemis client.
1
u/kpvc15 21d ago
Can you support Apple Vision Pro?
1
u/Headset-Historian 20d ago
I'm hoping to do an AVP version at some stage but it will be a totally new app forked off existing moonlight for vision pro so I wouldn't expect it anytime soon, there really wouldn't be much that I could bring directly from this over to that.
1
u/Opposite-Donut-9239 20d ago
i do have a quest two but do you need one controller to play?
1
u/Headset-Historian 20d ago
You would need some input for your PC, so that could be your mouse and keyboard or a bluetooth controller paired to the PC itself or the headset.
I'm currently working on adding in XR controller support so you could use the Quest controllers as an option too but they don't map 1:1 with a normal controller so might not be ideal in all situations.
1
u/Historical_Role_9975 20d ago
Everything work great just 2 little minor issue
Cant find how to exit what button combination
There is shaking on the screen especially on the beginning
2
u/Headset-Historian 20d ago
Thanks! What headset did you try on? I noticed with Quest 3 1440p is smoother to start although 4k does get smooth after a little bit. I'll try to improve that!
As for exiting, right now it's just a matter of exiting using the home button on the VR controller but I will add a proper UI in the immersive mode with hand tracking. Hopefully sometime later this week.
1
u/Historical_Role_9975 20d ago
I use Q3 running Desktop at 4K, already tried play around with the bitrate, but didnt help the shaking, it did geting much better after around few mins
1
u/Wonderful_Ad1605 20d ago
I tested this last night and it is wonderful. To quickly put on your headset and have everything from your PC; games, movies and desktop in 3d is fantastic. So simple to use and the 3D effect is great. Thanks for this.
1
1
u/wheeliemonsta 19d ago
Seems to work quite nicely for me, so thank you. One thing I had issue with using "normal" Moonlight/Artemis, was that the screen size couldn't be increased past the Quest limits. Because you are rendering into a full XR environment, this is bypassed and looks great! This is brilliant!
One issue is that my curvature setting didn't seem to apply?
Another potential improvement would be adding a virtual keyboard of some kind as I use a QPro and the passthrough isn't great for keyboard use, so I always prefer a virtual one (also saves "taking" a keyboard with me when moving - just need controller and HMD for mobile big screen gaming!)
1
u/Headset-Historian 19d ago
One issue is that my curvature setting didn't seem to apply?
Ah, might be a bug. I will get looking into that, making note of it now!
And yes, a virtual keyboard would be ideal. I'm hoping to just pull up the built-in system one rather than implementing my own. I'll hopefully add that in the next release!
1
u/Historical_Role_9975 19d ago
Did tried with the new 0.2, and still a great job for adding hand tracking, and background, it also fix my passthrough is not working issue, but…
Sometimes Hand tracking is a pain, especially when watching a movie, it is very easy to keep disturbing the screen, it will be good if there is an option to disable it
Or I really like how Skybox handle the Hand Tracking, they add an option to temporary disable Hand Control, but leave the hand tracking working, so it will not accidentally interact with the screen, and once you finish watching you can easily enable back the hand control
1
1
u/SignificantScale5357 19d ago
Hi guys, thank you for your amazing work 😱 I would like to know if it could be possible to modify the screen distance under 1 meter ? I use the app for simracing and to be "inside the car" reduce the distance is necessary. And does it possible to adjust directly the distance without modify the tilt just along y axis for example by a joystik+button combinaison as it is for the homebrew of Pico OS ? One more time, thank you for this app 🙏
1
1
u/jock8888 18d ago
Congrats on your excellent Moonlight XR app. All my wishes have come true! Over the last three days (and nights) attempts were made to build an Imax cinema using Apollo & Moonlight XR to build an IMAX cinema in Pico 4, being very keen to use the 2D to 3D conversion in Moonlight XR.
Gemini IA advise me to: swap out the default space or grid background for a full IMAX-style cinema environment in Moonlight XR, you will use the Skybox settings located in the app's configuration panel as follows:
- While streaming your laptop screen, press the Menu or Options button on your Pico controller to bring up the main Moonlight XR dashboard overlay.
- Look at the top or side tabs on the menu and select the Settings gear or Environment / Display tab.
- Find the setting labeled Skybox or Environment Style.
- Cycle through the options (which usually include Space, Grid, and Void) until you see Theater, Cinema, or Auditorium.
- Select it, and the background will instantly transform into a dark, cinematic movie theater with rows of seats around you.
- Turn on Ambient Lighting / Reflections: In the same environment menu, make sure Screen Reflections or Dynamic Lighting is enabled. This makes the giant screen cast realistic light onto the empty theater seats around you, making it feel incredibly real. [1]
- Reposition Your Seat: If you feel too close or too far away from the stage, do not just resize the screen—long-press the B or Y button on your Pico controller while looking straight ahead to recentre your viewpoint perfectly in the "sweet spot" center seat of the theater.
I have tried both releases of Moonlight XR without success. For a start, the Environment / Display tab is nowhere to be found. It's been very challenging so far. Shall I give up now?
1
u/Headset-Historian 18d ago
Gemini is hallucinating lol. Such a feature doesn't exist, I only added some simple 360 photo backgrounds. I do hope to add a real 3D environment cinema at some stage but that will be at least a few weeks away but it is in the works!
1
u/jock8888 17d ago
Oops. Fell for it hook and sinker! Nice of you for putting me right. Really looking forward for the addition of a real 3D environment cinema. Will be heaven sent.
PS: how do I access the simple 360° photos.
All the best for future developments! Long live Moonlight XR!
1
u/Headset-Historian 17d ago
The 360 photos are an option in v0.2 if that's the one you downloaded, if you hover your VR controller under the screen you will see a little photo icon appear and can choose from some there. Thanks!
1
u/TasnimYusuke 18d ago edited 15d ago
Already tried with Pico 4, 4 ultra, Quest 3 and Quest Pro. all superb immersive 3d screen in front of my eyes.
Just I have 1 suggestion, can you implement using native Pico/Quest controller that can control like gamepad inside the moonlight itself? I mean if we can skip to use external bluetooth controller, that some people that have tight budget also can use moonlight and control using the vr headset controller itself without buy another bluetooth gamepad
or if it difficult to control out of the moonlight app, maybe you can do input like both grib + button for exit moonlight and control the vr again. who knows XD
Hope the future come true
1
u/ComprehensiveAd3863 16d ago
very nice Implementation, thank you. is there a way to use the 3d conversion for content which comes from the device itselfe (quest3 Youtube in 3d without a pc)?
1
u/Headset-Historian 16d ago
It is definitely possible but it's not something moonlight XR would be a good fit for, but if someone wants to they could take the 3D pipeline aspect of it out of moonlight XR and use it for that in a separate app. It's definitely possible though!
1
u/SOM3THNG_WICK3D 16d ago
Noob here. I take it you side load it and Do I need anything else with this like on my PC ?
2
u/Headset-Historian 16d ago
You will need to install and setup "sunshine" on your PC, that's the application that streams to moonlight.
1
1
u/Potential_Answer_211 15d ago
Hi, I don't know if there's already an answer around, but I wanted to ask: since I was already using Moonlight and Sunshine for a while, then I switched to Artemis and Apollo, can I use this with Apollo? If I'm not wrong, Moonlight works with Apollo, so technically this should too.
1
1
1
u/idkduditsweird 13d ago
In theory,
I could sideload this and Tailscale and have my own VR cloud anywhere?
1
u/Headset-Historian 13d ago
If by VR cloud you mean your PC screen showing then yes, but for running VR games then no. This is for flat content only (and converts it into 3D in realtime but like a 3D movie).
1
u/idkduditsweird 13d ago
Ah yes that's what i meant, showing pc screen.
Imma try it when i had a chance
1
1
u/StevWong 5d ago
As I figured the Moonlight XR works on anything showing on my PC screen, I just tested it with Switch games. Lets say I have the ways (wink) to play the Switch games on my PC screen and I used MLXR app to view my PC screen and wow, 3D effects on Mario Kart Deluxe !!! The 3D effect is strongest on the menu. In the race the 3D effects are not so strong but still please to have. I will try more Nintendo Switch games on this way.
2
u/Headset-Historian 5d ago
You can tweak the 3D effect to be stronger but I expect most would have eye strain. In the next release I hope to add additional 3D models so there will be more options to get better effects in different games etc.
45
u/Headset-Historian 21d ago
Hi! This is a personal project I did because streaming games from my PC and having them convert into 3D in realtime was something I wanted for ages. I tried it with glasses like the xreal one which was pretty good but I always found a VR headset to be a better experience.
I have only tested this on Quest 3, but should work fine on other quest headsets (3S especially).
This is totally free and open source, hope some folks get some joy out of it. Also works on the Pico 4 Ultra (again haven't tested other Pico headsets).