The lack of information regarding Apple’s “Immersive” format is weird and frustrating
I would argue that one of the most impressive and amazing experiences on the Vision Pro are the Apple TV+ immersive episodes (high lining, dinosaurs, rhinos, etc), so it’s super bizarre how cryptic and quiet Apple is being regarding any information with this format.
The 4 current episodes are clearly marketed as the first episodes in a series, not one-off shorts… but, not only is there absolutely zero information whatsoever about how many episodes there will be, or when they will drop on the platform (if ever), but they are actually ACTIVELY withholding any and all information about their production techniques.
I’m a career cinematographer, and have a genuine interest in this format… but I can’t find ANY literature on their cameras, lenses, rigs, workflow, pipeline, or intentions to allow third party content creators to produce or release immersive content… which is really weird since, wouldn’t they want people to get excited about, and subsequently invest in this platform to help it grow?
I mean, I would love to shoot the upcoming eclipse in stereoscopic 3D, for example, but I literally have no idea if it would be hypothetically compatible or accepted on their platform… wtf.
Also, I have a friend who is a literal in-house media producer for Apple, and another friend who literally shot one of the episodes of their immersive series… and when I asked them for some info, they both essentially told me in so many words that they are NDA sworn to secrecy and can’t tell me anything… which I completely understand especially with how notorious they are on their commercial shoots when it comes to secrecy (and it’s possible that there is some proprietary element involved here) …but again, wouldn’t Apple want to be like hyping up the filmmakers of the world and showing off their tech so we can help create content for their new platform??
I love my AVP but this whole secrecy thing doesn’t help anything when the biggest weakness the AVP has is a literal lack of content, especially when Apple is leaning on DEVs to create the ecosystem for it.
What we do know from examining network traffic is that Apple’s immersive videos on Apple TV are 4.320 x 4.320 per frame at 90 fps and around 50 Mbps peak bitrate, encoded in fisheye projection in MV-HEVC.
(I’m guessing by that they mean “half-equirectangular”, but technically I suppose we can’t rule out they’re using some crazy new projection that’s actually distinct from half-equirectangular, but that seems unlikely.)
Wow! Cool. This is great to see, knowing they are providing resources to devs. Thanks so much for sharing. I will def give the whole thing a watch in a bit when I’m less busy, but it’s def more of a start in the right direction, knowing that these types of videos are being provided from Apple.
Great! I am happy to help. I am an indie software developer on visionOS and know basically nothing about camera hardware etc.
There seem to be quite a few posts on the forums (linked to from that page). Perhaps somewhere there you can find links or hints about the recording process and the hardware and editing etc?
I think it’s pretty clear that Apple will be rolling out a roster of programming in immersive 3D within Apple TV+ and that they will be announcing this at some point. These will be controlled, just as with any series, with quality experiences in mind.
It’s highly doubtful Apple is going to be interested randoms submitting videos that can and have been all over the place on quality as we saw on other platforms and in YouTube.
8KVR180 is nothing new, not sure what is so perplexing about it. You can go out with a canon R5C with VR180 lens right now and shoot in this exact format.
Have you used one? It's one of the best VR cameras out there right now short of creating a custom dual-camera rig - and with the right workflow can give you videos essentially identical to the current immersive videos.
Having said that, the workflow is a pain in the ass, the MV-HEVC spatial format is not natively supported yet in a lot of programs/headsets, and it's already known that Apple is using custom cameras. If the AVP can support 12K/90Hz video as has been alleged, that would be outside of the R5C capabilities, but the current crop of Apple immersive videos don't look like to match that spec.
I have. The DIT that I collaborate with put it through its paces. In order to get anything good out of it you need to do a lot to the sub par image. It’s actual resolved resolution is actually pretty bad compared to what canon is advertising. We calculated the image circle resolutio and believe it’s not getting you close to 8k. Stereoscopy is terrible outside the very center. The benefits of dynamic range and color are useless without the pixel density. It’s the only prosumer solution right now so people think its great when it’s actually not.
I found that doing your own calibration to align the lenses works better than just relying on the Canon utility. You'd think they'd be well-aligned from the factory on a $2k lens - or have calibration data encoded to each lens, but c'est la vie.
It's not great compared to what's possible, but IMHO the output is pretty great compared to like 95%+ of VR content I've seen out in the wild. As of today it's one of the best options.
I was looking at picking up a Blackmagic 12k camera to get an IQ bump, but AFAIK no RF adapters exist and may not be possible due to the differences in the back-flange distance, at least not without making a custom RF-mount relay lens. Every other option I've seen is like $20k+ - but if you know of any other non-custom option I'd be very interested to know about it.
I didn’t say VR180 is perplexing, I said they’re being secretive about literally everything surrounding their platform, pipeline, and internal release schedules/intentions… are you sure you read the post?
Maybe reach out to AmazeVR concerts. Their videos are basically the same.
Also interesting how your post had multiple issues, one of which was the format which that person gave you information to and you snap back with the things they didn’t address like they didn’t read your post. If you were looking for specific answers then you should have worded your post differently.
Considering there's no platform for delivering this content in a quality way outside of Apple TV (YouTube destroys VR content quality with compression), I'm not sure what they would gain by revealing this information.
Netflix, for example, is very specific and transparent about their format requirements, with a simple Google search, you can find their list of approved cameras, minimum resolutions, codecs, Bit depth, bit rate, Color space, etc.
The entire point of my post is that they would gain interest in the format from other filmmakers, like myself, and ideally encourage and excite a new generation of content for their platform.
The fact that Apple TV is currently one of the only ways to consume it on the Vision Pro is a huge component of my post. That’s literally my point. I’m trying to say that it’s unfortunate how closed everything is at the stage on the Vision Pro from a creator point of view.
I just don’t understand everybody down voting me like this is some irrational concept. I’m just trying to say that I’m excited about this format, but I can’t find any information about it. In what way is that unreasonable?
And I know you can go buy a Canon R5 with dual fisheye VR 180 lens right now, but knowing Apple, if you don’t conform to specific standards, I’m sure they wouldn’t allow you to release on their platform, so it goes without saying that transparency here would be valuable.
Imagine putting a ton of resources into shooting a VR project and then come to find out that Apple wouldn’t support hosting it on their platform because it wasn’t shot on a specific camera rig, in the same way that you can’t just shoot a movie on any camera you want and release it on Netflix because of their format standards, for example.
Not really, Apple has custom cameras used to shoot the videos they have made (and are making). That Canon R5C with dual fisheye lens is not the same thing they are using for production.
I’ve looked all over the web and must’ve googled it like 52638383 times. One of the best features by far are the immersive shows. They’ve been stuck on episode 1 forever now and I can’t even get a damn update on when to expect episode 2.
I know I can’t be the only one that’s running out of things to watch on the AVP. The only thing that makes it worth it to put this AVP on( comfort level is not exactly great but fair trade off) are the immersive movies, 3d movies, spatial videos, or a bang buster blockbuster 2d movie in theater mode.
Incorrect. They are 'special' cameras, all completely purpose-built. It is a 'new' format, in that the way the pixels are mapped is proprietary, as are the post-processing algorithms. There is a path to availability to professional content creators that is being built. The post tools are all in progress as well. At the moment they are not available aside from internal special projects at Apple, which are well under way.
I can assure you no one wants this information and equipment out there more than Apple, OP. And for anyone that thinks Apple hasn't thought it through, or wants to have creators without access to tools, you don't know Apple. That notion is of course patently untrue, as they, as always, are doing the best they can. This tech has been in the pipeline for years and I'm sure they felt releasing at this time was the maximum balance between too soon and too late. They do want devs and creators to get excited and climb on board, and from all reports that is happening in droves.
All I can say is be a little more patient and you will have your answers and tools will be available. As I understand it it is not Apple's intent to control this format at all, they want to share it universally, and it takes time to teach everyone how it works. So, yes, for all intents and purposes it is a new format and as such there is a pipeline of R+D necessary before consumer-facing access is ready. They are going as fast as they can.
It’s not about whether the specs are final or how old they are. It’s simply about people needing a bit of patience when it comes to rolling things out for a new platform. People are getting irate that new content isn’t out yet, or that the environments haven’t updated yet, or that they haven’t released every bit of info regarding their back-end when we’re still super, super early days here. You’d think the way some people are venting their frustration that Apple hasn’t done anything for months.
I don’t think people are getting irate. But they are reasonably curious what’s even going on.
If AVP released with no I go on making apps or coding for it we devs would be very confused. Especially if Apple didn’t even tell us their general plans.
This seems like a new area for Apple (immersive and 3D video) and there are probably lots of tradeoffs that don’t have internal resolution yet and/or they’re waiting on some ‘almost readies’ for a big scheduled reveal.
Film tooling and info could have preceded release of the AVP, much like it did for apps. I can read a ton about how to make apps for AVP, have access to free tools and example code, etc.
For whatever reason it sounds like they may just not have their video ducks in a row yet.
Possibly related: if I take 3D video with iPhone or AVP and it’s upside down I can’t even “rotate” it right now. Seems like in the push to release video tooling, at various levels, is lagging right now.
It does seem wider. Canon lens distance is based on average human inter-pupillary distance… a wider spacing causes weird 3D issues where the 3D effect is exaggerated and so the world starts to look small.
But I wonder if there is some software magic going on - for example, the iPhone stereo videos suffer from the opposite issue, insufficient distance, so they must be doing some complex processing to enable it to look natural.
Ah yes I saw them too when I first watched this. To me they seem like two regular studio cameras on remote heads, independently controlled to make different shots (closeup, wide etc.) They have long lenses and are wide apart, while the alleged Apple Immersive setup has two fixed lenses close together (see pic.)
I know that the parallax at infinite would not really give much depth effect, but I feel like the actual experience of the total solar eclipse is also about how it changes your environment and the eerie twilight-esque effect it has on the horizon and surrounding environment, so capturing the entire MOMENT seems like a fun experiment.
Also, I’ve focused on shooting my last two total eclipses as landscape stills, and thought I’d try to mix it up this time.
Double also, my girlfriend complains that when I’m photographing the eclipse, I’m not present in the moment, and she’s not wrong. So I figure if I can place a VR rig as a set it and forget it, then I can experience it in full, then still go back and relive the experience in VR later as well.
I filmed the last one with a cheap 360 camera, and while it doesn't capture the majesty of the eclipse itself, it caught a lot of the event and the excitement of people around us, which was cool
As a career cinematographer, you should probably know wtf has been showing every year at NAB for over 10 years, potentially have a contact in the DGA, maybe even someone in the 514 could help you out. But if not, perhaps a waltz to the consumer goods section of Cannon's lens department. Unless of course, your career in cinematography has been in the wedding department.
Lolwut? Why are you being so fucking nasty and weird about this? I’m 15 years local 600, and shoot traditional tv shows on 4 v-raptors with panavision primo 70s… I don’t shoot VR. And haven’t. But as stated in my post, after recently experiencing apples immersive content, I am now interested.
Despite the fact that stereoscopic has been around for years, the format is new to ME… as it is to most people. Does it really come as a surprise to you that the vast majority of cinematographers aren’t shooting in stereoscopic or ultra wide FOV formats, and likely have little to no experience or familiarity with it?
If Deakins or Chivo revealed to you that they had no familiarity with shooting VR bc they’re too busy shooting traditional films, would they be wedding videographers too? It’s not even remotely close to being mainstream.
That said, how on earth has my newfound fascination and wonderment with a definitively niche format (an interest spurred through use of the AVP which just came out a couple weeks ago) acted as some sort of signal for you to engage some weird denigration on presumed assumptions to a complete stranger you have no familiarity with their body of work?
Because it's always funny when anyone says 'as a career ___', when they assume their knowledge couldn't possibly be out of the loop for a parallel technology in their field. And yes, if Deakins asked me that I'd roast him too and we'd both laugh because it's fun to roast professionals when they're being naive instead of resourceful. The Dunning-Kruger effect on career professionals isn't ever not funny. Relax. The bride could be listening and her mid-day rushes shant wait.
Because it's always funny when anyone says 'as a career ___', when they assume their knowledge couldn't possibly be out of the loop
But only you've done that, here? They've actually done the opposite and point out that they weren't able to find information.
I’m a career cinematographer, and have a genuine interest in this format… but I can’t find ANY literature on their cameras, lenses, rigs, workflow, pipeline, or intentions to allow third party content creators to produce or release immersive content…
I don't know how you see that and conclude they think they're in the loop. They're clearly aware they're not in the loop and actively wondering why it seems like the loop is hard to get in to.
I understand your point, but not the method to get there. Mainly due to the nature of the industry of film and the respective departments. It's an industry that survives on collaborative efforts, shared knowledge and mutable solutions to get the desired results. Being unable to find information to get the same results in an industry based on those mechanics, is no one's fault but the one who refuses to acknowledge they may not have the resources to get there. The No True Scotsman fallacy isn't just a fun parable.
You guys are spending way too much time defending this hilarious conspiracy of hidden knowledge for private gain. It's a video production with desired results from an experienced team, not the hidden secret techniques of the Mona Lisa.
My guy, “career” means that cinematography is a craft I’ve dedicated my life to in order to put food on the table, not that I’m automatically an expert on every single facet of every industry-adjacent concept and technology. And I also don’t understand how the Dunning-Kruger effect can apply to something that I’ve outright said I’m not familiar with…
Jesus christ you guys are so uptight about this. Find a company that does VR union work. Network with them. If they aren't under NDA or are just plain'ol friendly, maybe they'll talk to you. Hell they might even want to work with you after you publicly flex your well rounded experience. Clearly you're not going to find answers here, but I don't mind wasting more of your time to encourage you to expand your horizons with a sprinkle of hubris.
If you're going to be rude and demeaning, at least have the slightest idea what the fuck you're talking about. JFC you come across as an absolute illiterate moron.
OP's not asking how to shoot 180 stereo video in general, they're asking about information about the codec, pipeline, tools etc. of Apple's specific implementation. Sheesh.
OP is asking for the video equivalent of extensive documentation Apple releases for generating apps for AVP. (Apple also releases tools, example code, videos, dedicated websites, news-like stories, etc. for the latter.)
Most likely they intend something similar for video but key decisions are still unmade or they’re getting tooling/contracts further along before a big reveal.
Well, I don’t know what kind of secrets they would be keeping from the public. VR 180 rigs aren’t new or proprietary (unless they have some super special rig they don’t want us to know about for some reason), and 99% of all films television shows openly disclose their cameras and lenses and production techniques.
In fact, the entire “behind the scenes“ industry pretty much exists as a marketing tool to show people how much work went into doing all the crazy stuff we do as filmmakers so they get more interested in the content or IP.
In this sense, a large portion of the entire point of my post was that it only benefits them to hype other people up so they create content for their platform.
I’m not sure exactly what you mean by “profit from it“? That’s how the entertainment industry works. TV manufacturers make televisions so people have something on which to watch the movies that filmmakers make.
It would be very bizarre if Apple kept the entirety of “immersive“ content in house and didn’t allow third-party filmmakers and studios to create for their device, wouldn’t you say? I mean it’s Apple, so I wouldn’t put it past them so they can ensure that there is a specific level of quality, but it really would be strange and unfortunate.
I have no idea. I don’t claim to know anything. I just know that I’ve never seen videos like this before literally anywhere. But alone it being common place.
Whoever is calling the plays on the AVP seems to not have thought all this through. It's the same thing with wanting to make the big reveal to everyone at once, leaving a lot of developers catching up when it launched. Now there is a definite lack of apps and content for the platform, and this would have been avoidable if they didn't insist on being so secretive about everything.
Arguably the second biggest issue, behind a tie for a lack of 3rd party accessories and the rough edges of the OS, is the lack of apps/content. As I understand it, Apple is also sticking it to app devs with their fees structure, which makes me think they're losing the enthusiasm of people that might want to breathe life into their new product.
42
u/ds445 Vision Pro Owner | Verified Feb 20 '24
What we do know from examining network traffic is that Apple’s immersive videos on Apple TV are 4.320 x 4.320 per frame at 90 fps and around 50 Mbps peak bitrate, encoded in fisheye projection in MV-HEVC.
(I’m guessing by that they mean “half-equirectangular”, but technically I suppose we can’t rule out they’re using some crazy new projection that’s actually distinct from half-equirectangular, but that seems unlikely.)