Apple introduces spatial video capture on iPhone 15 Pro
apple.com
apple.com
"Some features may not be available in all regions...." is appearing on my update text.
Now seeing it here, it feels like a real product all of a sudden!
Is the video passthrough offset to the right/left due to the camera location or are you able to correct that in app (asking due to the open peripheral vision).
We synthesise the views for the left and right eyes by projecting the camera image onto some assumed world geometry. Our 6-DoF experiences are generally designed for a table-top setting, and we project the camera onto an infinite plane on that table surface (there's a physical origin marker to define the origin of the space for the 6-DoF experiences). There's also a plane parallel to the device a few meters in the distance (to avoid a "horizon").
Sounds simple but for table-top experiences like viewing 3D models on a table it does give a convincing stereo effect - things like shadows from the model on the table just look right. Combined with the really low-latency display pipeline[1], the nice camera (1440x1080p @ 60fps), and the direct peripheral view of the real world thanks to the headset design gives a really pretty comfortable MR experience.
The next job will be improving the quality of the world geometry so that camera reprojection works better for static parts of the scene. We might try something live with lidar but for most rooms an offline process (or even manual placement and scaling of planes for walls / floors) would already provide a decent improvement.
The hardest part is foreground objects - hands, controllers, other people. Some scheme to detect and segment out those foreground bits and reproject them separately using an estimated depth map (whilst infilling the background layer for the static world) would be needed there. Complex in general but for things with known geometry ie controllers, it seems like something we should be able to do.
What's hopefully clear is that we're pushing to make the headset MR experience using your phone the best it can possibly be, given the limitations imposed by the hardware such as camera positioning. We're in a pretty good place already but there are definitely more improvements to come :)
[1] As this is HN, I can say a bit more technical stuff about the display pipeline... We have an iOS implementation for async timewarp - we've been able to circumvent the usual queuing of frames that happens on iOS to implement a late warp render just 3ms before a new frame starts scanning out. That means we can get a really recent camera frame (on iOS frames are delivered to the app around 20ms after capture) and also apply rotational reprojection for any movement since the frame was captured. Reprojecting for predicted translation at display time is on the roadmap too which should remove the last bit of noticeable latency when moving.
P.S.: If support for unity AR development is robust enough, I'd highly encourage you to aggressively pitch this to educational institutions. One major pain point is how challenging it is to provide a solution at scale to have students building in VR. A solution to be able to spend a modest amount of money, on something that is super robust, that leverages tech all the students have, and that I could get a dozen of for the price of two of the alternatives, would be a game changer.
So I guess they're doing some processing to generate 3D models, or something along those lines.
I don't know how it works, I'm not opticist, but I assume there's some kind of math involved. Or maybe it diminishes the 3D effect but it's still enough to look interesting.
Stereoscopic video that has depth information courtesy of the LiDAR sensor.
From people that have used the headset it feels like you're actually there.
The iphone cameras are close together, is there really a benefit in recording both at once?
So, are you confirming they record from two lenses or just here for the reductio ad absurdum?
And then you go on blithely accusing people telling you of refusing to participate (which is itself a logical fallacy) in a high handed, holier-than-thou manner.
It has a slow startup time, limited range, and low resolution.
Possibly it could be used for depth calibration but I doubted it would be used for depth information.
Can you link any such review? Gruber is Apple's biggest shill and even he said that it's definitely not that immersive.
That's a shame. I would have expected the ability to have very subtle rotation/parallax effects, a la Facebook 3D photos (https://www.facebook.com/help/iphone-app/414295416095269). If you want people to actually start recording media to be viewed on the device en masse, I feel you need to throw them a bit of a carrot other than "well, you'll be able to look at them on some device you might own in years."
As a mathematician I've long wanted to figure out the code to match up several photographs for a 3D model of an open book, to correctly flatten out the image of each page. Bonus points to use AI to clean up the image, or convert to source that faithfully represents the page layout.
There are apps that take one photo and then guess how to correct the distortion. They don't do as good a job as one could do with the actual 3D data.
Many of us don't want to dedicate the space or money for a professional book scanner, and can't find an affordable flatbed scanner that scans all the way to one edge to better support books. I'll sometimes slice a second used copy of a cookbook, to scan every page using a sheetfed scanner. Grabbing one's phone should win ease-of-use over any other solution, if it could match the quality of these other approaches. 3D cameras would help that.
I've seen the poor performance of current phone apps. My point is that one could do better for less coding effort with access to a 3D model. Either a pair of photos, or a 3D camera.
Something that could map the 3D surface onto a sheet would be awesome.
What is described above is something I’ve seen in a documentary. They have a very fancy camera and the person holding the book can just slowly flipping through the pages, and it can reconstruct the flat pages.
I guess the problem is not technology or mathematical model behind it, but how to make it cheap. It is analogous to Google’s self driving car, that you want to solve the same problem with much cheaper technology.
That said, you might be able to do this already with iDevices equipped with LiDAR (e.g., iPhones Pro and iPads Pro). They’re already used to create 3D objects, but I suppose no one has used them yet to take the next step you are talking about.
Tangentially, if you gave up on Foundation on S1 like a couple of my friends, S2 is much much better. And if you're a book purist they sort of explain why there are differences between the show and the books in a relatively decent way.
It also explains the kinda-cool but kinda-weird content released by Disney such as the Star Wars Biomes and Vehicle fly-throughs. Expect those to be released in 3D alongside the headset.
Ultimately, we're going to be able to play HoloChess against R2-D2 inside the Millennium Falcon, stand on the bridge of a Star Destroyer, or examine Uncle Owen's charred remains on Tattoine.
Actually... you probably can scratch that last one. ;)
The latter is boring and easy to implement, the former much more compelling and technically difficult.
When it comes to photography "AI" is a spice best used in moderation, cf eg https://www.theverge.com/2023/12/2/23985299/iphone-bridal-ph...
And while stereo video as a format is simple, IMHO it’s still quite a feat of engineering to capture it from the current phone generation. There have been stereo Android phones before but they were engineered from the get go with stereo in mind, and sacrificed other elements to achieve it
I followed [1] on a Pico 4 headset (video is about Quest 3) and it was pretty fun. The aspect ratio was a bit wonky.
But the 3d effect is very subtle. I guess thats what you get from two cameras sitting 2cm next to each other (instead of the typical 6cm of two eyes). IMO the early reports from people trying the vision pro have entirely oversold it. Yes, it's better then your typical 180°, but the 3d is disappointing.
The samsung galaxy and pixel pro with their 3 lenses next to each other, using the two outer ones could do much better, if they implemented something similar. So in typical apple fashion, they are hamstrung by their own design.
You can bet those demos were not recorded by an iPhone, but a pair of studio cameras spaced further apart. There were reports of apple working with movie studios to develop new hardware. We might still get that quality from professional productions.
It's still more "spatial"/3D than a regular (single lens) image.
Plus this has a wide lens and a "regular" lens (actually both wide iirc but one is ultrawide), so it's not like 2 equal lenses 50mm apart like in regular stereoscopic "3d" video.
You need to move the close objects further apart in left/right than they are in the camera. Then you need to fill the newly empty areas with something.
> Plus this has a wide lens and a "regular" lens (actually both wide iirc but one is ultrawide), so it's not like 2 equal lenses 50mm apart like in regular stereoscopic "3d" video.
This doesn't affect anything.
> The bigger problem is the inpainting needed to generate hidden detail when the movie is viewed from angles that are different from the one it was actually filmed from.
I have an Intel L515 which is pretty remarkable in that sometimes you can get some depth finding outdoors. This is just a hobby item for me, I'm not an expert, but this launched as a fairly impressively long range & capable $350 USB3 system, and it seems like the market hasn't much comparable to it. Phones certainly I'd expect to be significantly worse.
Aren't many "self driving car" sensors lidar? This would imply they can work in daylight - perhaps they don't necessarily depend on light on the sunlight spectrum?
(Or perhaps you don't consuder them consumer? Though those cars are consumer products, they're not made for military or industrial use)
The L515 I mentioned was somewhat advanced at least for it's day because it used MEMS to steer its light source. That gave it leading class performance/size but it's still big and kinda hot-ish. Maybe we can keep scaling that kind of system performance to smaller sizes but even this package was pretty cutting edge & gave much better falloff than many competing systems, and was still largely an indoor sensor.
It's for spatial video, not for holographic video. When you see a 3d movie in a cinema, it's not like you can look at it from widely different anges and go peek from the side or behind the actors or whatever...
Apple bought C3 Technologies a decade ago, and they use this technique to fuse photos from low flying charters to produce the 3d view in Apple Maps.
[ Paper: https://ui.adsabs.harvard.edu/abs/2008SPIE.6946E..0DI/abstra... ]
[ Coverage: https://9to5mac.com/2011/10/29/apple-acquired-mind-blowing-3... ]
[ Similar: https://web.stanford.edu/class/ee367/Winter2021/projects/rep... ]
Exactly. The steps back change perspective, not the lenses. That’s what I was trying to say above. In the iPhone both lenses are at the same distance to the subject.
It took a few steps to get it out of the phone and onto YouTube in a way I could watch it in my VR headset, but the result is kind of nifty and nice to be able to produce with the phone I already have with me. I think a recent update to the Spatialify app that I was using makes it so you can do everything on the phone now (I had to run it through ffmpeg on my computer).
When I saw the Apple Vision headset, I thought it was a big miss to highlight it as a content creation device. The parent around their toddler with the unwieldy headset on needed to be ridiculued. I thought that should be a separate device, and just to let users discover on their own that the Apple Vision could technically record too.
It was great to see that the iPhone 15 Pro was going to be that device, and just a software update away! Here we are now
This spatial/stereo/3D video format will die the same slow infamous death I guess?
https://www.samsung.com/au/support/mobile-devices/motion-pho...
My only gripe is of the unrealistic “want my cake and eat it too variety,” where I shoot more in RAW nowadays and use third-party camera apps, so those don’t get the Live Photos treatment.
I’ve “solved” this by purposefully using the stock camera app for quick snaps of moments I want to be memorialized for sentimental reasons and reserving the RAW stuff for more artistic purposes, going even as far as taking another photo with the stock camera app if the situation calls for both.
But honestly, I much more appreciate my Live Photos, if only because they’re rolled into the Moments the iPhone automatically creates, which make for a great memory.
This will give you more reason to buy a new iPhone with more storage or upgrade your iCloud account. Memories will be made and bills will be paid.
But I thought the parent question was about watching the spacial videos — as 3D — on non-Apple VR devices (e.g. Quest).
Any platform can implement playback of them.
Next is having them as sources in an NLE. Lots of ways to do that now, but that's been for separate L/R sources or top/bottom type of files. It's been so long since I've had to deal with that, I've not kept up with what the current state is for that world. Maybe they've all been updated to handle mv-hevc too??
I also haven’t seen an example of an NLE supporting it, but I imagine we’ll see some within 2024? I’m more curious about how they might support playback in various headsets since a Quest 3 is now a lot cheaper than a 3D display or projector.
That’s likely necessary to shoot this format, as it uses the pair.
On a 14 pro, you’d have to hold the phone diagonally and even then the sensors wouldn’t align with the axis causing cutoff.
1. Clearly the array of cameras on these phones is along two axes, so the pedantic definition of two points on a line is irrelevant
2. You’d be shooting everything at a Dutch angle to maintain stereo if they weren’t aligned horizontally or vertically.
3. The sensors are rectangles and need to be aligned along their longest axes to maximize the most field of view.
The iPhone 14 could offer a lower FoV. The video format encodes the FoV of the recording already.
Bear in mind this has to play back on one of the highest ppd displays on the market when the Vision Pro releases.
Resolution has always been my main complaint about VR headsets and they are nowhere near the PPD needed to simulate watching high resolution content on a high resolution screen in front of you yet.
The Vision Pro is supposed to have 23MP, and an aspect ratio of 1.165:1 which suggests 3660 pixels across each eye.
However, what most people fail to think about is the FOV. The headset's entire FOV is about 100 degrees horizontally. THX standards recommend around 35 degrees for video horizontal FOV, but if people like a really immersive FOV for their video I think it's safe to push this up to around 50 degrees.
This means that a virtual theater screen within your FOV on the headset has only half the headset's resolution to work with. So that makes the width of a virtual 50 FOV horizontal screen 1830 pixels wide, which is less than even 1080p video at 1920 pixels wide.
And this is probably a bit high too as there is typically some amount of pixels you can't even see through the lens of a VR headset because you don't normally want to be able to see the edge of the panel inside the headset. Rather you see the edge of the optics. So that 3660 panel width is probably more like 3460, then divide that in half to 1730 pixels wide for your virtual screen. And that's if you like a 50 FOV wide virtual screen. That would be like sitting 5 feet away from a 65" TV which is arguably closer than the vast majority of people sit.
So if you are watching video at a pretty immersive 50 degree FOV, in an Apple Vision Pro, the native resolution of the video needs to only be about 1730x973 (assuming 16:9 video) at most to max out the PPD of the headset for reasonably sized virtual video screens.
Therefore there would be some level of interpolation causing loss of clarity. In general, you’d want to double resolutions as a form of super sampling.
The 1080p here would be just about cutting it then at the initial 35 degrees you mentioned.
I think for the natural FoV of the phone camera and the types of content people tend to shoot (families etc) that a 50 degree view might be uncomfortably large. Perhaps for further afield content like landscapes etc
Why it would need to be higher resolution than the native pixel space that the virtual screen will be rendered over? Changing the angle or rotation of your head to the virtual screen only makes the virtual screen technically even lower resolution, and would make it so that you would notice an even lesser difference between higher and lower resolution videos being played on that virtual screen.
I chose 50 FOV because I think it's reasonable near upper-bound. I am heavily into home theater, and I have tried various VR headsets to replicated a home theater projector setup. Personally I am a fan of sitting to my projector (or TV) screens where the width of the screen is equal to my viewing distance. This results in a horizontal FOV of 53.1 degrees.
Now I often invite people over to my theater, or travel around and visit other people's HTs and I have found that most HT enthusiasts have no problems with this rough screen width-to-distance FOV for viewing movies.
However maybe HT enthusiasts are not very representative of the masses, but anyway I was only using it as more of an upper-bound and what I have been looking for in a headset.
If I want a virtual HT screen that has the native resolution of at least 1080p, and the headset has over 100 degree FOV (which most do), then I see that I need a headset to have more than 4000 pixels wide per eye, which sadly just doesn't exist yet.
But, at least in my experience, I don't think the resolution of the video itself needs to be substantially higher than the pixel space of the virtual screen. I think that regular real-time machine learning video up-scaling techniques are sufficient enough these days to render the video in a way that wont cause bad artifacts when the virtual screen pixels are skewed because the headset wearer is tilting or turning their head in relation to the position of the virtual screen.
Now I am fully aware that VR signal input also needs to be a higher resolution source than the native headset pixels, because there is some warping done to the image to make it look correct through the lenses, but that should only need to be applied to the 3D environment space the user is seeing. That means that the video being imposed into the virtual screen simply needs to be stretched via up-scale to a higher resolution and then this will be downscaled via the headset warp more or less back to its native resolution.
But what you are saying does make sense and I am sure is somewhat of a factor.
But then also, lower resolution video can still look quite fine once there is enough pixels to get good detail.
In VR you have two. You have the plane the video is played on and then the display. That disparity can cause issues like aliasing or generally weird bluriness.
You can see this even in relatively lower resolutin headsets like the Quest 2/3/Pro where even 1080p video often looks bad even though it’s occupying less than an actual 720p section of the display.
And yes you’re right you could employ an upscaler. But the name of the game in VR is latency and power draw. NVIDIA has done great work in this field but they have a lot more latitude due to form factor. To the best of my knowledge, Apple tend to prefer straight decode and playback without AI upscalers on their devices.
That it can play elsewhere is not a consideration.
I mean your blowing me of smugly but all those points can be overcome with just a crop and tilt. It shows the typical Hackernews mentality “your wrong even though I have no experience in this space and only thought through this superficially.”
Apple wants to sell more hardware.
1. Nobody would want to hold the phone at an angle to shoot all the time.
2. It’s already reducing resolution to 1080p because it’s cropping in on the Ultrawide when its axis aligned. How much lower would you go to do an angled shot? The resolution does matter because these will be playing back on one of the highest ppd headsets on the market.
Apple may want to sell more hardware, but that’s not particularly interesting insight. But at the same time, your theories of how it could be done on the 14 Pro would make for a rather poor user experience all around.
2. The censors are 24mp and 12mp, way way way above 1080p (2.1mp). A tilted crop sample will be no issue at all.
Also you are using the term 'appeal to authority' wrong as I am not referencing an outside authority here, I am referencing myself. I'm sorry I've made you a bit angry here, but just calm it down a bit and think it through. The overlayed images of two cameras are slightly mis-aligned, but the overlapping data is more than enough to produce a 2.1mp image. This can work both in traditional stereoscopic or with 3D generated depth data.
I think one thing that threw me though is the the advertising images for this are clearly deceptive. They show 3D generated video that can be viewed from multiple angles, when in actually HVEC is just stereoscopic.
Like, sure you can do any number of things if you don’t consider how it’s used at all or what the user experience is.
If instead, you said “hold it normally and we crop/rotate for you” , you would now have two point of views at different heights, you’d have very uncomfortable depth perception.
There’s not a solution here that would work with the 14 Pro camera layout unless you used the telephoto and then cropped the wide angle to match.
In which case you now have the issue that your users are stuck in effectively a 70mm fov which isn’t very useful for most use.
Again, I think you’re ignoring many aspects of the product design to try and say it’s possible.
To be honest I look at the 3 cameras on the back of my 14 Pro and I have no idea which is which.
Won't this look extremely soft and stuttery on a Vision Pro? Even on a standard screen these videos will look quite bad compared to the 4k60 recordings that every Apple phone since the iPhone 8 can record.