iPhone lidar with applications for the geosciences
opentopography.org
opentopography.org
[0]: https://pixel8earth.medium.com/ [1]: https://techcrunch.com/2021/04/26/snap-has-acquired-pixel8ea...
They're probably setting the stage for some future technology, maybe AR related?
Who the hell uses any of that stuff? But it lets them keep releasing new versions and charge huge amounts for new flagships.
Those gimmicks help to prevent the phones from becoming the next race to the bottom that the PC and cheap Android markets have become.
Well... you're right about the lidar, but I absolutely care about the camera and refresh rate. As a lifelong photography geek I basically upgrade for the camera improvements every year..
Camera, messaging, maps and dating apps are basically the only things I use my phone for these days...
As frequent user my good old Nikon full frame (D750), I am really astonished by output these tiny cameras produce. Sure, its not for big screen/print unless we're talking about sunny day. But for everything else, it makes fantastic portraits. I got fed up with the need to carry big heavy pouch around all the time, and missed way too many pictures of my kids to rely on it anymore.
It also sees in the dark much better than I do with my eyes, also thanx to Lidar - the pics I take from night walks where there is basically no light source hundreds meters around and I am in the forest (yes my night walks take me sometimes to interesting places, I normally don't use any light). Very dark scene becomes full of (usually) true colors and details simply invisible to me.
10x zoom on my phone allows me to read signs not readable to my eyesight (digital 'AI' zoom is very useful up to cca 30x). I was skeptical too, and its true full frame is still so far ahead, but at what cost - bulk, weight, the need to spend hours on postprocessing batch of photos instead of quick edits in phone in few seconds.
How does the lidar help with night shots though? Maybe focusing, but otherwise isn't it just a long exposure with algorithmic corrections?
For a photo you don't need much more. I can tell you that handheld shots are much better and easier compared to full frame dslr.
No wonder the earth is warming... You really need a new phone every year just for the camera ? I really don't get why anyone would do that without an actual professional need.
Makes them intriguing and fun to test out creatively. I can see the temptation.
The 13 has some things built in/enabled the 12 doesn’t have, which I had always wished the 12 had, such as video portrait mode, but I personally decided it wasn’t good enough for upgrade again, since its an ok assumption that the future models will also have that.
Yes, smartphone camera's are awesome now, but it's sad an awesome product like that will be thrown away after a year of use.
Even an iPhone 6s from 2015 with a new battery is faster than most low end Android phones and it is still fully supported.
(for context, my prior upgrades were in a more lenient trade-up program, which by coincidence resulted in me keeping up to date. not trying to pretend like avoiding a one year upgrade is some major sacrifice, its just what happened)
I've had some apps that stored files within them, and the app was no longer available for a higher version of iOS that a new phone shipped with. Would have been royally screwed if I had sold/traded in my prior phone because I wouldn't have immediately noticed except when I needed it.
Night mode in wide angle! Although it's quite a bit less capable than the other lenses. Maybe in the 14 ; )
And @eole666 of course I don't literally throw the old phone away every year, there's a whole downchain of very willing recipients of lightly used iPhones. Hell, my old iPhone 6 is still being used for QA.
See: it allows them to keep increasing the cost of their devices while simultaneously preventing them from passing the savings from their streamlined manufacturing to the customer.
And it's a gimmick because it's a tiny niche of people who do that, and I wish I didn't have to pay extra for a phone for a lidar. Would take a fingerprint scanner any day. Or a headphone jack.
This is simply untrue. Most 3D scanning apps on iOS make no use of photogrammetry or they do it as an alternate mode for smaller objects. I don't know of one 3D scanning app that combines the LIDAR with photogrammetry. They use one or the other depending on the scale of the object you're scanning.
> And it's a gimmick because it's a tiny niche of people who do that,
It's not the reason Apple added it. They added it for AR and photography. Arguably the former is a gimmick but the latter is future proofing and might end up being a game-changer.
The seek and flir imagers are cool for IR
I'm imagining a panda enclosure, with lots of people taking photos. 2D images are obscured by a line where the cage is. Eyes can correct for it by moving our heads around, but the photos can't.
There is an opportunity for cage-removal to be solved with software (guessing, or taking multiple photos and cross-stitching) or simpler hardware (camera on bottom of phone instead of top, use parallax like 2 eyes). But the lidar option might work too.
One thing that comes to my mind is portrait background blur - so far computational implementations on smartphones that I’ve seen were not very good, I suspect this can improve effects like that a lot which seems relevant to consumers.
On the forward looking side I think app developers can also get a lot of value out of this. Eg (just a thought) changing the way clothes or shoes are bought by letting you scan body parts to determine fit.
And this can generate a lot of data feeding into their vision models + what you already mention about AR/VR.
Edit: I forgot one super cool use of measure. Let's say you have a basement with an open ceiling. You can use the measure app to, say, measure a pipe. Then, when you walk upstairs, you can still see the pipe in 3D. So you can then understand just exactly under the floor where the pipe is. Very useful for things like pipes and electric wires.
That is really interesting. What app/service did you use to do this?
Other uses of LIDAR (measurement/data collection, AR, photography) are also likely to become more popular over time.
Turning the question round - why take photos of things? A 3D scan isn't that different. It's a way to capture a memory and to show things to other people. It's so quick to capture something that sometimes I just do it on a whim when I find something interesting.
I'd like to make an art piece out of them at some point but I'm still waiting for the right inspiration to come along.
Any app used on half their billboards (Shot on iPhone 13 pro) and featured on the Lock Screen seems “significant” to me. :)
The quality of the camera is one of the main selling points of the iPhone (Eg. The ad campaign on billboards is “Shot on iPhone”. So I would argue it is a very appropriate feature.
1) Biggest one Lidar is only on iPhones Pro in contrast to TrueDepth that after iPhone X landed in all iPhone devices
2) Lidar has very low resolution
I wish the added Lidar to all iPhones + combined this with TrueDepth (both in front and back of iphone) - true depth has better resolution but is slower (only 30fps + higher latency) and only works best for distance < 1 meter
Newer android phones do have a LDAF/TOF sensor, but it's much simpler than LiDAR AFAIK.
You should be able to get rgbd images from a phone supporting the AR Core Depth API: https://developers.google.com/ar/devices#google_play_devices
And I didn't have to switch my actual main phone which would have been far too much of a commitment to their walled garden.
The form factor is a bit clunky for scanning and I don't have the "it's in my pocket all the time" advantage you get with a phone.
Honestly the more mature photogrammetry techniques that stitch together multiple photos (from video or 360 shots taken successively) seemed much more impressive. Indeed, you can run most of the same apps with lidar off and still get similar results. There is a lot of depth info when you take pictures from different angles that a smart algorithm can figure out. The lidar adds more hinting but without professional grade equipment, the optical sensors (cameras) are much higher resolution and range.
The lidars on the i devices are cheap gimmicks.
Edit: just to add a bit more detail, lidar basically gets you a point cloud of depth data. The professional ones have big spinning mirrors that blanket the room in a matrix of powerful dots, like a Kinect on steroids. That takes a lot of power and, usually, moving parts.
The iPhone one uses more modern, smaller solid state technology but has a resolution of a few hundred dots and a range of 5m, vs the millions of pixels on the optical camera and a range of however much ambient light you have.
The lidar is an active emitter. It has to produce its own laser light, limited by its emitters and battery, and have a way to scan its emissions across 2d space (x and y axes) or else have a ton of micro lasers in a grid.
By comparison the optical camera is just a passive receiver, accepting reflected sunlight or light from the the weak LED flash. Those light sources are way stronger and more diffuse in terms of range and coverage.
Maybe in a few years the lidar will get better but for now it's just not useful for anything except face ID. Prosumer drones probably have better lidar and also better photogrammetry software.
More deets on i lidar: https://arstechnica.com/cars/2020/10/the-technology-behind-t...
And these match what I get - they aren't anomolous. It's not as good as photogrammetry but it takes minutes rather than hours.
It's not going to replace industrial laser scanners for surveying and site inspection but it's good enough for many of things we're currently using photogrammetry for.
And the sheer ease of use could open up new applications that wouldn't have been viable with slower workflows. Speed is a feature in it's own right.
It looks like something out of a video game dream sequence where your world is falling apart.
Measurement? Yes - it's accurate enough for many purposes like home furnishing and interior design mockups.
Games? Yes. It suffers from similar issues to photogrammetry (high poly count, need for repair and cleanup) but that's proved useful in many aspects of game content creation. There's also the fact that games are a creative medium and people develop new art styles all the time. There's some very creative uses of point clouds and 3d volumetric capture which is often glitchy as hell.
Sharing? Not sure what you mean.
Mapping? It's much better quality than the 3d meshes in Google Maps and they are incredibly useful for visualising and inspecting places you can't visit. Compare with the output from drone photogrammetry. This plugs a gap between large scale mapping via drones and laser site scans. It's better than the former and cheaper than the latter.
These days, the most common mapping technique is a similar workflow, but with a ruggedized handheld laser distance meter with digital compass and inclinometer. They may have on-device storage, and they may transmit measurements to a PDA or tablet via Bluetooth. Some people sketch the 2D projections of the cave on that digital lineplot, some use the data to sketch with pencil on paper. A cave is a very hostile environment to electronics, so "caveman technology" is often intentionally preferred.
Lidar and photogrammetry are quite difficult in a cave. The surfaces are irregular, of varying wet and muddy conditions whose reflectivity cause problems. And shadows - both visible and line-of-sight shadows - produce "swiss cheese" 3D models in all but the simplest caves. Lava tubes are often conducive to these techniques; their basalt composition wreaks havoc on magnetic compasses, so it's a welcome update.
Having used a $75k FARO Lidar device in an admittedly clean and comfy cave before, I sure wouldn't attempt to haul it into the average "wild" cave for fear of trashing it. The promise of SLAM-based devices, a computationally-expensive technique for realtime integration of 3D data building a 3D model as you traverse the cave, is amazing; but the expense still high, the hardware still fragile, and caves are still irregular enough to never make this easy.
Stone Aerospace in Texas has done some amazing work with autonomous underwater cave mapping using sonar. Their submersible robot has the same advantage that a quadcopter would have in an air-filled cave, whereas we human mappers are still stuck standing on (or crawling on) the floor.
This is more in line with parking sensors or something, not the long range lidar that has been used for some self driving car research that can spot traffic and obstacles at a distance.