Generating aerial imagery with your iPhone's Lidar sensor
jakecoppinger.com
jakecoppinger.com
No less cool, but I was rather confused for a lot of the article, wondering what in the world the range on this sensor is (usually you get a few meters on normal-priced sensors, not hundreds of meters, and the picture showed entire streets being captured). The article actually mentions the maximum range is 5 metres after you set it to low confidence.
This capture was just holding an iPhone in my hand - but you can place it on the front of a bicycle too.
The Apple ARKit part handles the SLAM [1] of fusing the camera/LiDAR/gyro and possibly GPS.
[1]: https://en.wikipedia.org/wiki/Simultaneous_localization_and_...
Not an issue of yours, it's all a matter of context -- and how closely one reads the article while trying to get an idea of what it's about. I just happen to be in the OSM ecosystem deep enough to be very familiar with the word aerial (as opposed to other types of area photographies), but not deep enough into the Apple sphere to know the capabilities of these devices and realize what the title clearly must mean. It might just be a me-issue :)
Do you know the functional limit of the iPhone’s LiDAR? Ie: could I strap my Pro to my old DJI drone and create some photogrammetry of my house or whatever?
Yep that should be possible - as long as you flow below 5 metres though!
If you have a drone and are somewhere that is legal & safe to fly, photogrammetry (3d model from just visual photos) will likely give a better result than relying on the iPhone LiDAR stitching.
Set the drone image self timer to 2 seconds, fly around for a bit and then upload all the photos into WebODM: https://opendronemap.org/webodm/
You can run it on your local machine (see instructions in blog post) or hosted for a fee. It creates brilliant models and is a lot of fun!
> it usually entails flying low enough to annoy your neighbors
In my experience the higher the better - you can get a really nice model from 120m (legal limit in Australia).
The problem (I guess) is that aerial/satellite/orthorectified are not equivalent.
Yep I agree - I would call it orthorectified georeferenced imagery but unless you're in the GIS/OSM space you probably don't know what that means. I agree that "birds-eye" would have been a good alternative.
Satellite imagery is technically incorrect whether it's from a iPhone or a plane but that's what most people call it and know it by. (I love this thread on how "satellite" almost became known as "bird mode": https://twitter.com/btaylor/status/1099370126678253569)
I chose aerial imagery as an in-between - understandable to everyone, still not correct but less incorrect than satellite imagery.
I thought he was walking around. The bicycle helmet thing was an alternative to this method.
But yes, the article definitely needs a more detailed section on the practical aspect of recording.
Usually this is easier with drones, but there are a lot of places you can't fly drones. I hope this guide comes in helpful, especially if you're contributing to OpenStreetMap.
I've found you can strap an iPhone to the front of a bicycle and if you go slowly the 3D models/LiDAR point clouds are quite good (it has a 5m range). There are drift issues if you try to complete loops though.
https://sketchfab.com/3d-models/babakawa-urban-design-projec...
https://sketchfab.com/3d-models/holiday-villa-iphone-3d-scan...
https://sketchfab.com/3d-models/street-art-on-rue-chapleau-m...
https://sketchfab.com/3d-models/day-124-colored-sidewalk-in-...
I live near Waymo and see the cars all the time - they do have huge things on top but the weather in Santa Monica is also very good. Not sure if I have seen them ever driving in the rain?
At the same time advancements also happen at the multi-model side, not fusing different deep learning outputs afterwards but right in the one model (not judging if that is a good idea... but the steps taking currently seem to get bigger and better).
Lastly, also camera has degraded performance under certain weather conditions.. some even say our eyes can be prone to that! :D
So that's just all the challenges that need to be taken. Yes, humans can drive with their eyes only, but is that perfect? Sometimes we e.g. would also benefit greatly from radar/lidar, as e.g. in dark conditions our judgement of distances is really flawed.
Refresh rate of some of this already aged car lidars was like 10 Hz, good enough I'd say? And it is truly amazing what details one can get out of those when getting close and move around in those point clouds.
Each approach comes with different trade-offs suited to different use-cases. In the case of the iPhone rear camera, I believe the approach used is single-frame pulsed Lidar using a VCSEL (vertical cavity surface-emitting laser), a dot-grid diffuser, and SPAD (single photon avalanche diode) CMOS camera.
Basically, a shifting grid of dots at a very specific wavelength is projected outwards from the laser, and an unbelievably, photon-level sensitive camera tuned to that precise wavelength measures how long each dot takes to come back after it's emitted. In this way, Time of Flight can be measured and a precise distance determined.
This system provides incredibly high resolution at a moderate distance (in this case, 5m or so) at a moderate cost.
For a car, this system wouldn't be very plausible due to range, resolution, and output power constraints. The size of laser required to emit a coherent dot matrix which could reflect at a long distance would grow to an extreme degree, the noise floor of the SPAD would eventually render the system useless, and the amount of data required to produce a higher resolution dot cloud would become quite challenging, as these systems generally need to be heavily filtered and sub-sampled, so an enormous amount of input data processing is required to produce a reasonable output.
The main reason that Tesla don't use Lidar is that they decided to bet on optical (IMO, foolish), but the reason they decided to bet on optical is complex and definitely in no small way cost driven. They couldn't just slap an array of iPhone sensors (I believe they're Sony-manufactured) on a Tesla and call it good.
There are several reasons why they would not just add new sensors even if they are in some/all ways superior:
* Tesla special cases different driving situations. This makes sense as a product company seeking to sell consumers on solutions to each situation as a feature but means they need to re-implement each situation individually when the sensors change rather than just swapping out some generic distance detector net. You can observe this from the loss of features when Tesla went full Vision over the last two years.
* Adding new sensors adds new errors which leaves you to chose between ignoring danger conditions present in only one sensor as possible errors or greatly increasing safety stops etc (some of which would be genuine previously undetected dangers). Because of Tesla's engineering culture the prudent second option or a sensible middle ground would be impossible to make since they would fail from a user experience point of view at the first false stop during a demo to the big man and then be forever poisoned (imho).
* Tesla has always been pursuing a full vision only solution on the reasonable assumption that if humans can do it then eventually computers will too. The only disagreement with that strategy is when exactly that will/should be.
There's no way you can argue it was done in good faith when it was an arbitrary transition where some functionality hadn't even been solved yet and was disabled if your car didn't come with ultrasonic sensors. And to add to the comical levels of disregard for buyers, the official advice for finding out if your order would have ultrasonic sensors was: "Look for the 12 sensors on your vehicle", ie you didn't know until you paid and it was a roulette wheel.
You don't know how manufacturing works, especially in a post-COVID world.
Tesla almost certainly couldn't source the ones they were using: That mystical "chip shortage" is a general component shortage.
https://gmauthority.com/blog/2021/12/these-chevy-models-get-...
-
Now it's not like there are no ultrasonic sensors to be had, but they would have had to pay for tooling changes, design verification, assembly changes... easily 10s of millions of dollars for what you're brushing off as "the price of a $10 sensor"
Their options were remove the sensors and be transparent about why, or throw up a smoke screen with idle gibberish about vision was definitely better despite needing to cut features that had been practically solved nearly a decade earlier. Guess which they chose.
> Price was a factor for not using the larger, longer range lidars though since that cost more than planned price of the vehicle ~10 years ago.
Good thing I didn't say or remotely imply that?
Overall I think your type of blustering works better on people not working in the AV field.
No, sorry that cannot be left as is.. I see your arguments but they are pulled of thin air, or is there any source that reasoning actually happened there. If fusion done right, this will never increase but decrease errors.
> they would fail from a user experience point of view at the first false stop during a demo to the big man and then be forever poisoned (imho).
And failing on the first situation that a second sensor set would have clearly prevented can easily be argued the same.. this is a strawman.
Here's a scan of several km (!) of length all done with Dot3D on an iPad [1]. You can also explore the point cloud on the web.
Also, digging a little deeper, here's a study comparing the accuracy and loop-closure capabilities of several 3D scanning apps. It's spot on. [2]
[1] https://twitter.com/netch87/status/1632392369143435264
[2] https://rafaels-blog.weebly.com/blog/why-loop-closure-is-cru...
It's a blog with only one post. Though it claims to be a comparison of tools, it only mentions the name of Dot3D, refers to all other tools by number, so you can't look at them, and is written by someone on the Dot3D team.
Do not underestimate the sheer amount of walking around and aiming your phone that this technique requires. A 5m range is not much, and in some cases the effective range seems to be less than that. If you just want a to-scale image of an area, you can stop after "Capturing the model" and take a screenshot of your captured scan, but it's still a lot of work.
Nothing against the iPhone, but it would be neat to also have (e.g. open source or DIY electronics) projects which enabled use of lidar tech at home-user prices.
It's worth noting (again, without trying to be a grinch - merely trying to add to the overall picture) that there are actual dedicated lidar scanners on the retail market, of higher spec and capability, within a similar price range as the Apple devices involved here.
Of course, you don't get an awesome iPhone with those! :)
I agree! I've heard of "cheap" lidar scanners (Intel Lidar Camera L515 for $375 apparently [1], or [2]) but the act of turning real time depth measurements from a UART output into a 3D model (or even just a coloured pointcloud) is no simple task!
The Intel one does appear to have been discontinued: https://www.therobotreport.com/intel-issues-end-of-life-noti...
Something that may scratch the itch though - I've thought it would be cool to build a bicycle mounted, microcontroller-controlled servo mount for an iPhone which moves it back and forth to increase the the width of scanned area (I assume a sine wave).
[1]: https://old.reddit.com/r/robotics/comments/liudco/where_to_g...
[2]: https://www.leishenlidar.com/product/m10-low-cost-tof-lidar-...
I’d put the table of contents at the very top of the page, so that I know to expect the whole thing.
I’d make the div for the subscribe block a different colour to make it stand out from the rest of the page.
An href link with “continue article…” that jumped down to the next section could help too.
FWIW I almost left the page and missed a really good read because of this issue as well. Glad I didn’t!
I'll improve the background/design of the subscribe block when I have some free time.
I'm not aware of an OSM/open place to store/contribute pointclouds though - please let me know if you come across one!
Anyone everyone else tho.. haha! :) ;p