Look inside iPad Pro 11s Lidar Scanner | Hacker News Reader

Look inside iPad Pro 11s Lidar Scanner

eetasia.com

148 points_Microft67 comments

This is the mass market application of PrimeSense's Time-of-flight sensor. They were first known by the Structured-light sensor used in the Xbox Kinect camera, but were later bought by Apple and used for Face ID. ToF is a more difficult technology than Structured-light, but the spatial and temporal resolutions are much higher. Rotating Lidar is also ToF, with lower spatial and temporal resolution.

https://en.wikipedia.org/wiki/PrimeSense https://en.wikipedia.org/wiki/Time-of-flight_camera https://en.wikipedia.org/wiki/Structured-light_3D_scanner

Hi, I work in this field professionally - you are correct up until the tradeoffs. It is not the case that the spatial resolution is higher - in fact, semiconductor manufacturers have struggled to get ToF sensors much above VGA in mobile form factors. In general, ToF systems have a considerably lower maximum precision than structured light of the same resolution, especially at close range - typically structured light systems have quadratic error with range, but the blessing that that curse gives you is that at close ranges, precision goes up superlinearly with horizontal resolution. This is one reason that essentially all industrial 3D scanning is based on structured light. The other is multipath inference, which is specific to time of flight systems (and you can see the effects if you look at a ToF’s results near a corner - the exact corner itself will be correct, but the walls immediately adjacent to it will be pushed out away from the camera).

Temporal resolution is more debatable, because ToF is a lot more conducive to “extremely short, extremely bright flash” than structured light is. But for example, there are systems that run structured light (specifically, single-shot dot pattern structured light, like that seen in TrueDepth or Kinect), or its two-camera equivalent (active stereo) at exposure times of 1ms or less. It is all about the optics stack and sensitivity of the camera. So I don’t agree that temporal resolution is a compelling advantage either.

The main advantages of ToF are that it can be built in a single module (vs two for structured light), it does not require significant calibration in the field when the device is dropped or otherwise deformed, and it is easier to recover depth with decent edge quality. In general the software investment required for a good quality depth map is lower, though in this case Apple has been chewing on this for many years. Another potential advantage is outdoor performance at range - while historically this has been a significant weakness for ToFs, more modern ToFs adopted techniques to improve this, such as deeper pixel wells, brighter and shorter duration exposures, and built-in ambient light compensation. These are hard to do with structured light without manufacturing your own custom camera module. Finally - and I suspect this is why Apple ultimately picked it for their rear depth solution - because time of flight is a single module, it can be worked into the camera region on the rear of the device without having to have a separate opening for the illuminator and camera. The quadratic drop in accuracy with range that I mentioned above can be offset not just by resolution but by the distance between camera and illuminator - for a rear mounted device, the temptation is to make that baseline large, but this would put another hole in the back on the other side of the camera bump. I don’t see Apple going for that.

Can you comment on the tradeoffs between indirect TOF (phase) and direct TOF (time), what made Apple opt for direct TOF here, is it Microsoft's patents?

Indirect ToF: easier to manufacture in small form factors, relatively cheap, well established technology. Easier temperature calibration and lower precision required when manufacturing the emitter, meaning cheaper components and more vendors that can make them. Because the technology can only measure the shift in phase, there is phase ambiguity between waves. The way this is dealt with is to emit multiple frequencies and use the phase shifts from each to disambiguate, but you usually only get a few channels so there ends up being a maximum range, after which there is ambiguity (aliasing, if you will) about if an object falls in a near interval or a far one. Multipath can commonly also cause such artifacts in indirect ToF systems. Finally, because they are continuous wave systems, they can (though modern indirect ToFs try to mitigate this) interfere with each other like crazy if you have multiple running in the same area. I’ll note that there are also gated systems that I would characterize as indirect ToF, that use a train of pulses and an ultrafast shutter to measure distance by how much of each pulse is blocked by the shutter. These suffer from more classical multipath (concave regions are pushed away from the camera), and are not very popular these days. You are right to call out that Microsoft is very heavily patented in the ToF space, and they ship probably the best indirect ToF you can buy for money on the HoloLens 2 and Microsoft Kinect for Azure.

Direct ToF is a newer technology in the mobile space, because it has proven challenging to miniaturize SPADs, which are really the core technology enabling them. Additionally, the timing required is extremely precise, and there are not that many vendors who can supply components adequate for these systems. While there are patent advantages, there are also some significant technical advantages. Direct ToF systems have better long range performance, are much less affected by multipath, interference with other devices is minimal, and most critically - you can push a lot more power, because you emit a single burst instead of a continuous wave. This is really important for range and SNR, because all active IR imaging systems are limited by eye safety. For eye safety you care about not just instantaneous power but also energy delivered to the retina over time. Its helpful to recall that for all these active IR systems that go to consumers, they need to be safe after they’ve been run over by a car, dropped in a pool, shoved into a toddlers eye socket, etc - so this puts pretty strong limits on the amount of power they can deliver (and thus ultimately on range and accuracy). Direct ToFs are also really nice thermally, because your module has a chance to dissipate some heat before you fire it again (vs CW systems where you’re firing it a much higher fraction of the time).

Kudos on the explanation. I'd love to see you do a blog pot that elaborates on these different methods with diagrams for laymen like me who find this fascinating.

Why Sony, and other chipmakers they went as far as developing a true ToF sensor, rather than switching to the continuous wave sensing?

And I also heard that Apple went as far as trying to develop a ToF ladar by itself. What's the magic with/what's particular reason to use ToF sensing, instead of CW?

Primesense never made a ToF sensor. When they were bought by Apple they had a Structured Light sensor on the market and apparently had a MEMS-mirror LiDAR sensor in the labs.

Despite the name, ToF sensor typically means using modulated flood illumination and phase shift to compute the depth. It's conceptually very simple but suffers multi-path errors and poor efficiency. Real Time-of-Flight sensors (that measure depth by timing light) are typically called LiDAR. These typically collect many pointwise independent measurements.

The LiDAR inside the iPad Pro and the iPhone Pro is solid state (no moving parts, not even a MEMS mirror). It was developed by Sony and they are now selling it to Apple.

You are right regarding Sony. I thought the Primesense Carmine was ToF, but it was only a miniaturization of their SL. There weren't many high quality comments when I posted, so I filled in what I could remember from years ago. It's nice to see some current industry knowledge!

> Apple adopted the term to describe a new sensor that measures depth

Not to be _that_ person, but someone should maybe inform the author that use of LiDAR has been around for quite some time already... the "sensor that measures depth" is pretty much a description of what LiDAR is used for and this isnt a new application of the term. And sensors to do this already exist so no "newness" in this field. I mean sure, if they were talking about it being included for consumer usage in everyday mobile devices, then maybe thats a new thing? But the wording of that phrase seems to want to credit Apple for something newly coined or invented, which isnt the case

Basic reading comprehension

It says adopted, not invented or created. If you continued reading for two more paragraphs you would have read the author say that the auto industry has also adopted lidar. Is the author implying that the auto industry has also invented lidar?!?

It’s not even close to insinuating that Apple invented lidar. And this article isn’t from Apple, anyway.

Wow this is a fairly harsh reply to the parent. Their point was merely that LIDAR is already a known thing; you would say “Apple adopted the term GPS for their location device” either. Why would anyone need to “adopt” a term that’s industry standard?

I agree that it’s not as if the author claims that Apple invented the term, but asserting the parent lacks “basic reading comprehension skills” is a bit mean, don’t you think?

Mean? No, I don’t think so. Finish reading the article before jumping to make a comment that is contradicted by the article itself.

You are claiming that being verbally abusive in response to a correct claim is justifiable as long as the author eventually wrote something that may be itself derived from the same correct claim.

More concisely: you're way out of line here and doubling down is pushing you further over it

But since when does using a known word for it's intended meaning is adopting ? It's just the right word. They did not adopt the word camera for their camera, it is what it is.

Apple isn’t claiming to have adopted it. The author is.

“Adopted the term” doesn’t mean “adopted the technology”. The former is used to implicate you’re doing something brand new and are naming it.

The sentence is a bit awkward but I think it is phrased that way to suggest that Apple’s sensor is different from other things called lidar which tend to involve scanning with a rotating mirror.

Solid state / flash lidar systems are also fairly established in autonomous systems. Usually the reason most these lidar systems use rotating mirrors is to reduce cost, power requirements, etc.

I almost forgot that Apple bought PrimeSense in 2013 (the company that used this IR pattern technique in the old Microsoft Kinects). It's really structured light depth based sampling which is a bit different than lidar.

Appled used PrimeSense technology to enable FaceID, take a closer look at an iPhone sensor array in the notch and compare it to Kinect - https://www.theverge.com/circuitbreaker/2017/9/17/16315510/i...

Even in consumer devices it's not new. Hobbyists pulled lidars from roombas for a while to wire into arduino-driven toys. Nowadays you can just buy inexpensive (and not very good) lidar hardware on aliexpress and such.

I remember hearing about LIDAR in the 2003 movie The Italian Job.

https://jorgeonline.me/The-Italian-Job

So, we’ve know about LIDAR for about 20 years.

removed

The lasers used are going to be class 1, which is so low-power it can't hurt and doesn't require a label.

Well, the lasers currently in use are approx. 2 mW at 905 nm. They become Class 1 due to the continuous scanning. If a laser beam were to become stuck pointing in one direction, as GP mentions, it becomes Class 3R.

And with the current limitations to stay in Class 1, range is 100m or less, so basically useless on a highway. There is a lot of work going on to find good tradeoffs in power and wavelength to give useful range while staying in Class 1.

You can't win this fight. Soon, LiDAR will be Apple's invention. And besides, it already has a lowercase "i" in the name.

It's similar to how Apple invented the tablet and the smartphone (though, curiously, they don't have any broad patents on these concepts).

>It's similar to how Apple invented the tablet and the smartphone (though, curiously, they don't have any broad patents on these concepts).

Did Apple say anywhere that they invented the tablet and/or the smartphone?

No.

But they could boast, and they would be right, that they invented the modern tablet and the modern smartphone, because all modern tablet and smartphones copy their iPhone and iPad designs (heck, the first Android smartphone came out a year after the iPhone), not the tablet and phone designs that came before them.

> they could boast, and they would be right, that they invented the modern tablet and the modern smartphone

Then why didn't they patent them, given that Apple always thoroughly and excessively patents stuff when they get the chance?

Answer: they didn't invent them.

>Then why didn't they patent them, given that Apple always thoroughly and excessively patents stuff when they get the chance?

Because "invented the modern X", doesn't mean created X as a raw technology (to patent it), but created the version of X most people actually care about from them on...

That said, they do have tons of patents on the iPhone and iPad, and on phone/tablet design. Not to mention they've made the Newton already, a defining 90s tablet...

As for the inventor of the smartphone, that was IBM, with the Simon Personal Communicator. Do you care much for those devices in 2020? Didn't think so, like I don't see people buying TV sets from the inventor of TV, or clothes from the inventor of clothing...

As someone who uses a smartphone mostly for browsing the web and who hates installing another app, I just can't see the big innovation here.

If however, you're a developer selling apps, then I can see your point. Apple took the business model of game consoles and brought it to smartphones. Most people love it, developers love it, and that's probably why they call it an "innovation" even though it isn't.

>As someone who uses a smartphone mostly for browsing the web and who hates installing another app, I just can't see the big innovation here.

Well, as someone who mostly likes raw meat from animals I kill, I also don't see the big innovation with this "cooking" thing.

But I hear it's popular with many....

>Apple took the business model of game consoles and brought it to smartphones.

They also added the multi-touch all-screen UI that everybody else copied immediately, gestures, various sensors and capabilities, native apps, proper web browsing (and not the crappy "mobile" experience of smartphones of yore), and several other things besides. Plus things like the ability to listen your voice messages in your order, or send text messages without the SMS/MMS toll via the same app you use for regular text messaging.

And the first iPhone didn't even had third party apps. People demanded for it to get an app store...

Also true when the iPhone appeared: most people didn't have any kind of smartphone, and the apps they could run were at best Java mobile crap. There were smartphones but they were not as popular, and not as user friendly (I had a few, e.g. Sony ones, there were Nokia etc. Nothing compared to the iPhone or the "modern style" smartphone. And most people had a regular phone or -if they did business- a BB). After the iPhone everybody wanted to get into the smartphone thing...

In the meanwhile they also somehow managed to file patents for rectangular products with four evenly rounded corners[1]. My kitchen cutting board might fit the bill.

[1] https://arstechnica.com/gadgets/2012/11/apple-awarded-design...

This is only true if you are lazy enough to not bother to read the patent or really understand how design patents work at all.

Because you can't patent a rounded rectangle nor is that what Apple did. They patented the very specific design for the iPad.

Regardless, it makes no sense to allow anyone to "own" rudimentary geometric design elements.

And yet, it's common practice, and it makes quite a sense:

"I think most people don't understand what it means that this is a design patent - it's not the same thing as a "regular" patent (a utility patent). Design patents allow a company to get an exclusive right to the form of a functional object so that a 3rd party can't make a different device with identical appearance (well, not legally at least). Almost every company that puts the time into making a distinctive shape for their devices gets one: Microsoft has one for the Xbox, George Lucas got one for Yoda etc."

It took Jim Henson's veteran designers several hours, if not days or possibly longer, to design Yoda and refine the puppet's shape and form.

It took someone 20 seconds in a CAD tool to do the exterior product design on the iPad, using intern-level skills.

There's a difference.

>It took someone 20 seconds in a CAD tool to do the exterior product design on the iPad, using intern-level skills.

You'd be very suprised.

Not to mention, if it was so trivial, tons of other companies would have made the exact same design before, not copy it after it was released.

You'd be very suprised.

Hardly, but that's irrelevant.

Not to mention, if it was so trivial, tons of other companies would have made the exact same design before, not copy it after it was released.

Gee. It's almost as if it's what's inside that matters, and lacking the technology inside the iPad, those other companies had no reason to adopt the corresponding product design.

(Except, of course, in the TV and movie industries, where any number of prop designers did exactly that, decades before Apple.)

>Gee. It's almost as if it's what's inside that matters

Gee, when it comes to design pattens (or product design in general), it's what's outside that matters. Division of labor and all that...

Agreed. Apple doesn’t, hasn’t applied to, and wouldn’t be granted such a patent.

I’ve been using LiDAR to scan rooms and it has been incredibly accurate. Issues remaining are mirrors and water.

Same here. I used an app called RoomScan Lidar on my iPad Pro to obtain very accurate plans of some rooms in my house that we are planning to renovate. Lidar is a real game changer for such applications.

I’m curious what the file types you get out of this system are? I’d very much like a 3D scanning solution that could accurately capture things like molding or other smaller architectural details. Is this system good enough on small parts?

No, the resolution isn't even close to good enough to get that kind of detail at distance. You would need a much more expensive (and larger) device or perhaps some approach using photogrammetry and proper lighting. To your first question, it spits out a point cloud that is then converted to a vertex-based mesh [1]. It's not in an easily exported format by default, but you could convert it into an OBJ (I'm sure someone already has an app or code snippet to do this).

I wonder will Apple move from structured light to dToF for Face ID, I’d imagine that would mean increased accuracy and a reduction in the component count.

So, what's the difference between this and those "real" LiDARS that are used in autonomous cars and robots? Such as https://velodynelidar.com which cost thousands of dollars.

Range and robustness to atmospheric conditions and sunlight.

For me it is a little confusing article, but it sound like Sony now has a chip that can emit infrared and in the same pixel measure the direct time of flight. Am I right?

So props to Sony for making this possible in a, I assume, much smaller chip.

No, that’s not quite the case. FTA:

——- In EE Times’ interview, Sylvain Hallereau, senior technology and cost analyst at System Plus, explained that iPad Pro 11’s “LiDAR scanner” consists of an emitter — a vertical cavity surface emitting laser (VCSEL) from Lumentum, and a receptor — near infrared (NIR) CMOS image sensor that does direct measurement of time of flight, developed by Sony. —-

So an emitter from Lumentum and a sensor from Sony.

Is that the thing that lets you measure stuff with the camera? There's an app on my ipad that does that, though I'm not sure it works using one of these sensors. Bought it earlier this year. The app lets you point at stuff to measure the distance between things and seems to work.

The distance and area measuring tool seems to use the camera and IMU (inertial measurement unit = "accelerometer") as it also works on older iPhones which have no lidar sensor.

AFAIK it’s augmented with lidar data if avilable. It feels more stable on my iPad with lidar than on my iPhone without.

What's the state of the art in low cost (like .. ipad pro pricing or less) sensors for depth that could be strapped to an embedded Linux system?

Are the best options still just stereo cameras or structured light solutions?

Do any other smartphone shops plan to add lidar to their android models?

Many Android smartphones are available with "depth sensors" which tend to be useless. I'm not sure what the difference is between those and the iPhone lidar sensor but it has to be officially supported by Android if it's to be useful.

IIRC LG had a "laser" on one of their G phones a few years ago to improve autofocus speed.

I had an Asus Zenphone AR. Used it to 3d scan environments, worked quite well, until the screen died (they all have this defect)

The Note 10 5G and the Huawei P30 pro were the first with ToF on the rear in a mass market phone, I believe.

Does anyone have specifications for the field of view that the iphone lidar sensor operates on?

I don't want lidars mapping everything all the time, especially on systems I don't have root on... add one more sensor to the surveillance engine list I suppose.

If that’s genuinely how you feel, you’re best off not carrying any sort of mobile phone at all.

(Or at the very least, keep yours in a faraday bag until you need to use it.)

FWIW, it is also how I feel, and is similar to my thoughts on microphones and location logging and pretty much everything surrounding social networks; I still use iPhones and Facebook as the alternatives--not for any legitimate reason correlated with this property (it isn't a tradeoff or anything)--are much less effective due to vertical supply chain monopolization and network effects from the major oligopolies in hardware and service providers which would make me less effective, ironically, at my attempts to educate people about the downsides of this technology and achieve regulation on its usage and deployment... to me the real question is why you _don't_ think this is a problem :(.

> I still use iPhones and Facebook

iPhones I think you can legitimately assume are at least trying to allow you control of your privacy.

But Facebook? Not only are they actively trying to scrape your personal data, they're also so easy to drop entirely.

Fun fact, its not that hard to block those sensors and still use the phone... This sort of servile apathy is pathetic to me.

Its too late to delete but I apologize, though the comment wasn't intended as a personal attack and was more intended as a general lament, rereading it didn't sound like that. I should have taken the opportunity to get into the issues I have with the prevalence of this attitude more in depth. Sorry.

Reply on news.ycombinator.com