Hacker News Reader

Comment by hahahaa | Hacker News Reader

Oh yeah I mean how do the phones do that. I guess it uses dead reckoning with visual cues and some machine learning.

"SLAM" is the term to google.

on iPhone 12 Pro Max and later (maybe only on Pros, I forget) they have a LiDAR sensor built in, allowing for impressively accurate 3D scans of objects with a free app like Scaniverse. But yeah I think AR was the intention there

Your phone has an accelerometer, which can be integrated twice to result in slightly accurate dead reckoning; just keep accumulating 3-vectors with every time-slice. That's a helluva lot easier than visual cues.

"slightly accurate" is the correct description, though -- dead reckoning on a cell-phone-class IMU accumulates drift too quickly to keep objects anchored in space for more than seconds at a time. You need something (visual odometry, SLAM) that provides long-term stability, even if it has short-term noise or poor availability, to fuse in.

When this topic comes up, I'm always happy to share this incredible lecture from many many years ago about sensor fusion and dead reckoning and how you _cannot_ integrate acceleration twice:

https://www.youtube.com/watch?v=C7JQ7Rpwn2k&t=1401s (the whole video is incredible, but this timestamp is the magical bit)

12 years ago I had to write the firmware for an IMU. It took me some real effort to convince my bosses that double integration was, for all practical purposes, impossible. I wish I had known about this video back then...

IMU are very useful to known your orientation though. Even an inaccurate IMU is useful to provide an initial estimate to feed vision-based algorithms.

So you tend to need additional data or assumptions to constrain the double integral?

You don't double integrate, period. You essentially have three options:

1) An absolute positioning system like GPS.

2) A visual positioning system like fiducial markers or SLAM.

3) Something that can measure linear speed and then you integrate the speed only once. Easy to do on vehicles, but even the most accurate system will quickly drift.

In my experience, on Android, it definitely relies on camera: AR doesn't work in dark room, and barely works in slightly-lit room. So it must be fusing camera + accelerometer + ...

Reply on news.ycombinator.com