Hacker News Reader

Comment by bitL | Hacker News Reader

Can it manage 4k @ 1000fps for self-driving inferencing? If so, it could solve most of the perceptual problems we have these days with self-driving cars.

View on HN

What?

What about dynamic range, exposure control, color information? What happens when the sun is on the horizon and then the car enters and exits a cool-white lit led tunnel so the entire scene briefly goes black and everything changes colors?

There are tons of huge problems in this space these benchmarks wouldn't be meaningful at all for. 1000fps and 4k might be some of the least meaningful benchmarks I have ever heard someone discuss in the realm of self-driving.

Doesn't that depend on hardware? If you have a proper HDR camera that can handle overload of light and records enhanced spectrum without clipping information (those exist), I can't see how does it affect the underlying perceptual algorithms themselves and how would that make those problems suddenly unsolvable. Having a very high sampling frequency allows you to do reasonable decisions on millimeter scales instead of yard/meter ones with lower sampling frequency; if you have some clever voting/heatmap/etc. mechanism in place, you'd get higher certainty that likely scales well with further increase of sampling frequency. So the perception problem is "solved" because a simple increase of frequency eliminates the need of applying more sophisticated methods and gets you to required accuracy demanded by safety protocols.

> proper HDR camera that can handle overload of light and records enhanced spectrum without clipping information (those exist)

_No_, they don't. CMOS sensors are consistently limited by readout speed (bits-per-second off of the sensor), so FPS, resolution, and dynamic range are fundamental tradeoffs. The bleeding edge of HDR, high-framerate, high-res sensor are full of dirty hacks here like PWL and hardware bracketing to achieve what they do, and it is nothing like your quoted figures.

1000 fps limits your exposure time to 1ms unless you are achieving that 1000fps with sensor arrays, which adds tons of complexity to still sensors, let alone trying to get a reasonable picture from a fast moving vehicle. There is a reason if you ever look at a high-framerate camera setup they are using like 3+ studio lamps to throw enough light on the subject.

This "high sampling frequency allows you to do decisions on millimeter scales" is speculative nonsense, and does not even try to address camera spacial frequency limitations that would prevent this.

I work on HDR CMOS cameras for the automotive market as my day job. I would love a link to one of these enhanced spectrum cameras you speak of.

Sorry, but this is a very uninformed opinion. The problems with perception go way deeper than cranking up the resolution and fps.

Why is that resolution and that speed special? I’d expect the biggest problem to be learning with fewer examples, given what people have written about the few lethal accidents it has had, specifically that it didn’t recognise stationary vehicles as vehicles “because that’s rare“.

1000fps+ allows you to have a larger sequence of decisions "to vote" for the next action taken with higher certainty and lower decision latency. Imagine you have a 5fps self-driving car system; if you drive fast, one frame in 0.2s must give you a correct decision for a quite large travel distance; with 1000fps you can chain e.g. 10 decisions in a row via heatmap or Bayesian rule and get a pretty good certainty about what is going on around you and what action needs to be taken right now. In perception you hit false positives/negatives all the time but appearance of the same issue on two-three consecutive frames goes to 0 pretty quickly.

I... Don't think that's how this works. It helps, for sure, but not nearly as much as you think, because consecutive frames in a video are not iid rvs

Does it really matter? UKF & PF work quite well even without any IID RV requirements. Where can I learn more about this relationship in the aforementioned context?

>In perception you hit false positives/negatives all the time but appearance of the same issue on two-three consecutive frames goes to 0 pretty quickly.

If the frames are all very similar, wouldn't the probability of failure be correlated? In other words, if I feed a picture of a panda to a NN, and it incorrectly classifies it, why would it be correct the second time around?

Sure, but I bet Andrej Karpathy's team is using SOTA and mitigates most of the issues like the ones you mentioned. OK it's an "argument from authority" one might believe or not and I doubt they'd ever disclose things that make it work in their specific case. But realistically, if you e.g. observe results of real-time semantic segmentation, you see that surfaces flip a bit in each frame but mostly stay correct; you can e.g. use average IoU/coverage from past n frames to estimate what is going on. They have a fleet of cars driving around receiving more training data in all kinds of environments to handle various perceptual conditions as well.

>Sure, but I bet Andrej Karpathy's team is using SOTA and mitigates most of the issues like the ones you mentioned.

Can you give an example of an image classification technique where classification failures are independent, or independent enough?

Time-distributed 3D convolutions/attention-enhanced 2D CNNs, i.e. you look at a sequence of images at once, not at a single image.

> In perception you hit false positives/negatives all the time but appearance of the same issue on two-three consecutive frames goes to 0 pretty quickly.

This does not sound like a technology that can be used to produce predictably safe results.

Reply on news.ycombinator.com