Yes, there are tradeoffs to be made, but having to reconcile multiple sensors is not intrinsically a negative.

But also, if you didn’t get the right result, I don’t care how quickly you didn’t get it.

The latency from video capture and recognition is going to be so significant that it makes all other latency sources not even worth mentioning.

Cars and roads are built for human reaction times. That's why you have low speed limits in urban areas. You can have a pile of latencies attributable to processing a scene and still have superhuman reaction time that contributes to outperforming humans.

It's analogous to communications latency. High latencies are very annoying to humans, but below a threshold they stop mattering.