Seems really scary to exclusively use a neural network for safety-critical tasks like this, without having an explicit method guaranteeing safety.

You can't prove that a trained neural network is always correct, and thus this is likely going to kill someone at some point.

I think you definitely need a LIDAR (or in general something that can give an accurate 3D map of all surroundings) and some explicitly written code that can be proven to result in never hitting a car going the opposite direction in the opposite lane (provided they stay in their lane), as well as never hitting stationary obstacles and never going off the side of a mountainside road.