This is a really important problem to solve, for more than just self driving cars. We have no good way of controlling AIs right now. Particularly reinforcement learners. All we can do is make a fitness function and hope they maximize it. Often they will find a way to exploit it in a way the programmer didn't expect or desire.

And even making a fitness function is a pretty difficult task. You can't exactly let a self driving car get into thousands of accidents until it learns not to do that. Making a good simulation is extremely difficult and very expensive. Training it to mimic humans means it will always be worse than humans.

As AIs get smarter this will become more of a problem. They will become more clever at exploiting the reward functions and even fooling human supervisors. One could imagine a self driving car causing accidents indirectly in some way. But it doesn't care because the car itself didn't collide with anything.

In the extreme case, AIs much smarter than humans would be extremely difficult to trust. They could figure out how to manipulate humans to get them to press their reward buttons or whatever equivalent. The famous thought experiment is an AI in a paperclip factory that desires to maximize paperclip production. So it invents nanotech, tears apart the Earth, and eventually turns the entire solar system into a massive paperclip factory.

Perhaps that's a long way off. But it's not comforting to know that AI ability is increasing faster than our ability to control AI.