When you are barreling down a highway at 65 miles per hour and are not paying attention (and you wouldn't, because the car drives itself just fine), giving you controls is much more dangerous (for you and others around you) then not.

Urmson talks about it here: https://youtu.be/Uj-rK8V-rik?t=14m3s