Deep learning automatic driving systems that want to work have to have world models.

It's Automatic Driving 101.

These models don't have to be as explicit as formulas but can be approximations of reality through beam search (having multiple steering hypotheses at once and then picking the most likely one etc.), model ensembles, some bayesian state exploration or anything that isn't random search.