For example if the car in front did not even slow down while going through some obstacle (the proverbial plastic bag), then it's very likely that we don't have to either. But if something looks like a drunk guy trying to cross the highway, then it's caution time.
That said, without the actual data/studies/access on what Tesla does and how, it's hard to say how good they are at this. (Sure, in theory everything needed to drive a car well can be inferred from regular human vision visuals. But that means their model has to be able to do counterfactual reasoning - I see this thing as a wall, but other cars go through it, so it's not a wall, so maybe it's a marking. Otherwise it'll very hard to just figure out things frame-by-frame and simply by raw object detection.)
We are very, very far from that. And having such a developed visual processing system is a prerequisite for feeding an executive processing system that can do counterfactual reasoning.