You just described the problem nicely. The "magic" that turns 3 lane roads into 2 lanes etc, is a situational awareness that is really, really difficult to impart on a learning system. The big problem is that probabilistic models don't have a notion of "common sense" solution to an odd situation. They need to have seen the situation, or something very similar to it, enough to make a reasonable calculation of what to do.