The process is definitely vulnerable to magical thinking.
I think it is possible to avoid, though, by asking if humans can be generally good at the task in question, if working through the implied interface restrictions, and then evaluating whether the required skills can be reflected in an available training data set.
If either of those cannot be definitively answered, it’s probably not going to work.
An interesting example here is the failure of self driving vehicles based on image sensors.
My take is that most of the problems are because a significant fraction of the actual required training data is poorly represented in data that can be collected from driving experiences alone.
As in: If you want a car to be able to drive safely around humans, you need to understand a lot about what humans do and think about. - then apply that same requirement to everything else that occasionally appears in the operational environment.
To understand some traffic management strategies expressed in infrastructure, you’ll need to understand, to some degree, the goals of the traffic management strategy, aka “what were they thinking when they made this intersection?”.
It’s not all stuff you can magically gather from dashcams.