Dr. Mary Cummings has been working on talking to NHTSA about implementing V and V for autopilots/AI in unmanned vehicles for a few years now. She's also been compiling a dataset exactly like what you are talking about.

I think the idea is to build a "Traincar of Reasonability" to test future autonomous vehicles with.

You might want to check out her research https://hal.pratt.duke.edu/research