Actually, I think a test environment could be a lot more complex.
It's mind blowing to me that there doesn't seem to be some big track somewhere with automated crash test dummies riding their bikes across the street in the dark. Automated cars running red lights and stop signs. Automated accidents in front of self driving cars. Automated kids chasing a football across the road.
It's mind blowing to me that governments do not require self driving cars do well on all these test before they allow the cars on the road. Governments should prepare these test, and car technology should complete the tests before they are allowed on the road.
And when a car fails any real world situation, we can step back and create new tests.
Really: We can test to any complexity we want. A plan lading on the road? A flood? A heard of sheep or flock of birds? A dust storm or other weather than might prevent the cars sensors working.