You might be on to something there. I wonder if it was a vision-only version.

Anecdotally, my wife and I used FSD out in Oregon while on vacation a year ago and had these these "phantom brakes" occur two times on the same stretch of road. It was very sunny day through the gorge to Hood River from Portland and I'm thinking there definitely would have been sharp contrasting shadows from the trees and sunlight.