This actually makes a lot of sense...how many video titles do you remember off the top off your head, versus some background imagery or color that predominates the video?
Unless the colour was quite strange, like eye blinding fluorescent pink or purple, I cannot imagine referencing colour. Or some background image. It wouldn't even register.
Am I unusual here?
Surely you can find it more effectively in your watch history. This feature is to discover videos, not rediscover.
Except you can't rely on that. Youtubers can and do update the thumbnail to A/B test which gets more views.