I don’t think there’s anything wrong with starting warm and cooling down. The only thing that matters is that you can go cold enough for a decent amount of time.
For me, there is some value in facing the cold water head-on in the morning ("Wow, this is going to be cold, but here goes!" Great way to start the day. I have to admit though, the cold water is not super cold here this time of year.