Latitude is one factor but not the sole factor, proximity to oceans/seas for example matters as well as it has a moderating influence in both dimensions - not a coincidence that the coldest states in the continental US are all northern and in the center of the landmass.

Other factors are things like prevailing winds, mountain ranges, altitude and so on and on - the climate system is one of the most complex systems on the planet and even with decades of heavy study and insanely fast computers we still struggle to predict it accurately out past a week or two at most with any degree of success.