The human body needs to maintain core body temperature at 37°C. Whenever we extract energy from our food to do stuff, some of that energy gets dumped into heat. If we assume (just for round numbers) that we eat 2000 Calories/day, and that 10% of that energy is heat, then that's 200 Calories/day of heat that needs to be dumped into our environment. Abusing units somewhat, if a human being is 100 kg of mostly water, that's 2°C/day of heat. If we didn't dump that heat, in a day, core body temperature rises to 39°C and we get heatstroke.
Heat naturally flows from hot to cold at a rate proportional to the difference in temperature. If we're fully passively cooled at an environmental temperature 25°C at a rate of 2°C/day, then at 1°C, we'd be passively cooled at 6°C/day. However, we can also modulate heat flow by introducing some thermally nonconductive layers--or, in layman's terms, put on a sweater. When it's cold, all you have to do is slow down the rate of heat flow.
But hotter temps are more difficult. We can't take off thermal insulators or modulate the main thermal barrier between core body temperature and the outside world (i.e., skin). At 31°C, passive cooling is down to 1°C/day, and we need to turn to active cooling. Sweating is the main mechanism for humans: it takes heat for liquid water to become water vapor, and this heat is drawn from your skin surface, cooling it (this kind of allows heat to move from a colder to a hotter environment, unlike passive cooling). At 37°C, there's no longer any passive cooling, and active cooling needs to handle 2°C/day.
But hotter than that, and things get worse: the environment is now passively warming you up. By 43°C, that's now 3°C/day of heat that needs to shed, and 49°C ups it to 4°C/day. At some point, you're going to overwhelm the ability of active cooling to cool you down, and the passive heating is still going to warm you up even if you shut down metabolism. Even worse is if sweating isn't effective at cooling down--at high humidities, the sweat won't evaporate and the cooling effect won't happen. (This also illustrates why heat-and-humidity can be deadly even at below-body-temperature environments.)
This analysis was purely done for humans, but it's pretty similar for most organisms, just with different set points for core body temperature and active cooling mechanisms. Fundamentally, extreme cold is "easy" to handle--just slow down heat loss; extreme heat requires developing ways of pumping heat from a hot reservoir to a cold reservoir, which thermodynamics does not look kindly on.