The cold side heats up due to the compressive 'forces' (or whatever it's called) AFAICT. Same reason superchargers need also need aftercoolers to run at peak efficiency.
One of those pesky laws of thermodynamics seems to imply that if a turbo only used the kinetic energy of the exhaust it would not be able to overcome the losses of transferring the energy into the intake and would be a net drain on power.
Of course, I'm no thermodynamic engineer and am more likely to be wrong than not.
--edit--
Though, certainly, a turbo is harnessing some of the lost energy of the combustion process since the piston can't do a 100% energy conversion.