The space surrounding you may not have a temperature but your spacecraft does, and it will lose heat via radiation, batteries and electronics will stop working after your temperature reaches a certain low.
Designing for a wide range of temperatures in space where shedding heat is a challenge and running heaters (during dark/cold periods) is necessary, and where at the best of times re-deploying radiators and heat shields is difficult, let alone when the craft is lying on its side.
So ambient temperature matters. As another comment notes, in most cases, the problem is excess heat and the shedding of it. Particularly for Earth-orbiting satellites, which are subject to 1kW/m² solar radiation and are in shadow for only a small portion of their orbit (less with higher altitudes).
Just to be pedantic, it actually does, which is the microwave background radiation. But that doesn't detract anything from your point.