This is true, but the fact that heat absorption scales with the surface area is pretty brutal for tiny cooled objects.
And you can't just make the walls reflective once the cold object gets smaller than the wavelength of the radiation. The colder the object, the longer that wavelength.
Souce. I have a PhD in physics where I used equipment cooled to 4K.
To be useful in a data center you could cool a slab of copper the size of a fridge and surface mount thousands of chips on it.