Yes, there are several factors that make that location particularly good, not only temperature. The cold and absence of sunlight for months at a time causes a relative absence of water vapor in the atmosphere during winter, which is extremely important for CMB observations. The high elevation (~10k ft) means there is less atmosphere to look through as well. During summers the sky heats up, contains more water vapor, and reflects so much more light that our telescopes become substantially less effective. Another advantage to the South Pole is the ability to continuously observe the same part of the sky, as the telescope is located on the earths axis there is no rising and setting of sky overhead. There is also a particularly 'dark' patch of sky in the southern hemisphere called the 'southern hole'. This is located up and away from the plane of our galaxy and contains very few objects standing in the 'path' of the CMB which also helps obtain better quality observations of the CMB.

Edit - it is as good as it gets on earth and way cheaper than a satellite! Other benefits are rapid upgrades and repairs with newer technology. A satellite, by the time it is deployed, is already pretty old! But the South Pole still isn't cheap!