I am sure the region is stunningly starlit, but if it were Olber's-paradox- bright, would not the dust in the region also be glowing white-hot?

AFAIK, we are unable to see the center of the galaxy with visible-spectrum sensors due to the immense amount of dust and gas. We can see the center with IR sensors, however.