"Colour" is oversimplifying it a bit. We're usually concerned with the distribution of light received with respect to wavelength. In the case of distance, it's not just that "it's red", but that the entire spectrum has been shifted in the direction of red. That means the features that correspond to, say, a specific Hydrogen emission, appear slightly offset from where they would if you were looking at it in a lab. When you look at age and size, you're looking at the distribution of the spectrum irrespective of a shift and seeing if it's heavier in the red, blue, or somewhere else. That tells you a lot about composition and mass, which helps with the age and size questions.

TL;DR: the use of color terms (e.g., "red") is oversimplifying what astronomers actually do which is an analysis of the overall spectrum and its properties.