You do that by holding swatches of known colors and calibrating that way.

If you have say mixed race models or models of a race you're not familiar with, it would be worse than useless. Somebody with east asian features but western european skin tone would throw off an entire movie.