We have an old saying "te rafte pika e diellit", may you get a spot, sun spot, aka cancer from the Sun.
People knew excessive sun is unhealthy.... if you could you avoid it during the summer. It was not necessary to make you look 'rich'. Ability to avoid it, meant you were rich. And people wanted to be pale, because you look rich, that is a second order effect, that came because of the original reason (aka, too much sun is unhealthy).