Yes there is a difference in South India. The Brahmins are descendants from central Asia[1]. But this is often refuted by the Brahmins in India as they claim to be from an authentic Indian origin. The natives in South India which includes me are darker skinned than the Brahmins. So people here can figure out that someone is from a caste if they are light skinned. A person with proper scientific understanding can explain in more detail but this is the overall view.

[1] https://economictimes.indiatimes.com/news/science/steppe-mig...