It would be kind of crazy if visible ethnic diversity didn't drop consistently generation after generation after the Roman empire moved east, but especially between 1200 and 1700 given waves of xenophobic hate, evictions of the moors and paranoia that anyone with the potential for external connections, (such as connections with the newer hubs of international trade) was poisoning wells.