Look back a few hundred years and you'll find that the country you grew up in, in Europe, was constantly in that situation. People moved a lot back then too. And the countries are today.. the countries. It'll be fine.
And places become more boring if the have the same migrants as every other place, not less boring.
Since the Anglo-Saxons, we've had numerous intense bursts of migration driven by the first and second Viking periods, the Normans, the Hundred Years War, Black Death, Border Wars, Plantations, Thirty Years War / Wars of the Three Kingdoms / the Huguenots, Plantations, Colonialism, Inclosure, Clearances, Industrial Revolution, Famine, New Colonialism, World Wars, and the Commonwealth.
Yes, the Brexit migration wave was particularly sharp, but the movements in the 1850s-60s were proportionately greater (albeit spread over a dozen years rather than just five).