Another factor is if they're all from the same (sub)discipline. It doesn't take a lot of established researchers moving to shift a field's center of gravity somewhere else. When you start seeing research conferences that used to be in the US be held elsewhere instead, you'll know that the change has happened.