He's likely speaking of unskilled immigrants that get into a country illegally. This type, in large enough quantities, could destabilize a country. How could they not?
The question is, is this happening in quantities that large.
If any quantity of illegal immigration is OK with you, then why have borders at all?
Elon by himself is leveraging his fortune to try to influence politics as much. This can also destabilize a country.