We’re not talking about international migration; we’re talking about having different classes of citizens within the same nation state.
The British Isles wouldn’t need to let in an enormous number of people, but they could if they wanted to. The roughly 300,000 BN(O) and 63,000 BOTC passport holders could easily be accommodated.
The UK grew by 14 million people between 1950 and today, that’s net growth of 200,000 per year for 70 years. The idea that Britain is now full is just nonsense. Now that the birth rate is below replacement levels, mass immigration is required just to sustain the current population size anyway.
That being the case, most people are understandably reluctant to leave their homes unless absolutely forced to by war.
In short: all British citizens should be free to migrate around British territories any way they please, but they probably wouldn’t exercise that right anyway.