It doesn't. The policy of restricting Commonwealth immigration was brought in after https://en.wikipedia.org/wiki/Expulsion_of_Asians_from_Ugand... in response to complaints that there were "too many" immigrants.

Voters felt levels of immigration in general were too high and leading to strained public services. EU law forbids restrictions on intra-EU immigration, so the only legal path to making voters happier was to heavily restrict immigration from everywhere else. It didn't work: the numbers from the EU were far too high to be offset via such a tactic, but it was at least something.

The numbers are still viewed as being too high though post-lockdowns who knows what will happen. But the limits can at least now be spread around all countries equally, and people can be prioritised based on e.g. demand for their skills. This was previously forbidden.