A country's "number of people" isn't an independant parameter that you can just slide down so that your GDP/person increases.

If you reduce the "number of people" (i.e. by reducing immigration) you are typically reduce your GDP at a much higher rate, and thus will have a lower GDP/person overall. Your eating into your economy's ability to support the division of labour, specialization and other efficiencies that can only emerge at a large scale. Intuitively, think of it this way, do you think a village of 10 people, or a city of 2 million people will produce more GDP per capita?

Going back to the example of immigration, you're basically reducing a stream of young, educated workers who are consistently net creators of jobs, especially in Western countries with a very high ratio of aging senior citizens (Boomers) to young workers.