There's a lot of people on this rock. Even if only a "small portion of the population" can do the work, that means there are tens of millions (literally) around the world who could do your job as well or better than you.
You are the lucky beneficiary of restrictive immigration and employment policies.