And bear in mind that fertility is an exponential system, in both increases and decreases. So groups that are removing themselves from the gene pool will do so with a rapidity that is quite counter-intuitive. Like a fertility rate of 1 doesn't sound that insane (it obviously translates to literally every single woman having one child on average), yet it results in a generational decline of 50%, with a generation tending to be around our window of fertility - about 20 years.
So imagine two groups start at 16 people and one group maintains a fertility rate of 1, and the other group maintains a fertility rate of 3. After just 5 generations, about a century, the low fertility group will have 1 person, and the higher fertility group will have 81. You went from equal size to an 8100% difference, after a single century! And that's with fertility levels that are entirely realistic and not just comparing extremes like Nigeria or South Korea.
[1] - https://www.pewresearch.org/short-reads/2017/04/06/why-musli...