IMO, this is a pretty silly idea.
The fundamental problem with this theory is that there are sub-populations that have been able to maintain high fertility despite the decline in broader society.
For example, the number of Amish doubles every 20 years. Today, there are 1/3 of a million Amish. At the current rate, it'll only be 200 years until there are more than 300 million Amish in the US.
No idea if that will actually happen, but if everyone else dies off, there no reason they wouldn't just take over.