there's no reason to think that we started out with lots of phonemes and that these have been whittled away over time
I suspect the reasoning is more along the lines of languages picking up more phonemes over time, such that older languages, having been around longer, would have picked up more phonemes.