It absolutely does not. Going backwards in time like this,
most people will have many generations where the number of their ancestors in that generation go down, not up.
For the majority of the past four thousand years, almost everyone lived in farming villages that had ~50-100 families living in them, and mostly married inside those communities. Even just going to a neighboring village for a bride was something that only happened occasionally, but even when that happened, it was almost always the same few neighboring villages. This means that when you trace ancestry backwards, you always end up related to everyone living in a few villages, and then every few generations this extends to a new village. Then if you are related to a successful trader or soldier or something, it will suddenly double the amount of ancestors in a generation, but I really need to stress that this is rare.
The introduction of the bicycle did a lot for reducing inbreeding in farming communities, because it was something that a normal people could own, and allowed day visits to neighboring communities. Before that, it was entirely ordinary to live and die without ever traveling more than 10 miles from the place where you were born.