Older people are not as productive or innovative as younger people. Therefore, there will be fewer things produced and less to go around for "the remaining people."

Everyone ends up poorer.