I think that people who argue a perpetual growth requirement are fundamentally doubting that an economy can store (sufficient) value for the future. They are rejecting that life-long earnings can be saved and invested in a person's own non-productive retirement. They are presupposing hand-to-mouth living and assigning roles to the actors in that final scene.
I suppose it is true in isolation. How many years of food, energy, and medicine could you stockpile for yourself to consume in your end years? There are practical limits due to spoilage, etc. If you instead have to devise a way to keep producing for yourself, we might quibble over whether that is really retirement...? It's also easy enough to run thought experiments and demonstrate how a collapsing population has an unpleasant end-game. Like musical chairs, eventually someone loses. The last people to die will die alone, with nobody to nurse them. Before that, systems will collapse, reducing the efficiencies that survivors depend on for quality of life.
But, what are all the phases in between? How long can a society sustainably function at a fixed or slowly reducing size? Could we gracefully transition to earlier modes such as functioning agrarian, hunter-gatherer, or frontier communities? Can those retro communities still enjoy technological benefits developed by their larger forebears? Or do we have to assume war, famine, and post-apocalyptic wasteland?
This seems like more of a psychology test than a useful debate. We can establish our various moods and dispositions, but not prognosticate a large-scale future. At a more personal level, can't we live a good life even if we are the last one to turn out the lights? Don't we all die alone, regardless of the number of attending nurses and medical billing?