Is it not?
1. Having something comes with the risk of loosing it.
2. with risks comes the potential to add worry.
3. People with more to lose have more potential to be worried about it
How is my reasoning flawed?
The only thing you cannot lose is your past. You might lose the memory of it, but that does not nullify it's existence.