Later in years, a realization of one's mortality (an emotional/personal one, not just a theoretical understanding). Coupled with signs of an ageing body -- fractures take longer to heal, chronic diseases start to poke their head, etc.
No matter how successful you are, how much you figured out what you wanted and how well you did it -- there will be something within the sphere you've committed to that is left untried. Eventually it sinks in that some of those somethings will always be left undone. There will be no time for them. This feels like a loss, and letting go is hard.
One other side to this, even within our chosen area(s) of interest that we have worked hard on: Unless we are very unimaginative, we all ultimately must fail, despite many successes. As this sinks in, letting go is hard.
Being able to acknowledge that there are things you won't be able to address in this lifetime is reassuring to me, as it allows me to focus more on the things that "immediately" matter.