I did some thinking along exactly the same lines as you, long ago, and asked the same question: if you want to live forever, but your capacity to retain memories is necessarily limited, you have to basically forget. If you only remember the last 1000 years, then you're only living to 1000 in some sense and your experience as a whole is made up of numerous life spans that do not overlap at all, experientially. So what is the difference between that and death?

The answer I came up with: quite simply, the difference is that you don't ever die. Old memories fade away and are forgotten. But we have experience with that already in our limited lifetimes and a lot of it is not too horrible, compared to dying. If it's slow and gradual, and you always retain the recent memories that are relevant to your current life, this "continuous dying" may not be such a bad alternative.