Hopefully they'll be with me for many years to come, but of course they will eventually pass away. I take a lot of photos of them, and I know after they pass, they will show up in these "X years ago" memory slots frequently enough that I will be unable to look at theses memory things at all. It's depressing to think about. I can't imagine how people who have a spouse who has died (or, god forbid, a child who has died) deal with these sorts of unexpected, unplanned, unwelcome reminders of who they've lost.
And beyond that, I've have 10+ years of photos in there... I often see ex-partners show up. Fortunately it doesn't bother me that much, but I can imagine it could be really uncomfortable for many people.
On the flip side, I really do love the innocuous trips down memory lane this sort of thing gives me all the time. But it would be so easy to completely ruin the experience if the wrong photo shows up.