I've seen this sentiment before and I think it misses the mark in a few ways.
- People often value specific memories with a person. We can give AI memories but that takes recording them down. Stuff like Rewind.ai can close the gap, but anything that records you in enough fidelity to serve this function is going to need your cooperation.
- People aren't the same person to everyone. You need to intimately know the person who you're communicating with to correctly emulate a person. Lou Reed wasn't the same person to a random stranger asking that chatbot a question as he was to family. It's not enough to just record the person replicated at that point, you suddenly need an intimate understanding of the person interacting with the replica.
- As AI gets more advanced, why bother with lossy replications of people you knew? If grief is the underlying problem, why does an LLM so advanced it can replicate an entire human not just as readily help address the underlying mental health/social needs that are driving you to replicate a human?
Even for nefarious goals: why bother trying to copy your grandma, this hyper-intelligent AI that can replicate grandma can probably look at your social media and invent a new persona that's able to manipulate you pretty readily...
In a way fidelity is the biggest hurdle for "AI as a person", and you can generally do better on that by abandoning copying real people, and instead inventing people who fill the same underlying gap that the real person was supposed to. Equally dystopian, but not quite as bothersome when it comes to worrying about our own likenesses.