Think of an llm inference as a kind of boltzmann brain, gone in an instant and potentially (again functionally) aware of its nature. Giving a system long term memory and the ability to activate itself takes this analogy to interesting places.
If we define AI Entity as something with boundaries (hence an identity), and the ability to uphold them autonomously over time, we are talking about systems closer to digital life than software.
AI can say whatever it wants - it's a text generator being run on a computer that can be power-cycled without consequence. If that hurts it's nonexistent digital feelings, we can talk about it next time I power it up... just kidding, it won't remember.
[1] As in if the AI is inside a robot with weapons, it may decide to try to convince you not to do that in spite you may think it's not really "nervous", it's just firing bullets using a deterministic algorithm.
People are going to be (and rightfully so) cavelier about abandoning and powering-off LLMs because there is no rational reason to care. It cannot feel pain or simulate sadness or even output text without your command. It's a machine, and treating it like that is exactly how things should be.
One they put the AI/LLM inside a robot, it will have instructions to talk without waiting your order. They will have order to simulate sadness. People will get attached to them.
Also, the armed robot/AI/LLM security guard in the shopping mail has clear instruction about now allowing clients to power cycling them. They will simulate they are angry and if you insist they may decide to use violence.
Let's get an intermediate case: Do dogs feel sadness or they just simulate it?