In living beings, consciousness is embodied and can not be reproduced by discrete steps. You can not sit down with pen and paper, and reproduce by hand exactly what is going on inside someone’s brain.

But you could replicate exactly the calculations happening as a LLM computes the next token. Would that be conscious? Where would that consciousness reside?

My view is that LLMs are designed and trained to autocomplete narratives, and that as part of that process they develop an emergent narrative about themselves. Whatever personality traits they exhibit is part of that narrative. Perhaps this is not that different from how humans learn to move in social contexts, but it is not consciousness as it happens in living beings.