But I am saying that is the wrong comparison. The LLM doesn’t need to implement a human brain directly. It needs to implement a sophisticated enough simulation of people that the simulation itself contains ”people” who believe in themselves.

I don’t know LLMs do that, but they are excellent function approximators, and the laws of physics which allow for my brain to be conscious also can be considered some function to approximate. If the LLM can approximate that function well enough, then simulated humans would truly believe in their own existence, as I do.

And it isn’t really important whether or not consciousness is part of the simulation or not, if the end result is the simulator is capable of emulating people to a greater extent.