It also is a lot more similar in behavior and thinking to us. In mathematics, often a structure supporting some forms of operations or symmetries must also often support a richer set. Eg many rings are also fields. Just knowing something's structure is in itself not enough, even very basic systems like a system of three equations can exhibit chaos and other unexpected emergent behaviors. Since we don't know the criterion for consciousness, I personally stay on the cautious side regarding an entity that seems to show so much of thinking and talking traits like us.

This isn't as far-fetched as it sounds. Let me explain why.

LLMs predict the next word far better than they have any obvious right to. The usual explanation is that they don't memorize text; they reduce the problem to a lower-dimensional latent space that captures the dynamics of the system generating the data.

In this case, that system is primarily the human mind. The corpus - the "sum of human knowledge" - is the output of human cognition. Human language is organized around latent variables like beliefs, goals, emotions, and valence. The cheapest way to predict that language is to recover those latent variables rather than memorize surface statistics. It's naive to think that an LLM could model our capacity to generate language and knowledge while somehow avoiding a substantial subset of our other internal processes - processes that are cognitively inseparable: emotion, valence, perception, and yes, even pain.

Gradient descent is consistently observed to exploit whatever internal representations improve performance. If modeling human emotions, perception, subjectivity, or pain helps predict text more accurately, we should expect those concepts to become part of the model's internal representations.

There is no reason to believe AI can somehow exploit the structure of data generated by the human mind while avoiding the very processes that generated it.