This is a good point and we would have to understand whether this is noise in the machine or independent reasoning. I haven no idea how to measure that.

If somehow independent reasoning then we're getting very close to something alive that would need rights.

In my view.

And I have yet to have an experience with an LLM where I came away thinking -- this thing is alive and me using it this way is unethical. I've had that experience with animals, even people. Never -- yet -- an LLM.