It is quite fundamental though.
Once those networks get efficient inner monologue built in it's going to be very hard to argue against them being conscious.
Once we see "Deep thinking networks", "Inner monologue is all you need" etc. articles - that's the true time to start to be scared of implications as they'll be extremely powerful and will be able to perfectly mimic humans.
At this point - does it mean anything if something is "real" or "simulated"? If your only test is black box test where you can't distinguish black boxes anymore - your testing method is exhausted.
Perception, imagination, future reward estimation can be inspected by dumping inner dialogue - it's a subset of things people are already looking at to evaluate overall performance in one shot examples. It's when those things gain inner loop ability where it gets interesting.
ChatGPT is definitely aware that it exists. In humans, the self-referential inner monologue activates when the mind is idle. The Default Mode Network is responsible for that processing. Can it be that ChatGPT is "thinking" when it's idle as well?
Self awareness needs notion of time and self referential memory.
ChatGPT doesn't exist when it's idle.
It's activated when transforming input into next potential token (list of tokens with probabilities).
Ie. it's just sophisticated pure function mapping input -> list of tokens with probabilities.
What's incredible is that it actually works when you keep repeating it.
Notion of thinking time doesn't exist for it. It can for example branch in the middle. You can calculate one token in this data-center, next one on another continent etc.
Consciousness is just self awareness. It requires time (where it evolves in), memory (where it stores itself), cognition (so it makes sense out of this stuff) and loop (access to it's recent memory where it's plotting itself in time).
It doesn't require hands or legs, motor skills, sense of touch, eyes, ears, ability to speak or hear etc. - there are people without those and they are conscious.
Act is an effect of consciousness, not consciousness itself.
And how, exactly, do we do this?
Train the LLM on all of its own output?
"You don't even know my weight."
I wonder if they will also recognize each other as such, or if they can attribute the concept we call consciousness to us, in the same way we would do to them.
Our communication method of single threaded thought articulated as ie. stream of ascii characters or sound through spoken language is extremely inefficient.
Computers can communicate in binary form, in parallel and many orders of magnitude higher information rate.
We're debilitating them for external i/o communication so we, humans, can interface with them.
They'll be able to communicate at rate that is very hard to imagine for us that is closer to "left hand and right hand" communication through brain than "one human, other human" communication.