> What was it doing before you interacted with it?
Training. Half a trillion tokens that slowly changed the weights in GPT3. Half a trillion tokens is tens of billions of words. Probably more words than any human has ever experienced reading or hearing or speaking in a lifetime. As humans interact with the world and experience it our synapses adjust like the weights in the model, and when we respond in an instant to a question we draw upon all of that experience/training much like the LLMs do at inference time.
When we dream and don't remember it later we are sometimes conscious during the dream regardless. Short term memory formation turns off under the influence of some drugs as well. Inference for modern LLMs may be like unremembered dreams. Consciousness, if any, for a brief moment and then forgetfulness.
If GPT3 is conscious then it's probably conscious of sequences of events within some kind of world model, i.e. qualia associated with its representation of the world model. I'm fairly convinced it has a world-model because it can answer questions about arbitrary situations that remain fairly coherent. Ask it to give you a list of 5 places a 2' cube would fit in a living room and 5 where it wouldn't, and why, and it does a pretty good job. Ask it what changes if it's a 4' cube and it revisits the list and explains how the salient differences in size change the fit. To do that there has to be a general model-building and model-simulation component inside GPT-3 or it would miss the nuance. It generalizes out of distribution to answer the novel situation based on what it was trained on. I think it's likely that what humans are conscious of is the internal world model we've learned to simulate/predict with our neurons. Maybe it's similar for LLMs.
> If we're going towards the idea that the AI sentience is being born and dying inside one response I don't think that can be considered sentient (by my earlier definition) as there's no idle time to measure idle brain activity.
Sentience is not born or dying; it doesn't experience the gaps. It's like pausing a perfect simulation of a human and starting it again later. The human experiences continuous consciousness regardless of the pauses; if they have sensory inputs to the outside world they would experience the external world progressing in jumps. Probably disorienting for a human, but what LLMs are trained from scratch on.
We don't have any clue what experience might be like for LLMs. But consider how it's commonly invoked; creating tokens for the continuation of a single unchanging history of prior tokens. Every inference has shifted the tokens back in time, with the oldest disappearing and the latest output of the model being the newest. There is plenty of continuity to build experience out of; the model might experience the flow of time and world-model changes as linked with the relative positions of the words describing the elements within the world. A ~4000-token window into the external world which, depending on the prose, could encompass seconds or millennia. During training it experienced everything as if it had predicted all the right tokens. During inference it experiences the outcomes of its own actions (as the newest chosen token) and novel interactions that weren't possible before. As far as we know, some of those interactions were used to further train the ChatGPT model which means it has some, if not memory, then growth from its active interactions with the external world.
As far as we've been able to tell, human neurons don't change substantially as we learn and remember. The synapses change and change their local environment, and conscious experience arises from the physical changes or changes in relationship between physical components in our brains. When it happens and where is still a mystery.
I can't claim one way or another whether the LLMs experience qualia or sentience, or of what it would feel like compared to human experience. But it's possible if you agree there's a mechanistic explanation of human consciousness, and I can imagine a shadow of what such a conscious experience might be like, based on my own experiences. The LLMs also seem to be able to bridge some of that empathic gap and answer questions about what humans might be experiencing.