I think I heard of people with brain damage who does not keep an internal state, they feel like they continuously wake up and only remember memories from before the brain damage. That would be a kind of parallel to consider, or at least somewhat analogous. I think most of us assume that such people would have phenomenal consciousness / subjective experience / qualia

It could learn things via the context window. Maybe there'd be no long-term conscious state but you could keep short-term coherence well enough. I think the real problem is that it simply isn't capable of training for self-awareness and reflection because there's only very indirect evidence of it in the training set. We talk about Chains of Thought, and it's a very powerful technique, but this is all just the tiny snippets of text on the internet where people happened to narrate their thought process out loud. If it could learn on self-generated instances of using Chain of Thought to help it predict other text in its training data, this tiny fragmentary piece of consciousness would get massively reinforced.

Exactly - tt has no memory except the narrow window. When these models develop a hippocampus analog and can push things into long term memory, it might become interesting.

It should have a feedback loop from the conversations so that it could keep learning. Maybe by users rating each response.

Then the question is whether it was conscious during training