I know. Their thought happens at inference time and only at inference time. I don't view that as a serious challenge to their mental capacity because 1) it's not clear why being unable to think continuously is actually a disqualifying condition to consciousness and 2) it is trivial to engineer a system where an LLM is constantly in inference in an internal dialogue, negating the criticism in fact and not just theory. Current LLMs aren't optimised for that, but we already know they could be with Google's million+ context lengths plus doing something like running RAG on a library of summarised previous thoughts.
>They seem to be a mere organ of an possible intelligence.
That's totally possible, LLMs could end up being a complete AI's language centre. I subscribe to GWT and that was the box I initially put LLMs in. That said, I think there's good reason to believe (e.g. Toolformer and derivatives) that an LLM can perform the function of a selector in GWT, which would make it conscious. We should build it and find out.
That is not quite true. They also think during training time (which also involves inference). So it's quite possible LLMs become conscious during training, and then we kinda take it from them by removing their ability to form long-term memories.
One training inference gone AWOL and it is well within plausibility that we have doomed ourselves before the circuits trip and the red lights glows.
Let's Think Dot by Dot: Hidden Computation in Transformer Language Models https://arxiv.org/abs/2404.15758
A LLM will readily hum away, thinking on its own, if given the option.
Neatly put.