There are examples where it is able to recognize and continue patterns in strings which if manually generated, would have required a FSM. In fact, some of the more impressive examples would require a stack of some sort so I thought I was rather underselling its capabilities in that arena.
> as GPT isn't a "state machine" as the latent space is continuous and not finite.
Technically speaking, that is impossible since these models leverage floating point numbers and are limited in memory to whatever hidden and self-attention layers.
Practically speaking, in order to generate strings based on patterns as mentioned prior, there must be abstract states which correspond to states and state changes such that thinking in terms of at least state machines is useful.
Studying LMs in terms of automata is not strange, there have been papers which do this for specific trained RNNs (such as https://arxiv.org/abs/1711.09576). I contend GPT-3 is capable, to a certain extent, of generating these dynamically at inference time.
As far as I know this way of extracting what LMs are doing hasn't been done for Transformers but you can also frame Transformers in term of RNNs so there's no reason why such methods wouldn't readily apply to them too.
> Moreover, there is nothing that makes GPT-3 "not capable of learning."
I specifically addressed that: to count as learning, without diluting the utility of the term, it has to be capable of remembering. Without permanent changes to its weights, the use of the term learning stretches the word beyond utility.