you say obviously doesn't. These language models do indeed work by computing a distribution over all possible next words given the previous words using transformers, and it seems using enough training data and compute gives you the results we see. Everyone I know is completely surprised that it works so well by just adding more data and compute (and probably lots of training tricks)..