Recurrent neural nets is the general term for nets with memory as you describe. Indeed LSTMs, a type of recurrent net, used to be state of the art on language tasks until the GPT transformer models. I'm sure somebody somewhere is working to make a transformer with recurrency. The neural turing machine mentioned in another comment is such an example but it seems to have been abandoned.

The main problem with recurrent models is its hard to train them with backprop. For example the GPT-3 can handle sequences up to ~2000 tokens? I'm not sure what the largest sequence LSTMs could be trained on but it was probably less.