A BackwardsGPT would be interesting

I think it would philosophically be extremely interesting to play around with a backwards GPT. That is, a model that is trained to predict the previous token.

Decoder only transformers very much have an orientation, the order of things can be crucial for their ability to compute things. The classic example is that generating a (random) word and then it's hash is (difficult but) computationally feasible. But generating the hash of a word and then the original word is computationally infeasible for a transformer, because to generate the hash it has to internally come up with a random word and then has it, but when it is time to print the original word the decoder only transformer does not remember it (no hidden state). It has to invert the hash.

I claim that it is similarly computationally extremely difficult to generate reverse text for a transformer.

Both if these issues can be resolved with prompt engineering: The transformer can be told to use it's output as memory, so it can generate a random word / forward text first and then hash / reverse it.

Because of this forward orientation all LLMs right now have I am very curious to see how a BackwardsGPT would behave. Many aspects of LLM behaviour were quite unpredictable and I think what we have learned from experimenting with them is philosophically very valuable. Who knows what surprising emergent effects a BackwardsGPT would have. It would certainly be much worse at it's task than a GPT, because I believe previous token generation to generally be computationally more difficult than forward generation (most problems in our lives have an easier forward problem than it's inverse problem).

But as the SGD does its holy work I am sure it will find weights create some interesting behaviour.