I would be curious to know how much when we write, how much of it is self-attention and how much of it is our fore-brain actually trying to make sense? My guess is that the more tired / rushed / burned out you are, the % of self attention increases.

Sometimes watching the news, it seems like 90% of what they say when they are 'vamping' is just self-attention.

Has anyone posted any GPT / Hacker News generated text yet? Wisdom of the crowds, indeed. It'd be interesting to post using it with light editing, especially something that uses upvotes for training.

One of the things I was thinking about was training on your favorite novel, so you could have a sort of conversation with it / ask it questions. A kind of interactive cliff notes. However, as looked into it I realized it was still too much of a markov chain like thing to be functionally useful. Fun idea though.

The real win, in all of this, of course is auto completion in different mediums. Code completion demos are pretty wild - https://tabnine.com/blog/deep/ Come to think about it, you could probably use it for writing academic papers as well assuming you know the content well.

Self-Attention and Human/Computer interaction is a very brave new world. I don't think people really yet know the potential for seismic shift here.