I would be interested to play with a long term memory plugin. It could be a note-taking system that would summarize prior conversations and pull their context into the current conversation through topic searches. This would enable the model to have a blurry long term memory outside of the current context.
I played with some prompts and GTP-4 seems to have no problem reading and writing to a simulated long term memory if given a basic pre-prompt.