I've done some testing on this idea of real-time training for memory purposes using prompt tuning, prefix tuning, and LoRAs. Its hard to pull off in many cases but I've seen it work in surprising situations. I got the original idea when I was testing out the concept of using LoRA swapping for loading different personalities/output styles per batched user prompts. Those were pre-trained, but it occurred to me that soft prompts could be trained much faster to potentially remember just a handful of facts.

The basic idea is that you summarize chunks of the convo over time and create tiny ephemeral datasets that are built for retrieval tasks. You can do this by asking another model to create Q&A pairs for you about the summarized convo context. Each sample in the dataset is an instruction-tuned format with convo context plus the Q&A pair.

The training piece is straightforward but is really where the hair is. Its simple to, in the soft prompt use case, to train a small tensor on that data and just concatenate it to future inputs. But you presumably will have some loss cutoff, and in my experience you very frequently don't hit that loss cutoff in a meaningfully short period of time. Even when you do, the recall/retrieval may still not work as expected (though I was surprised how often it does).

The biggest issue is obviously recall performance given that the weights remain frozen, but also the latency introduced in the real-time training can end up being a major bottleneck, even when you lazily do this work in the background.