Comment by gwern | Hacker News Reader

Comment by gwern | Hacker News Reader

Parent Full thread

You can do this with dynamic evaluation (https://gwern.net/doc/ai/nn/dynamic-evaluation/index): literally just train on the data as you are fed new data to predict, in the most obvious possible way. It works well, and most SOTAs set in text prediction 2010-2018 would use dynamic evaluation for the boost. (And it has equivalents for other tasks, like test-time adaptation for CNN classifiers.)

In fact, dynamic evaluation is so old and obvious that the first paper to demonstrate RNNs really worked better than n-grams as LLMs did dynamic evaluation: https://gwern.net/doc/ai/nn/rnn/2010-mikolov.pdf#page=2

Dynamic evaluation offers large performance boosts in Transformers just as well as RNNs.

The fact that it's not standard these days has more to do with cloud providers strongly preferring models to not change weights to simplify deployment and enable greater batching, and preferring to crudely approximate it with ever greater context windows. (Also probably the fact that no one but oldtimers knows it is a thing.)

I think it might come back for local LLMs, where there is only one set of model weights running persistently, users want personalization and the best possible performance, and where ultra-large context windows are unacceptably memory-hungry & high-latency.

I've done some testing on this idea of real-time training for memory purposes using prompt tuning, prefix tuning, and LoRAs. Its hard to pull off in many cases but I've seen it work in surprising situations. I got the original idea when I was testing out the concept of using LoRA swapping for loading different personalities/output styles per batched user prompts. Those were pre-trained, but it occurred to me that soft prompts could be trained much faster to potentially remember just a handful of facts.

The basic idea is that you summarize chunks of the convo over time and create tiny ephemeral datasets that are built for retrieval tasks. You can do this by asking another model to create Q&A pairs for you about the summarized convo context. Each sample in the dataset is an instruction-tuned format with convo context plus the Q&A pair.

The training piece is straightforward but is really where the hair is. Its simple to, in the soft prompt use case, to train a small tensor on that data and just concatenate it to future inputs. But you presumably will have some loss cutoff, and in my experience you very frequently don't hit that loss cutoff in a meaningfully short period of time. Even when you do, the recall/retrieval may still not work as expected (though I was surprised how often it does).

The biggest issue is obviously recall performance given that the weights remain frozen, but also the latency introduced in the real-time training can end up being a major bottleneck, even when you lazily do this work in the background.

Reply on news.ycombinator.com