The most limiting factor I’ve come across is hitting the context window. Eventually your new eager employee starts to forget what you’ve taught them but they’re too confident to admit it.
Seems very realistic!
Like a Topic-based Personality Construct where the model first determines which of its “selves” should answer the question, and then grabs appropriate context given the situation.
Here's a bunch of other related methods,
Summarizing context - https://arxiv.org/abs/2305.14239
continuous finetuning - https://arxiv.org/pdf/2307.02839.pdf
retrieval augmented generation - https://arxiv.org/abs/2005.11401
knowledge graphs - https://arxiv.org/abs/2306.08302
augmenting the network a side network - https://arxiv.org/abs/2306.07174
another long term memory technique - https://arxiv.org/abs/2307.02738
However there's been some work that to "get extra milage" out of the current models so-to speak with rotary positions and a few other tricks. These in combination with finetuning is the current method many are using at the moment IIRC.
Here's a decent overview https://aman.ai/primers/ai/context-length-extension/
Rope - https://arxiv.org/abs/2306.15595
Yarn (based on rope) - https://arxiv.org/pdf/2309.00071.pdf
LongLoRA - https://arxiv.org/pdf/2309.12307.pdf
The bottleneck is quickly going to be inference. Since the current transformer models need the context length ^2, the memory requirements go up very quickly. IIRC a 4090 can _barely_ fit a 4bit 30B model in memory with 4096k context length.
From my understanding some form of RNNs are likely to be the next step for longer context. See RWKV as an example of a decent RNN https://arxiv.org/abs/2305.13048
There is nothing stopping someone from keeping an LLM in online-training mode forever. We don't do that because it's economically infeasible, not because it wouldn't work.
What works in my experience - structuring the task similar to a human-driven workflow, breaking it down into small steps is needed. Each step could be driven by a small prompt, relevant document fragments (if RAG is used) and condensed essays/tutorials/guides that were written by a powerful LLM (ideally, GPT-4 pre-Turbo).
Using this approach, you could stay well below 8k token limit even on the most demanding tasks.
(Big size contexts are leaky on all LLMs anyway)
I start to feel like a one eye king under blind people.
I even remember sometimes when I told people specific things.