While the race to incorporate longer and longer context (2K PaLM -> 32K now) is interesting, I don't think that'll scale. It'll just add too much noise to the history: how do you establish a causal relationship between what you're holding in your hand, versus the million other things (context) that you've seen in the past. You'll end up with spurious correlations.

What I think (and this is just me talking out of my ass) will be required is some form of associative long-term memory. Basically, give the model a way to store some embeddings in some form of memory, and then retrieve them based on context: so it doesn't matter if you encountered that item 2 tokens ago, or 2B.

At least this is what my current intuition tells me.