Hacker News Reader

Comment by viking123 | Hacker News Reader

Does the massive context still needs to be dragged around? Until get a neural network that adjusts weights in real time without relying on a huge clump of context being cycled there, I don't think there will be an AGI or even an impressive "agent". Current agents are just LLM looping lmao sold to people with no knowledge how they work at all.

View on HN

I've been wondering about context compression, actually. I remember a bunch of prompt compression tricks from a couple of years ago that must be usable. If you were to say "anything past half context gets compressed, including the previously-compressed context" would mean you'd still have reasonable workspace and potentially infinite recollection of the most important things. Then yes, you'd be dragging a massive context around, but you're maximising the return on using it. I presume somebody's got a toolkit for this that I don't know the right terms to google for.

Compress first half (even recursively) is a nice idea, but you'd have to sort it by importance first – ie. you don't know if the most important sentence didn't exist at the beginning (ie. the whole goal could have been stated at the beginning, you don't want to loose it – you want memory to fade in terms of importance instead).

I don't know that, but the LLM might be able to make that judgement. If it's involved in the compression process, it can choose to drop cruft from the middle. Maybe "summarise then compress" is a better description.

Yep, let the LLM operate like functional programming functions. Explode(context) … map, reduce, flatten, explode, filter, find, filter, map, flatten, explode … reduce(context).

This is the imagination loop. OpenAI sold the prior reasoning loop. But that’s all this is.

Quite frankly, I wouldn't come to HN for anything less than this quality of content:

https://www.youtube.com/watch?v=DX3qLIwHoUo#t=1m29s

The machine reforming visually (explode, .., reduce) is what I'm describing.

There's been some work on memory lately like Transformer² and Titans. But that may not be necessary for decent agents. Even the "context in a loop" is getting better as general reliability of tasks increases, and as people find better ways to manage context. Like Cline has the Memory Bank prompt where there's a folder of markdown files at diff levels of abstractions of info on the project, current feature, current task progress, etc. And so it can update those files and read from them as a consistent memory.

There is https://www.rwkv.com/ which is an LLM based on RNN's, thus having "infinite" context length, it comes with its own tradebacks though. (Notably that its impossible to actually store infinite information in the network, so it prunes based on which information it finds more important.)

The "massive" context is still tiny compared with the size of the model whose weights you propose to update.

Reply on news.ycombinator.com