Hacker News Reader

Comment by lucubratory | Hacker News Reader

Basically, yes! There are some technical quibbles (tokens don't line up to words, there's recent research on how to make LLMs stateful so they don't need to pass back the entire conversation for every token but can simply initialise to a remembered state), but this is basically correct as to what LLMs are mechanically doing.

The really exciting stuff is how they're doing that! Research into that question is generally called interpretability, and it's probably what I'm most interested in at the moment.

View on HN

> recent research on how to make LLMs stateful so they don't need to pass back the entire conversation for every token but can simply initialise to a remembered state

I’m very curious about the insights that will give us in to LLMs. My initial instinct is that it won’t work unless you have essentially the entire state remembered since each “point” in the LLM can lead anywhere. I’m imagining an experiment where the task is “Get a person who’s never been to this city before from point X to point Y without describing the destination and while providing as little information as possible”. Giving them the full map with directions is the equivalent to giving the LLM the full previous conversation.

Intuitively there is low-hanging fruit (just provide the directions of single optimized route), but it feels like it gets chaotic quite quickly after that. How many steps could you leave out of turn-by-turn directions before the person is hopelessly lost?

This is based on my current understanding of LLMs so YMMV, but I’m definitely curious about ways we can start from a remembered state.

It's already being implemented into the OpenAI API to reduce per-token API costs. It's not as complicated as I think you're envisioning, you just take the current state in memory of the model at a given token, and then put it in storage when you want to suspend or share the conversation. When you want to continue from that point in the conversation, you just load the state into memory and after inputting your next prompt it works normally. Think of it more like Quick Resume from the Xbox technology stack or the way save states work in gaming emulators.

Those are solid analogies, thank you. I can see how that would work and also a little disappointed because it doesn’t need to give us any additional insight in to LLMs themselves.

How are they doing that?

We don't now for sure, which is why it's so interesting to research! There are some tantalising hints though, for example the OthelloGPT results (here: https://thegradient.pub/othello/) which show that LLMs can form internal world models and then use them in inference to make decisions. Another interesting result is this one (https://www.nature.com/articles/s42003-022-03036-1), which shows that neuronal activations in some parts of the human brain linearly map to neuronal activations in deep learning models when they are both doing the same task; this doesn't tell us anything about how either is doing what they are doing, but it does suggest that the human brain and LLMs may be doing a similar thing to achieve the same end result. We definitely need a lot more research before we can say anything definitive about this question, but there are a bunch of useful research directions that AI and neuroscience researchers are pursuing.

Some interesting follow-ups on OthelloGPT https://www.neelnanda.io/mechanistic-interpretability/othell...

That was fascinating, thank you!

> neuronal activations in some parts of the human brain linearly map to...

Holy.shi..

As I have been observing the ways AI struggles and the ways it produces output, I’ve had this growing suspicion that LLMs have been somehow showing us things about the human brain, but it wasn’t until this moment that I had any validation of the theory.

If you think about the attention mechanism in LLMs, it may not work the same way as the brain, but there should be some functionality of the brain that also deals with attention. And if you think about that, you might also think that attention has some role in consciousness: you need to pay attention to things to be aware of them, and you need awareness of the self for consciousness, etc...

You can get a Turing award by definitively answering this.

Reply on news.ycombinator.com