Hacker News Reader

I don't get your argument.

Let's say that the forward pass that selected "Aha" produces activations that indicate a wrong assumption, and a plausible explanation.

It puts learned projections of the activation into the KV Cache and outputs Aha.

Both the cached projections and the current Aha token can now influence further activations in an additional Forward pass that the Aha bought the model.

At least that's how I thought it works.

View on HN

A cache is just a cache. I'm not sure what significance you're ascribing to it.

What is put in the cache?

Things that the software running the model would otherwise recompute, if not for the cache. What special meaning are you assigning to it?

Beats me how it works, honestly can't wrap my head around it.

From what I understand, at position Aha in each layer it's constructing a query based on the current activation and looking at the key of each other token position for that layer, in order to decide how much attention to pay to the value.

In this way it attends to the previous values, such as perhaps the incorrect assumption and plausible explanation.

Reply on news.ycombinator.com