Hacker News Reader

You'd need to put the input first for this approach to work, but in my testing models work better if you lead with a question.

View on HN

Hmm. I admit that I haven't thought about this deeply, but I'm not sure that's true? It seems to me that you could extend the KV cache either backwards or forwards equally easily.

You can’t. The later values depend on the earlier ones, so changing the early tokens invalidates your whole cache.

This is also probably why leading with a question works better in the first place. All later processing conditions on the question in this way.

BTW, in my very limited testing, GPT4 doesn’t care about the order.

I could be reading this wrong, but my assumption is/has been that the prompt goes up to the end of the JSON field name, and the LLM is only filling in the actual value, not the key. I could be wrong on this one, however.

Reply on news.ycombinator.com