Hacker News Reader

This sounds pretty much like beam search (https://en.wikipedia.org/wiki/Beam_search), which is in fact a common generation technique! See eg. https://huggingface.co/docs/transformers/internal/generation...

Sounds like a good avenue to research, but you probably want to generate more than 2 tokens ahead. Try 20 tokens, but I suppose you don't want N^20 executions of the LLM, but more like a representative sampling of say 200 combinations of the next 20 tokens. I don't know how you'd do that.

Novice here.

I like the sound of that!

I don't know the answer but I might experiment with it. Probably a researcher has tried it.

You would need N times the compute per token generate of course.

You could either pick to N, or sample N (with temperature adjustment to logits if needed)