This sounds pretty much like beam search (https://en.wikipedia.org/wiki/Beam_search), which is in fact a common generation technique! See eg. https://huggingface.co/docs/transformers/internal/generation...