Looks like you don't have a 'most common/simple words' vocab restriction? That's a pretty fun one too. https://twitter.com/jonathanfly/status/1144345934857474048
I think this vocabulary filtering technique will pick up steam again now that good quality open source LLMs are proliferating. I've always wanted to try these techniques with ChatGPT, and the inability to do so has caused me to somewhat publicly complain about their lack of access to ChatGPT's vocabulary probability distribution (they don't give more than the top 10 most likely continuations).
As an aside - do you have any academic references for masking / constraining based next step transitions? It's a trick I've been messing with for years, but I probably picked this up from some older poetry generation work. Haven't been able to find any direct reference for simple 1 step masking, let alone more involved things like constraining based on the unrolled, modified markov sequence or things bigger than 1 step.
https://huggingface.co/blog/constrained-beam-search
This technique has been quietly available within HF for awhile, and has a lot of real world use-cases.
I've also been reading your paper on these topics, based on previous comments on HN. So thanks for that! Always interested in work about constrained generation.
I'd really like to see some version of the techniques from this palindrome generation https://www.ijcai.org/Proceedings/15/Papers/353.pdf revisited in modern contexts. Or some of the other Sony CSL work on this topic (e.g. https://www.francoispachet.fr/wp-content/uploads/2021/01/pap...) which I will admit I (still) do not understand yet.