I wrote a whole paper about how to make language models rhyme all the time https://paperswithcode.com/paper/most-language-models-can-be...