You are completely right it's a what is the next token guessing machine so without corpus it's guesses are worse as expected.