It literally is how they work. They look for the next most probable token. Always keep that in mind when writing a prompt or reading output. If you don't supply it, it will fill it in with the most obvious stuff.