I wonder if anyone will end up employing the "human" solution of padding sentences out with filler words like 'Ummmm' or 'So like' while the generator is determining the actual next token...
Once the model finished reading the prompt and starts answering tokens are generated at constant speed, there are no Ummm... gaps. But some tokens are already decided from the first layers and other tokens need the full depth. So there is a notion of "thinking harder" on particular tokens. We could detect token difficulty and fake the output to show the pauses.