It's because the input tokens can be batch-processed in a single forward pass through the model, while generating tokens requires one forward pass through the model per token.

If you do the math of how much memory bandwidth is required by a forward pass vs. how much compute, you'll see that inference is entirely limited by memory bandwidth and will use compute resources very inefficiently. In contrast, input processing is able to fully use the available compute.

Of course, there are ways to mitigate this problem, like processing multiple token streams in parallel, but the fundamental problem remains.