Hacker News Reader

Prompt length mainly impacts prefill latency (FTFF), not the decoding speed (TPOT)

Decoding speed won't matter one bit if you have to sit there for 5 minutes waiting for the model to ingest a prompt that's two sentences long.

With ~1000 input, the TTFT is ~10 seconds