Hacker News Reader

CPU inference is both bandwidth and compute constrained.

If your prompt has 10 tokens, it’ll do ok, like in the LinkedIn demo. If you need to increase the context, compute bottleneck will kick in quickly.

Prompt length mainly impacts prefill latency (FTFF), not the decoding speed (TPOT)

Decoding speed won't matter one bit if you have to sit there for 5 minutes waiting for the model to ingest a prompt that's two sentences long.

With ~1000 input, the TTFT is ~10 seconds