Hacker News Reader

30ms for a 0.8B decision model is crazy fast. Most of my inference pipelines struggle to hit that with smaller models.

Is it though? I got sub 10ms for 200M params vision model way back in 2020 with TensorRT optimization running on RTX 2080Ti. 30ms for 0.8B classifier model doesn't sound special to me.

how many ms is it for a 32k token prefill?

Reply on news.ycombinator.com