Hacker News Reader

Generally infill is significantly faster than inference due to batching. Is that not the case here for some reason?

They mention it here: https://github.com/argonautlabsai/deltafin/blob/main/k3-publ...

>device bytes read during the prefill window, all four drives (arm csv) 8,977 GB at 24.1 GB/s aggregate

I.e. low memory bandwidth.

Prefill is generally faster than generation, but not by much on older Mac processors. I get around 70-60 tps in prefill on my M1 Max for Muse Glimmer (not sure about the generation speed, probably between 15 and 30).

They allegedly improved this by “up to 7x” with M5 but I’m not sure about the exact numbers here.

Reply on news.ycombinator.com