Hacker News Reader

That is impressive. Interesting that the prefill (i am guessing this is prompt processing) is so much slower than decoding.

Its my understanding that under normal circumstances decoding is memory bandwidth bound which prompt processing isn't due to batching. Is there some quirk in your setup?

Strange. I'm running Llama2 70b on Chrome Canary on a 64GB MacBook M1 Max...~1.5 older...and seeing better performance.

It's slow but usable!

prefill: 2.1963 tokens/sec, decoding: 3.4708 tokens/sec

RTX 4080 Llama-2-77b-chat-hf-q4f32_1

210.6742 tokens/sec, decoding: 20.7758 tokens/sec

Can't tell if its really taking advantage of the all the power of the card, though.

Mac Studio 2022 64 GB RAM M1 Max. Lots of other stuff running. prefill: 1.1620 tokens/sec, decoding: 2.4105 tokens/sec

This is a crazy good performance!

How's the quantization?