What’s a useful amount of tokens/second?
What’s a speed like ChatGPT 4?
I’m trying to understand if there are agreed upon metrics such as “frames per second” in other fields or niches
What’s a speed like ChatGPT 4?
I’m trying to understand if there are agreed upon metrics such as “frames per second” in other fields or niches
Depends entirely on your application. If you want a real time chatbot, anything more than 10 tokens/s is probably generating text faster than the user can read, so it's fine. Do you want code suggestions or completions? Probabaly also fine, but a bit slow. Do you want to create summaries for 1000 documents? That's gonna be really slow. But at this frontier of the field, tokens/s is not really the issue. Performance vs. quality is. If you lose a whole lot of accuracy by quantizing floats down to single digit precisions but in turn are able to run 70B parameter models, you often still get better results than less thoroughly quantized 7B parameter models. It's all about getting big models to run on memory limited GPUs.
What feels "good" depends on the person and the type of content, but 35 tokens/s should feel very fast.