Hacker News Reader

Using tokens to evaluate models is an outdated approach. Cost per task is what matters. Not all tokens are created equal

View on HN

Yes, but… more thinking tokens also means longer solution generation time. That said, v4 Flash is a fast model. I use it all the time because it’s very smart for the price. But it is verbose sometimes.

The thinking trace was (preview) frustrating to read, I think I'd prefer a summary view of it at this point.

It’s not outdated at all to use tokens to estimate performance, it’s directly related.

But why should I care? If my metrics are speed and cost? How many tokens it takes as a user is arbitrary to some extent.

If speed is a metric for you, tokens required to solve a problem affects that metric.

All else being equal passing triple the amount of tokens through a model to solve the same problem makes it slower.

Doesn’t mean this model is bad, and it has to be considered how cheap it is, but it’s a factor.

Still, seems useless to measure tokens when you can measure speed.

Reply on news.ycombinator.com