Hacker News Reader

HNHacker News

Top New Best Ask Show Jobs

Comment by mjhirn | Hacker News Reader

Parent Full thread

mjhirn·

You mean, comparing the performance after training a model?

View on HN

Performance in terms of model accuracy. How accurate will be those models.

No accuracy is not included, since that is something that should be constant for each model across frameworks.

These benchmarks aim to highlight the performance differences in terms of speed/memory usage across frameworks and machine configurations.

There is also the practical hurdle that training imagenet models to maximum accuracy takes 1 week+.

So, are you saying that output of all frameworks are exactly and always the same?

Sorry, if I am asking stupid questions.

Not exactly, no. But not even the output of the same framework will always be the same since you are usually randomly initalizing the weights in a network and randomly picking the samples used in SGD[1] (the seed for the RNG could of course be a fixed one to mitigate this somewhat).

But in the end, if you are using the same model, the same solver and the same RNG, yes the output of all frameworks should be the same. In practice this also mostly holds true, since the stochastic processes involved are geared towards finding a good local minimum, which is the same given a model and a dataset.

[1]: https://en.wikipedia.org/wiki/Stochastic_gradient_descent

So, there is a strong chance that you are comparing very different things, and frameworks which perform not good in terms of speed can be far superior in terms of models quality.

Reply on news.ycombinator.com